MLR·01 field manualStream processing engine / Go

MAILER

Keyed state. Event-time windows. Exactly-once Kafka. One process.

Mailer is an embeddable stream processing engine for Go — Flink-style semantics without the cluster. Build pipelines in the Go SDK or declare them in YAML; both compile into the same stage-based runtime with durable Pebble state and barrier checkpointing.

Get started →Read the internals
source▶filter▶keyBy▶window▶reduce▶sink▶source▶filter▶keyBy▶window▶reduce▶sink▶source▶filter▶keyBy▶window▶reduce▶sink▶source▶filter▶keyBy▶window▶reduce▶sink▶

exactly-once

Kafka → Kafka, end to end

Source offsets, operator state, and sink output commit as one transaction.

1 process

no cluster, no JVM

An embeddable Go library. Runs on a laptop; import it, build, execute.

~75 ms

checkpoint at 5M keys

Pebble hard-link checkpoints scale with changed data — not total state.

58 ms

state restore after crash

Rewind to the last checkpoint and replay. Measured, not aspirational.

§ 01 Authoring

Write Go. Or don't.

The SDK gives you custom transforms and full connector control. YAML workflows give you validated, declarative pipelines with a CLI runner. Same planner, same state, same guarantees.

Go SDK

main.go
env := mailer.NewEnv().
    WithCheckpointing(30*time.Second,
        checkpoint.NewFileStorage("./ckpt")).
    WithStateBackend(state.Pebble("./state"))

env.FromSource(src).
    KeyBy(byCustomer).WithPartitions(4).
    Reduce(sumAmounts).
    ToSink(sink.NewTxnKafkaSink(
        sink.TxnKafkaBrokers("localhost:9092"),
        sink.TxnKafkaTopic("order-totals"),
        sink.TxnKafkaTransactionalID("orders-v1"),
    ))

env.Execute(ctx)
SDK reference →

YAML workflow

order-totals.yaml
name: order-totals

pipeline:
  - id: completed
    type: filter
    filter: {field: status, operator: equals,
             value: completed}
  - id: by-customer
    type: keyBy
    keyBy: {field: customer.id, partitions: 4}
  - id: totals
    type: reduce
    reduce: {function: sum, field: amount}

sink:
  type: stdout
Workflow guide →

§ 02 Runtime anatomy

Small surface. Sharp guarantees.

Five mechanisms carry the whole engine. Each one is documented down to the file and covered by crash tests.

Execution engine docs
  1. 01

    Plan

    stage planner + bounded edges

    Operators fuse into stages that run as plain function calls. Bounded channels between stages are the only buffers — a full edge blocks upstream, so a slow sink throttles Kafka by construction.

  2. 02

    Keyed state

    hash router + worker clones

    KeyBy routes each key to one worker, and every worker owns an isolated state backend — in-memory or a per-worker Pebble LSM on disk.

  3. 03

    Barriers

    broadcast + strict alignment

    Checkpoint barriers flow in-band. Parallel stages broadcast them to every worker and re-align at the exit; operators snapshot synchronously as the barrier passes.

  4. 04

    Exactly-once

    two-phase commit + txn marker

    A transactional Kafka sink stages each interval; the coordinator commits sink output, offsets, and state atomically. A marker record resolves crashes between commit and completion.

  5. 05

    Observability

    Prometheus + dashboard

    Per-stage and per-edge metrics. An edge pinned at capacity names your bottleneck; send-block seconds make backpressure measurable.

§ 03 Delivery guarantees

Say what you mean by “delivered.”

Full guarantee table →
GuaranteeConfigurationWhat happens on a crash
at-most-onceNo checkpointingRestart begins from the configured start offset.
at-least-onceCheckpointing + any sinkState is exact; replay may re-emit records the sink already wrote.
exactly-onceExactly-once source + txnKafka sink + checkpointingCommitted output visible once, under read_committed.

§ 04 Ship it

Start local. Point it at Kafka when the shape is right.

go get github.com/ASHUTOSH-SWAIN-GIT/mailer

Get startedExamples