Mailer Documentation
Build and run Go stream processing pipelines with the SDK or declarative workflows.
Mailer is a Go stream processing library for long-running pipelines. It reads records from sources, applies operators such as filter, keyBy, reduce, and window, then writes the result to sinks.
You can build pipelines in two ways:
Go SDK
Compose sources, operators, state, checkpointing, and sinks directly in Go.
Declarative workflows
Define common pipelines in YAML or JSON and run them with the workflow CLI.
Internals
Learn how stages, keyed workers, Pebble state, checkpoints, and windows work.
Operations
Run pipelines with durable state, Kafka, Postgres, metrics, and delivery guarantees.
Current scope
- Local generator workflows can run without external services.
- Kafka sources and sinks are supported for real streaming pipelines.
- Transactional Kafka sinks validate exactly-once requirements.
- Postgres sinks support inserts and upserts.
- Memory and Pebble state backends are available.
- Checkpointing can recover keyed and windowed state.
- Secret placeholders use environment variables and are not printed in metadata.
- The current workflow CLI reads local workflow files; remote workflow storage is future work.
Start here
Run the local order totals workflow from the Mailer repository:
go run ./cmd/mailer-workflow --file examples/workflows/order-totals.yamlExpected data output includes rolling customer totals:
key=c1 value={"sum":10}
key=c1 value={"sum":15}
key=c2 value={"sum":7}The exact timestamp and offset fields can vary by source.
Learn the system
- Start with Getting Started.
- Use SDK Overview for Go usage.
- Use Workflow Format for YAML.
- Use Pebble State Backend and Windowing Internals for internal behavior.