Workflow Schema

name: string
version: string
env:
  bufferSize: number
  shutdownTimeout: duration
  checkpointing:
    interval: duration
    dir: string
  state:
    backend: memory | pebble
    dir: string
source:
  type: generator | slice | kafka
pipeline:
  - id: string
    type: filter | selectFields | renameFields | setFields | keyBy | reduce | window
sink:
  type: stdout | blackhole | kafka | txnKafka | postgres

Field paths

Declarative JSON operators use dot-separated paths:

customer.id
payment.total

Filter operators

Filters support equality, inequality, ordering, and string/list containment comparisons through the workflow operators package.

Validation

Invalid workflows are rejected in three layers, before anything runs:

• Parse rejects unknown or misspelled fields (strict decoding) and invalid duration strings.
• Validate accumulates every remaining problem into one error: unsupported types, missing or duplicate operator ids, invalid operator ordering, and unsupported delivery-guarantee combinations.
• Compile rejects unsafe Postgres table/column identifiers and unresolved ${VAR} references.