Skip to main content

Event lifecycle and ACKs

Before the request​

The producer assigns a stable event_id, records event and observation time, and constructs the typed event body. The future agent must preserve that identity and those timestamps across retries. Today, the backend development probe exercises this contract; there is no functioning V2 agent delivery loop.

Durable admission​

  • Telemetry producer
  • ingest-gateway
  • Redpanda
  1. 1Telemetry producer to ingest-gateway: EventBatch over verified mTLS
  2. 2ingest-gateway: Resolve roster tenant; overwrite identity and reception
  3. 3ingest-gateway: Validate the entire batch
  4. 4ingest-gateway to Redpanda: Publish, wait for all in-sync replicas

  5. ifAll publication succeeded

    1. 5Redpanda to ingest-gateway: Durable publication result
    2. 6ingest-gateway to Telemetry producer: accepted=true, durable=true, received=count

    elseTimeout or partial failure

    1. 5ingest-gateway to Telemetry producer: 503; retain and retry the batch

Validation is whole-batch, but broker publication is not an atomic batch transaction. Some records can become durable before publication returns an error. Retrying the entire request is correct and can repeat those records. An idempotent Kafka producer does not eliminate duplicates from a separately retried HTTP request.

What the ACK proves​

BatchAck has accepted, durable, and received. It contains neither an echoed batch_id nor per-record acceptances. The producer must correlate the reply to its request and require all three conditions. Inventory ACK counts are counts of records, not items.

The gateway's all-in-sync-replicas acknowledgement is only as resilient as the replication and minimum-in-sync configuration. The development topology has one replica. It provides a broker durability boundary, not replicated high availability.

Downstream effects​

  • Redpanda
  • Consumer
  • Consumer-owned store
  1. 1Redpanda to Consumer: Records, possibly repeated
  2. 2Consumer to Consumer-owned store: Persist output or quarantine
  3. 3Consumer-owned store to Consumer: Success
  4. 4Consumer to Redpanda: Commit consumed offsets

The event writer projects the observation into ClickHouse. Analysis independently normalizes a working representation and evaluates a pinned ruleset. Detections enter a second topic. The detection writer preserves analytical evidence; the alert writer creates operator work above the configured floor and applies alert-plane folding and suppression.

An event can be acknowledged before any of these effects is queryable. Trace consumer lag and writer progress before treating a query delay as lost ingestion. Duplicates must be investigated at the relevant layer rather than suppressed indiscriminately.

Source evidence​

Reviewed against the source baseline. Seagull-backend-v2/internal/ingest/http.go · Seagull-backend-v2/internal/broker/publisher.go · Seagull-backend-v2/internal/broker/consumer.go · Seagull-contracts/proto/seagull/ingest/v1/ingest.proto.