Skip to main content

Consumer lag keeps increasing

Symptoms​

Committed lag grows, query visibility falls behind, or the engine repeatedly rebuilds state.

Likely causes​

Slow/unavailable stores, refused broker commits, repeated dependency retries, state recovery after rebalance, incompatible activation, or more input than measured processing capacity. Partition counts and sole-reader requirements constrain scaling.

Diagnosis​

Compare input rate with durable output and commit progress. Read writer/engine errors before tuning. Inspect partition-held/moved and state-rebuild metrics; check whether a new replica invalidated a sole-reader declaration. Confirm retention still covers unprocessed work.

Resolution​

Restore the dependency first. Adjust resource/capacity only after measuring the bottleneck. Reduce needless rebalance churn. Scale consumers only where their partition/group semantics permit it. Do not add analysis replicas blindly for cross-agent temporal rules.

Verification​

Committed lag declines under a representative workload, writes remain durable, state activation is executable, and no required partitions or observations are silently skipped.

Continue with health and metrics and failure semantics.

Source evidence​

Reviewed against the source baseline. Seagull-backend-v2/internal/broker/metrics.go · Seagull-backend-v2/internal/broker/recovery.go · Seagull-backend-v2/docs/decisions/0023-state-is-owned-by-the-partition-and-rebuilt-by-reading-it-back.md.