Skip to main content

Backup, recovery, and upgrades

What must be protected​

Broker data and topology preserve retained event history, immutable rulesets, activation pointers, admission replicas, and advisory state. ClickHouse preserves long-lived analytical projections beyond shorter broker retention. PostgreSQL preserves operator lifecycle, trails, registry decisions, and certificate history. CA signing material and endpoint installation keys have separate confidentiality and recovery requirements.

A backup containing endpoint keys is a credential copy. Restrict and encrypt it according to deployment policy; treat compromise as identity compromise. Do not treat a backup as an ordinary telemetry export.

What replay restores​

Within retained history, consumers can reconstruct projections under their implemented replay semantics. Analysis rebuilds bounded state on assignment. Replay does not recreate an operator's triage decisions, a deleted signing key, or broker records already aged out. Changing a ruleset while replaying can also change the analytical question being asked.

Current procedural limit​

The workspace has persistence and recovery tests, but no complete production backup scheduler, cross-store restore orchestration, HA failover runbook, or guaranteed rollback script. Use supported procedures for the exact deployed store versions and rehearse a restoration into an isolated environment before relying on them.

Upgrade review​

Record source/image revisions, contracts versions, schema migrations, ruleset identity, trust bundles, and active configuration. Review schema changes before starting new binaries; migrations are owned by dedicated executables. Verify admission, query visibility, detections, and operational state after the change.

Restore testing must confirm identities and tenant bindings as well as row counts. A "healthy" service after restoring an inconsistent registry and admission log may still make an incorrect authorization decision. Preserve or reconcile authoritative control state before reopening ingestion.

Retention planning​

Choose replay windows longer than the outages and recovery periods you intend to survive, then measure catch-up time and storage use. No finite retention setting proves unlimited lossless operation. Future agent disk-pressure exceptions must be accounted for separately.

Source evidence​

Reviewed against the source baseline. Seagull-backend-v2/deploy/compose.yaml · Seagull-backend-v2/tests/integration/recovery_test.go · Seagull-backend-v2/internal/postgres/schema/0004_agents.sql · Seagull-agent-v2/README.md.