Skip to main content

Data ownership and persistence

DatasetOwner / sourcePersistenceConsumersRecovery implication
Admitted eventsGateway; typed producer observation with stamped identityRaw broker topic, then security_eventsAnalysis, event writer, query APIReplay only within retained broker history
DetectionsAnalysis engine and immutable rulesetDetection topic, then security_detectionsDetection writer, alert writer, query APIDeterministic identity supports repeated delivery
Alerts / occurrencesAlert writer creates; control API triagesPostgreSQLControl APIReplay cannot reconstruct operator decisions
Incidents / trailsCorrelation detection creates; control API triagesPostgreSQLControl APIPreserve story evidence and operational history
Agent registry / certificate historyControl planePostgreSQL, replicated admission logGateway roster and control APIRoster is a projection; registry owns decisions
Ruleset versions / active pointerControl publicationCompacted broker logAnalysis and control registryBootstrap files are not a substitute for published history
Inventory items / scansGateway records; inventory projectorInventory topic; ClickHouseInventory consumers and future matchingFull-scan semantics determine currentness
Advisory versions / feed attemptsImporter-stamped upstream intelligenceCompacted topic; ClickHouseFuture matching, operatorsFailed/absent feed data does not mean withdrawal
Agent installation / keysEndpoint installationPrivate local directoryAgent foundationLosing keys requires deliberate identity recovery
SessionsControl processBounded in-memory allowlistThat process's guardRestart invalidates sessions; replicas do not share them

Why these stores exist​

ClickHouse serves high-volume analytical evidence and projection workloads. PostgreSQL owns mutable transactional control state and append-only trails. Redpanda owns durable asynchronous handoff and retained replay. Agent local persistence protects installation identity today; a durable telemetry spool is planned.

Redis, Elasticsearch/OpenSearch, and object storage are not part of the current V2 deployment. Legacy technology choices are not inherited requirements.

Retention and replication​

Broker retention and analytical TTLs serve different purposes. Raw events default to a seven-day replay window. The analytical schemas define their own retention, including the detection evidence retention documented in ADR 17. Inventory and advisory schemas preserve different histories; read the migration reference rather than applying one global retention policy.

The development deployment uses local volumes and one broker replica. Database replication, off-host backups, retention sizing, and tested restores are production responsibilities without a complete supplied HA topology.

Source evidence​

Reviewed against the source baseline. Seagull-backend-v2/deploy/compose.yaml · Seagull-backend-v2/internal/clickhouse/schema/0001_security_events.sql · Seagull-backend-v2/internal/postgres/schema/0004_agents.sql · Seagull-backend-v2/docs/decisions/0012-storage-is-owned-per-workload.md · Seagull-backend-v2/docs/decisions/0017-noise-is-removed-from-the-alert-and-never-from-the-detection.md · Seagull-backend-v2/docs/decisions/0027-inventory-is-a-record-kind-of-its-own.md · Seagull-backend-v2/docs/decisions/0028-vulnerability-intelligence-is-read-from-its-source.md.