Skip to main content

28. Vulnerability intelligence is a record kind of its own, read from its source and never inferred from what a feed left out

Historical decision record

This decision is reproduced from the pinned backend revision. Read amendment notices and the current architecture before treating historical statements as current behavior.

Context​

ADR 27 gave the platform one half of vulnerability management: what an asset currently has. BE-035 is the other half — what is known to be wrong with software — and BE-036 will be the match between them. The card asks for a pipeline of feeds that keeps where every piece of intelligence came from and how fresh it is, that normalises ecosystems and package names, and whose failure cannot corrupt the inventory.

v1 did this from the agent. Its vulnerability scanner inferred an OSV ecosystem from the package manager, queried api.osv.dev from every endpoint, and sent findings to the server. Almost everything that went wrong there is a lesson this record answers:

  • the ecosystem had no release. dpkg became Debian or Ubuntu with no version, so a Debian 12 host was compared against whatever release OSV chose; dpkg on any distribution it did not recognise became Debian; pacman became Arch Linux, which is not an OSV ecosystem at all, and every Arch host was reported clean;
  • binary packages were matched against advisories that name source packages. Debian's advisories name openssl; a host has libssl3 installed;
  • an unchanged package list skipped the query, so an advisory published after the last upgrade was never matched until something else changed;
  • a scan reported complete with nothing observed resolved every open finding. The server reconciled findings against the list a scan observed, and a skipped query observes nothing, so every other scan closed everything and the next one opened it again;
  • a finding was keyed by the OSV id, so the PyPI advisory and the GitHub advisory for one CVE were two findings.

What the feeds actually are was measured on 2026-09-13. OSV's public export has, per ecosystem, an index of every current record and when it last changed (modified_id.csv) and each record on its own. Debian's index lists 66,660 records and modified 54,643 of them in thirty days, mostly by re-exporting them unchanged; Ubuntu's archive of the same records is 682 MB and one of its kernel advisories is 1.5 MB. Debian, Ubuntu, Alpine and Rocky Linux name source packages; AlmaLinux names binaries; Red Hat scopes an advisory to a product stream such as rhel_e4s:8.8::baseos and SUSE to a product's marketing name.

Decision​

Vulnerability intelligence is a record kind of its own — its own contract, topic, store and processes — read from its source by the one process that reaches the internet, kept with where and when it was read, and never concluded from a feed that failed or from a record a feed stopped listing.

The one process that reaches the internet holds no store​

seagull.vulnerability.v1 carries an Advisory — what a source says about one vulnerability, translated — and a FeedSync — what one attempt to follow a feed found. cmd/advisory-importer reads the feeds, translates what they say and publishes both to security.advisories, a compacted topic of one partition keyed by what each record is about: an advisory by its source and id, so compaction keeps the newest version of each, and a sync by its feed. cmd/advisory-writer consumes that topic into ClickHouse.

The importer writing the store directly was the smaller design and was weighed properly. It loses on the reasons that decide this platform:

  1. The importer is the one process exposed to content from the internet. It parses JSON anybody on the path to a feed could have shaped, and reaching out of the stack is its whole job. On the backbone its whole reach is producing to one topic, which BE-004's ACLs can restrict to exactly that; with the store in front of it, it would hold a credential to the database every tenant's telemetry lives in. tests/architecture refuses cmd/advisory-importer any import of a store or of the inventory, and internal/advisoryfeed any import of the inventory domain.
  2. A feed outage and a store outage fail apart. A feed that cannot be read stops nothing but the importer; a store that cannot be reached holds up neither the feeds nor anything else, and becomes lag on one consumer group.
  3. Intelligence changes are events. BE-036's matcher can consume the topic and re-match what an advisory names when the advisory changes, which is exactly what v1's skipped query never did.
  4. The store is rebuilt from the backbone without reading the internet again, and it is the grain every other record kind already uses.

The cost is a contracts release, two processes where one could have done, and a replay of the topic when the importer starts.

What the platform reads​

OSV and nothing else, because it is what the need requires: it carries the distributions' own advisories — Debian's security tracker, Ubuntu's CVE tracker, Alpine's secdb, Rocky's and AlmaLinux's errata — with the CVE each derives from and the CVSS vectors their authors assigned. No NVD feed, no vendor feed read directly and no score computed; each is a source to add when a need names it.

The importer follows a feed's index and fetches each record whose listed change is newer than what the platform holds. That is one path for the first read and every later one, needs no disk and holds one record at a time; the per-ecosystem archive would be 13 times less transfer the first time and is not worth a second path. The export is read over https only, redirects included, from OSV's bucket or from a mirror laid out the same way and signed by a bundle a deployment names: what a feed says decides what the platform calls vulnerable, and a feed read in the clear is one anybody on the path can edit into saying a package is fixed.

Five distributions are supported: Debian, Ubuntu, Alpine, Rocky Linux and AlmaLinux. The others are left out on purpose and the platform says so:

  • Red Hat and SUSE scope an advisory to a product stream or a product module no inventory field names, and a guessed stream is a wrong answer;
  • Ubuntu Pro fixes land in pockets of their own, Ubuntu:Pro:22.04:LTS, which apply only to an asset attached to Pro, and nothing says whether one is;
  • language ecosystems — PyPI, npm, Go — have no collector;
  • Arch Linux has no OSV ecosystem.

An ecosystem is named with its release, and both sides meet on one key​

The platform's name for an ecosystem is OSV's spelling with the release in it — Debian:12, Ubuntu:22.04:LTS, Alpine:v3.19, Rocky Linux:9, AlmaLinux:9 — because a package is compared against one release of its distribution and the release is part of what is being named. internal/vulnerability builds that name from a feed's spelling and from what an asset reports about itself with the same function, so an advisory entry and an installed package are a lookup rather than a guess. A package is named the way that distribution's advisories name it: the source package for Debian, Ubuntu, Alpine and Rocky — read out of dpkg's openssl (3.0.11-1) or rpm's openssl-3.0.7-27.el9.src.rpm — and the binary for AlmaLinux.

An asset the platform cannot assess says why. A derivative that names no distribution the platform reads, a package another manager installed, a host that names no release: each is an error of its own. Cannot assess is not the same answer as not affected, and v1's Arch hosts are what conflating them looks like.

An advisory is what its source said, translated​

OSV is translated and never adopted, as Sigma is under ADR 22. What the platform's advisory can say is kept faithfully: identity, when it changed, when it was withdrawn, the three kinds of name a record has for other records — aliases for the same vulnerability, upstream for what a distribution's advisory derives from, related for the rest — its severities as the vectors or ratings their authors wrote, and per package the ranges of versions affected and the versions listed. A version list is kept whole even where it only expands a range: telling the two apart needs the ecosystem's ordering of versions, which belongs to the matcher.

What it cannot say is left out, and nothing is guessed in its place: an entry in an ecosystem no asset can be placed in, a range stated in commits, a severity on a scale no build reads, references and the database's private fields. Prose is cut to fit because it names nothing; a name, an id, a version or a boundary that cannot be read refuses the record. A record in a later major version of the format is refused rather than half read. A time left at the zero instant is read as no time at all — Debian's converter writes one where it knows no publication date, and the first real import found 37 — and a time the platform cannot hold is refused where the advisory is read, rather than published and then refused by the store while the importer believed it held it.

Provenance.normalization is the version of these rules. A build that changes what a translation says raises it, and the importer reads every record it holds from an older reading again.

Provenance is the platform's to write​

Every advisory carries the feed it was read from, that feed's index version, the URL the bytes came from, when they were fetched, their SHA-256, the format and its version, and the rules it was translated with. A translation says what the feed said; only the importer knows where and when it asked, so it writes the rest itself. vulnerability.Validate refuses an advisory missing any of it, at the importer and again at the writer, so no piece of intelligence reaches the store without saying where it came from. The digest was checked against the source's live bytes after the first real import and matched.

Freshness is two instants, and an attempt that failed moves neither​

A FeedSync says when the importer asked (checked_at), when the platform last held everything the feed listed (synced_at), and the newest change the feed itself lists (newest_listed). The first two separate a platform that is behind from one that is asking; the last separates a stale feed from a stale copy of a fresh one — OSV's per-release folders stopped updating in October 2024, and read through this they would say so. An attempt that failed or stopped part way carries synced_at forward from the last complete one, so freshness never looks better than it is. The same instants are gauges per feed, which a deployment alerts on.

A gap in intelligence is never an answer​

Nothing is concluded from what a feed did not say. An advisory a feed's index stops listing is kept exactly as it was, because an index is not a tombstone — OSV leaves records behind in its bucket without withdrawing them and renames records between exporters — and an advisory is withdrawn only when its source says so, as a newer version. A failed attempt publishes nothing but its own failure; one that stops part way publishes what it read and owes the rest; a record that is not an advisory is refused alone, counted, and asked for again only once the feed changes it. The platform's copy of a feed can be old; it is never wrong in a way it does not report. BE-036 inherits the rule: a finding is not resolved by intelligence that failed to arrive.

Every version is kept, and a moved clock is not a version​

vulnerability_advisories holds a row per version of an advisory — its source, its id, the modification time the source gave it and the rules it was read with — and vulnerability_affected a row per package a version affects, ordered by ecosystem and package, which is how a matcher asks. Nothing replaces another version and nothing is deleted, so a finding made from a version stays explicable after the source moves on. The newest version of an advisory is the one with the greatest modification time, read with the greatest normalisation that version was stored under; the definition is stated once, as a query, in the integration suite, and BE-036 starts from it.

Most of a feed's churn is re-exports that change nothing but the modification time. The importer compares what a record says, apart from when and where it was read, with what it holds, and publishes it only if it differs; a new reading with newer rules is always published. A version older than the one held is never published, so a record that took the long way through a cache cannot put an advisory back; and an advisory that says it changed more than an hour after it was read is refused, as is anything on the log the importer could not have read yet, because holding it would keep every real change older than it out.

Caching, retry and failure​

The importer reads the whole topic back before it asks a feed anything, so it knows what it holds, and its cache is the backbone rather than a disk. The index is asked for with If-None-Match; only records listed as newer than what is held are fetched, eight at a time by default across every feed. An origin that is unreachable or busy is asked again with a widening delay a bounded number of times, and then the attempt stops rather than hammering it; an origin that answered anything else was answered. After an attempt that did not complete, the feed is tried again sooner, backing off to the interval. Each feed is followed on its own, so one that fails holds up no other.

What this deliberately does not do​

No matching and no findings. That is BE-036, and so is the policy that picks a severity out of the vectors an advisory carries.

No retraction by absence. Reading a complete index as a snapshot, the way ADR 27 reads a scan, would retire advisories a feed stopped listing. It is left out because the only case seen is an exporter renaming its records, and the cost of being wrong is v1's: intelligence that quietly stopped saying a package is vulnerable. listed beside held on every sync makes the drift visible.

No read API. Nothing outside the platform reads advisories yet. The contract is the message a read will return, as ADR 13 requires.

No archive of the raw records. The digest and the location say what was read; keeping the bytes would make the platform a mirror of OSV.

Consequences​

  • Contracts v0.14.0 carries seagull.vulnerability.v1.
  • Two processes more, the ninth and tenth that run for good, and two topics: security.advisories, compacted and kept, and security.advisories.quarantine for what the writer refuses.
  • Three tables more, none partitioned and none expiring but the one holding sync attempts. ADR 12 has one more class decided.
  • The stack gains a network with a way out, feeds, and only the importer is on it.
  • The first read of a feed is its whole index. Alpine's 4,657 records took three minutes at eight fetches at a time and a restart afterwards re-read none of them; Debian's, fourteen times as many, took thirty-eight minutes. A deployment follows only the distributions its estate runs, and there is no default.
  • Whoever can produce to security.advisories decides what the platform calls vulnerable, and a plausible forged version also stops the importer asking for the real one. In the development stack that is anybody on the backbone network, which is one more reason BE-004's ACLs matter: only advisory-importer should produce there.

Source evidence​

Generated from docs/decisions/0028-vulnerability-intelligence-is-read-from-its-source.md at fa3bf69. Refresh the checked-in snapshot before regenerating; a normal build does not access another repository.