How the Catalog Works

The Catalog never stores your data; it stores pointers. A Project state is two small maps: one from names to Resources, and one from each Resource to the exact version selected. Every guarantee that your application relies on follows from that: a query pins one state and never sees a later change, a commit swaps several pointers at once, and branches cost nothing.

A Project state is two mapsCopy Anchor Link

names:     frames        -> resource 6f1c…
           frame-search  -> resource 91ab…
versions:  resource 6f1c… -> version v7
           resource 91ab… -> version v3

The data of each version lives with the service that owns the Resource. A Table version is a set of files in Project storage, and a Location version is a storage root plus a file inventory. Every version is immutable and has its own ID. A commit records a new pair of maps, so it is small no matter how much data the versions hold.

A query reads one stateCopy Anchor Link

A read pins one commit, looks up each name in that commit's maps, and reads the exact version that it finds. It never looks up "latest" again partway through, so a commit that lands while your query runs cannot change its result. Reading history works the same way with an older commit. This is why a commit ID or a tag is a complete record of what a job read.

A commit collects changes, then publishes onceCopy Anchor Link

A writer pins one commit and prepares its changes privately. Each write creates a new immutable version, such as a Table version whose files are already uploaded, and nobody else can see it. The writer then stages which version each Resource moves to and which names change.

Publishing sends every staged change in one request. The Catalog checks that each Resource still has the version that the writer started from, then swaps all the pointers at once. Readers see the whole commit or none of it.

When two writers raceCopy Anchor Link

A publication names the commit that it started from, and it can also name the branch head that it expects. Without an expected head, two publications that started from the same commit both succeed when they touch different Resources. When they touch the same Resource or the same name, the later one gets a conflict and publishes nothing. With an expected head, any commit on the branch in the meantime rejects the publication, even one that touched other Resources. The Rust SDK always names the head that it pinned. Either way, a rejected writer's versions were built on data that is now out of date, so it must pin the new head and redo its write.

A lost response is different from a conflict: the publication may or may not have happened. The client sends the identical request again under the same idempotency key, and the Catalog reports whether that exact publication already happened. It never publishes twice. See Retries.

What this gives youCopy Anchor Link

• Atomic multi-Resource changes. One commit can publish new versions of several Tables and new names. A Location refresh publishes its file inventory and its new Location version together.
• Free branches and tags. A branch or a tag names a commit; nothing copies data.
• Exact staleness checks. A derived Resource such as a Vector Index records the source version that it covers, and compares it with the version that your commit selects.
• History that stays readable. Old commits keep resolving, so pinned training runs and audits read years later exactly what they read when they ran.