How Vector Indexes Work

A search makes two passes. First it scores compact codes in the few partitions nearest your query vector, which is cheap and approximate. Then it reads the exact stored vectors of the best candidates and ranks them precisely. The index holds only codes that it can rebuild from the Table, never your data, so dropping or rebuilding an index loses nothing.

What a build createsCopy Anchor Link

A refresh of frame-search reads the embedding column of one exact frames version twice. The first scan selects a training sample. The second scan writes the index:

1. Partition. The build clusters the vectors into partitions groups, each with a center. Each vector joins its nearest group.
2. Compress. Each vector is stored in the index as a short code, using the RaBitQ method, that estimates distances without decoding the vector.
3. Keep the row identity. Every index entry records the row ID of its source row, not a file position, so it still finds the row after maintenance rewrites the Table's files.

The index engine is named vortex.ivf-rabitq, after these two techniques.

A refresh records its exact index version, source Table version, and embedding field identity in the immutable manifest. These identities belong to the artifact and remain independent of later catalog changes.

The default partition count follows the square root of the row count, up to 4,096. An explicit count above 4,096 fails.

The default build limits the retained training sample to 256 MiB, including sample entries. Fewer vectors fit at larger dimensions. This limit changes the sample size, not the indexed row count. A refresh fails if the sample cannot train the requested partitions.

Centroid calculation, partition assignment and code generation share the Worker CPU budget, up to eight workers.

Source batches split into smaller slices before vector conversion when they exceed the host byte limit. Each flush writes one posting object directly to Project storage within the host row and byte limits. Construction uses no local spill files. The default limits permit 65,536 rows and 64 MiB per posting object.

The Worker sizes posting buffers from its memory budget. Small buffers across many partitions produce many small posting objects. After the training scan, the build checks a conservative bound on posting runs. A build fails before the posting scan if that bound exceeds the manifest descriptor limit. More Worker memory or fewer partitions can reduce the bound.

The manifest uses separate byte and descriptor limits because it describes every posting object. Refresh and search share default limits of 32,768 posting descriptors and 4 MiB per manifest. A refresh fails before publication if its manifest exceeds these limits. These limits exclude allocation overhead and library workspace.

The Worker reserves memory for computation before it admits sample and output buffers. This estimate grows with vector dimensions. A refresh fails if the Worker cannot admit that workspace. The resource profile sets the available Worker memory.

Invalid vectors and build settings cause permanent Task failure. Transient storage failures can retry.

What a search doesCopy Anchor Link

A search reads the probes partitions whose centers are nearest the query vector, and estimates the distance to every entry in them from the codes. It then takes the best rerank candidates, reads their exact vectors from the covered frames version, and ranks them by true distance. The candidate step is approximate, so a true neighbor in a partition that the search did not probe can be missed, but every distance and rank in the result is exact.

Raise probes to miss fewer neighbors, and raise rerank to rank more candidates exactly. Both make a search cost more.

The default execution limit permits at most 100,000 candidates. A search that needs more candidates fails instead of reducing its candidate count. A build rejects a default rerank value above the host candidate limit.

Versions and stalenessCopy Anchor Link

Each refresh writes a new immutable index version to Project storage and records the exact Table version that it covers. A Catalog commit then makes it the current version. If two refreshes run at once, both can finish, but only one becomes current.

Because each version records its source, staleness is an exact check, not an estimate: when the commit that you read selects a different frames version from the one the index covers, a search fails as stale. Spiral never searches a stale index silently, and it never swaps an approximate index into an ordinary query: a search always names the index.

RebuildsCopy Anchor Link

Index state is derived data. The vortex.ivf-rabitq engine supports full rebuilds only, so every refresh reads the whole source version. A later change to the index encoding can also require a full rebuild. Incremental refresh, which reads only the rows that changed since the last build, is planned.

Refreshes produce codec version 3 by default. The packed_postings engine option enables codec version 4 with compact code planes. Artifacts with JSON manifests or without exact source identities require a full refresh. A full refresh writes a replacement in the current format. Existing artifacts do not change automatically.