Writing
Write new Table versions from Arrow input, edit rows, columns and schemas, chain writes, and recover a write whose registration is unknown.
Your application writes Arrow data to Tables through the write methods of
Table and Project. Each write streams its rows to storage and returns a
new, immutable version of the Table. Nobody else sees that version until a
Commit stages it and publishes, so all of a commit's
versions become visible together.
For the step-by-step tasks, see Create a Table, Write rows, Change the schema and Add columns. This page covers the Rust methods and their input.
Write methods
| Method | Returns |
|---|---|
project.write_table(input) | The first version of a new Table. It has no name until commit.create_table(name, &table) binds one. |
table.append(input) | The next version: the Table's rows followed by input. |
table.overwrite(input) | The next version, with only the rows of input. The Table keeps its name and ID. |
table.delete_rows(filter) | DeletedRows, which holds the next version without the matching visible rows and the number of rows removed. |
table.patch_rows(filter, set) | UpdatedRows, which holds the next version, with the set expressions' values recorded as patches in the matching rows, and the number of rows updated. |
table.merge_rows(source, on, actions) | MergedRows, which holds the next version with source merged in by the on columns, as MergeActions say, and the rows inserted, updated and deleted. |
table.alter_schema(changes) | The next version with each SchemaChange applied in order, written as metadata only. |
table.add_columns(columns) | The next version with a new nullable column for each NamedExpression, filled for every row. |
table.column_writer(fields) | A ColumnWrite<Table> that writes values for new nullable fields, one fragment or chosen rows at a time. |
table.writer() | A TableWrite for the next version, built from several appends, blob uploads, blob packs, and patches and deletions of rows named by row ID. See Combine several changes in one version and Upload large blobs with their rows. |
project.table_writer(schema) | A TableWrite for the first version of a new Table. |
TableWrite::finish and ColumnWrite<Table>::finish register the version and return
it as a Table. table.schema() returns a version's Arrow schema without
reading rows.
TableWrite::patch_rows and delete_rows name their target
rows in a ROW_ID_COLUMN column of non-null u64 row IDs. A scan returns that
column when ScanOptions::row_id_column names it; see
Reading.
To stage a new version of an existing Table, open the Table from the Commit's
own head, write, and pass the result to commit.update_table(&table). You can
chain writes: append to the result of an append, and stage only the last
version. The commit replaces the head's version with that one, and the versions
in between are never published.
Commits bind new Tables at the Project root under a single name, such as
frames. A Commit cannot bind a Table under a nested path such as
robot-a/frames yet.
Delete rows
delete_rows takes an optional Expression. Pass Some(predicate) to select
rows or None to delete every visible row. The backend must support the
complete predicate exactly; an unsupported predicate fails before anything is
written. A null predicate does not select a row.
The method writes deletion files rather than rewriting retained rows, so
retained rows keep their row IDs. It returns DeletedRows { table, rows }.
When no row matches, it writes nothing, rows is zero, and table is the
Table you called it on. Staging that Table publishes no new version.
Arrow input
write_table, append, overwrite, merge_rows, and the methods of
TableWrite and ColumnWrite that take rows consume one Arrow input of one of
these types:
| Input | Use it for |
|---|---|
RecordBatch | One batch. |
Vec<RecordBatch> | Several batches. Must not be empty. |
(SchemaRef, Vec<RecordBatch>) | Input that may be empty, because the schema cannot come from a batch. |
Box<dyn RecordBatchReader + Send> | Streaming input that does not fit in memory. |
The SDK streams the input to storage batch by batch. Because it consumes the input, the SDK cannot read it a second time, which is why a lost registration is recovered with a receipt rather than by writing again.
Recover a write
After the SDK uploads a write's files, it registers the new version with the
Tables service. If the response to that registration
is lost, the write returns Error::RegistrationUnknown with a receipt for the
files already in storage. Keep the receipt and pass it back instead of
uploading again:
- For the next version of a Table, call
table.recover_write(receipt)on the Table you wrote from. - For the first version of a new Table, call
client.register_prepared(receipt).
Both calls register the same files and return the version as a Table.
Stage it in your Commit as if the write had succeeded.
Backends
Writes use the backend of the client. The embedded backend writes files to object storage from your process; the network backend sends Arrow streams to a hosted ScanDAL service, which writes the files.
Limitations
- Commits bind new Tables at the Project root only.
- Streaming writes to Spiral-managed storage through the embedded backend need S3-compatible storage, because they use multipart uploads.
- A write that is cancelled, or never published, can leave files in the Table's storage that no published version references.