StudioHomepage
ServicesTables

Change the Schema

Add, rename, drop, move and widen a Table's columns without rewriting its data.

Available

Add a column that your pipeline starts recording, such as an exposure time for each frame, rename a column whose name no longer fits, or widen a number that outgrew its type. A schema change publishes a new Table version like any other write, but it writes metadata only: no data file is copied, so the change costs the same for a Table of a thousand rows or a billion. Jobs that pinned an earlier version keep reading it with the schema it was written with.

Before you start

Add a column

Add exposure_ms after captured_at, and publish the new version:

main = client.project("demo").branch("main")
changed = main.table("frames").alter_schema(
    [add_column(pa.field("exposure_ms", pa.float32()), after="captured_at")],
)
print(changed.schema())
commit = main.commit()
commit.update_table(changed)
commit.publish()

alter_schema returns the next version without publishing it; update_table stages it and publish makes it visible, as for any write. An added column must be nullable, and every existing row reads it as NULL. Without after, the column goes last. To give existing rows values, write them afterwards with patch_rows, or add the column and its values in one step as Add columns shows.

Change existing columns

Pass several changes to make them in one version. They apply in order, so a later change names a column by the name an earlier one gave it:

main = client.project("demo").branch("main")
changed = main.table("frames").alter_schema(
    [
        rename_column("exposure_ms", "exposure"),
        widen_type("exposure", pa.float64()),
        move_column("exposure", after="camera"),
    ]
)
commit = main.commit()
commit.update_table(changed)
commit.publish()

The Python SDK builds each change with a function, and the Rust SDK with a variant of SchemaChange:

ChangePythonRustRule
Add a column or a struct fieldadd_column(field, after=None)AddColumn { field, after }The field must be nullable. Existing rows read NULL.
Drop a column or fielddrop_column(path)DropColumn { path }A column the Table is partitioned or sorted by cannot be dropped.
Rename a column or fieldrename_column(path, name)RenameColumn { path, name }Values are kept.
Let a column hold nullsset_nullable(path)SetNullable { path }A nullable column cannot become NOT NULL.
Widen a numberwiden_type(path, data_type)WidenType { path, data_type }Only without loss, such as int32 to int64 or float32 to float64.
Move a column or fieldmove_column(path, after=None)MoveColumn { path, after }Without after, the column moves first.

A path names a top-level column by its name. To reach a field inside a struct column, give the names from the top-level column down: a sequence of names in Python, or ColumnPath::nested([...]) in Rust.

Verify the result

schema() returns a version's schema without reading any rows. Read it on the version that alter_schema returned, or open the Table again from the branch's new head after you publish.

Troubleshooting

The SDK checks every change against the version you called it on before it writes anything, so a refused change publishes nothing.

ProblemCause
The added column must be nullableThe added field is not nullable.
A change names a missing columnThe path does not exist in the version, or an earlier change in the same call renamed or dropped it.
The type cannot changeThe new type loses values, such as int64 to int32 or an integer to a float. Add a new column and write the converted values into it instead.
The changes leave the schema as it isEvery change was already true of the schema.

How it works

Each column has a stable field ID, and data files store values by field ID, not by name. A rename or a move therefore changes only the schema in the version's metadata, and an added column reads NULL until files for it exist. See Schema changes.

Found a problem? Edit this page or report an issue.

On this page