Maintain a Table
Merge small files and fold in updates and deletes with a maintenance Task, once or on a schedule.
A Table that receives a small append every few seconds ends up with thousands of small files, and its scans slow down. Maintenance rewrites those files into fewer, larger ones in the background. It never changes query results, row IDs, or older versions; it only makes later reads cheaper. Run it as a Task, once or on a schedule.
When to run maintenance
Run maintenance when a Table receives many small appends, such as a robot that uploads frames every few seconds, or when it has accumulated updates and deletes. Both leave work that every scan repeats: opening many small files, or applying change records on top of the data files. A Table that you write in a few large batches rarely needs it.
Before you start
- The branch ID of the target branch, and the Resource ID of the Table. Get both from the Catalog. See Select an exact state.
- Permission to view, read, and write the Table.
- A Worker Pool that can run Tasks for the Project.
Submit the maintenance Task
Open the Table on its branch and maintain it, or submit a tableMaintenance
definition with the branch and the Table:
frames = client.project("demo").branch("main").table("frames")
task = frames.maintain()
if task is None:
print("maintenance of this Table is already waiting")
else:
print(f"published {len(task.wait().versions)} versions")The read target must be latestOnBranch, not a fixed commit, because each
step of the Task reads the current branch head and publishes on top of it. The
Task uses the cpu-2 profile unless you select another: cpu-1, cpu-4,
cpu-8, or cpu-16.
The response has "type": "accepted" and the Task. If a maintenance Task for
the same Table and branch is already waiting, the response can be
"type": "skipped". When the Task succeeds, its published field lists the
Table version that each compaction step published.
Run it on a schedule
For a Table with a steady stream of small appends, let an Automation submit the same Task on an interval, for example every 15 minutes. The Automation page shows this exact setup.
What the Task does
The Task runs the recommended operations in this order, and repeats until none is recommended:
| Operation | What it does |
|---|---|
patch_materialization | Folds row updates into new data files. |
delete_materialization | Folds row deletes into one deletion bitmap per fragment. |
deleted_row_purge | Rewrites fragments without their deleted rows. |
small_file_coalesce | Merges small files into fewer, larger ones, toward 512 MiB each. |
fragment_coalesce | Merges small groups of rows into larger ones. |
Each step publishes its own commit, so a large backlog drains in pieces and readers benefit as it goes. Each commit saves its step result, so a later Attempt resumes from committed steps after a Worker failure. Committed steps survive cancellation and failed Attempts. The Task allows three Attempts. The Table and branch IDs keep it attached to the same resources when you rename them.
A write that lands while maintenance runs is not lost. An append adds rows that maintenance did not touch, so the maintenance step still publishes. A change to the same files that a step rewrote makes that step conflict, and the Worker prepares fresh work from the new head. A long gap in Table history can also require fresh preparation.
Every step keeps these guarantees: every row keeps its row ID and its values, the row count does not change, and older versions read exactly as before. See How Tables are stored.
Verify the result
Read the Task until its status is terminal. The result contains complete,
completedSteps, and operations, which counts the committed steps for each
operation. complete: true with completedSteps: 0 means that the Table
needed no work. See Inspect Tasks.
The Task does not delete the files that it replaced. They stay in your bucket, because older versions still reference them; physical deletion of unreferenced files is planned.