Refresh a Location
List a Location's storage root and publish its updated file inventory with a refresh Task.
A Location's inventory is the list of files under its root the last time it was refreshed. A refresh lists the root with the Location's Connection, reconciles the inventory with what it finds, and publishes the inventory and a new Location version together. Run it as a Task whenever new data lands in the bucket.
Before you start
- The Location's name and branch. Over HTTP, its Resource ID and the branch ID instead; get both from the Catalog. See Select an exact state.
- Permission to read, list, and update the Location.
- A Worker Pool that can run Tasks for the Project.
Refresh the Location
In Studio, open the Location on a branch and choose Refresh on its Preview tab. Studio submits the refresh Task with the resource profile you select, follows it, and shows the new inventory when it succeeds.
In code, resolve the Location on a branch and refresh it. The refresh runs as a Task; wait on it for the versions it published:
location = client.project("demo").branch("main").location("recordings")
task = location.refresh()
if task is None:
print("a refresh of this Location is already waiting")
else:
for published in task.wait().versions:
print(f"{published.resource_id} is now {published.version_id}")The refresh succeeds the Location version at the head of the branch. It keeps
running if you stop waiting; only cancelling the Task stops it. If a refresh of
the same Location and branch is already waiting to run, no new one is
submitted: the Python and Rust SDKs return None, the TypeScript SDK returns
null, and the HTTP response has "type": "skipped".
What the Task does
The Task lists every object under the root and matches it to the inventory by
path:
| Listing | Inventory |
|---|---|
| A path the inventory lacks | Adds a row. |
| A path whose URI, size, modification time, or ETag changed | Updates the row. |
| A path whose metadata is unchanged | Leaves the row as it is. |
| An inventory path the listing lacks | Deletes the row. |
It then publishes the inventory and a new Location version in one commit. The version records how many rows changed; zero means that the bucket matched the inventory, and the refresh still publishes a new Location version. If the Location or its inventory changes while the Task runs, it lists again from the new head.
Verify the result
The result lists the versions the refresh published and the commit that
published them: the new Location version, preceded by the inventory's new
version when the listing changed it. Over HTTP, read the Task until its status
is terminal; the result is its published field. See
Inspect Tasks.
The new Location version records how many inventory rows the refresh added, updated, or deleted.
Availability
Refresh runs for Locations whose Connection vends Amazon S3 or Google Cloud Storage credentials. Refreshing Hugging Face Locations is planned. The Task holds the whole inventory in memory while it merges, so very large roots need a larger resource profile.