Data Retention¶
Overview¶
lakeFS keeps every version of your data. That promise has a consequence worth planning for: nothing leaves your object store unless you ask for it. Retired datasets, temporary data that was never deleted, and abandoned branches accumulate over time, and they bring storage costs and compliance exposure with them.
Data retention in lakeFS is handled by three features, each removes a different kind of clutter:
- Object Lifecycle expires objects on the branches you choose, according to rules you define on prefix and age.
- Branch Lifecycle deletes stale branches once they match a name pattern and cross an age or inactivity threshold.
- Garbage Collection permanently removes objects from the underlying storage after every branch that referenced them has dropped them and their retention period has passed.
The three are complementary rather than interchangeable, and a complete retention policy usually needs all of them.
What each feature does¶
| Feature | What it removes | Where it acts | Reclaims storage |
|---|---|---|---|
| Object Lifecycle | Committed objects matching a prefix and older than a retention period | Target branches you name in the rule | No |
| Branch Lifecycle | Branch references that are stale by age or inactivity | Any branch matching the policy pattern, except the default branch | No |
| Garbage Collection | Object data in the object store | The underlying storage | Yes |
How they fit together¶
Deleting an object in lakeFS does not delete any bytes. A deletion is recorded as a change in a commit, so the previous commit still contains the object and any branch or tag pointing at that history still resolves to it.
Garbage Collection is the only feature that frees storage, and it will only remove an object that no live reference reaches and whose retention period has elapsed on every branch that once held it. Object Lifecycle and Branch Lifecycle are what create those conditions: one retires the data on your long-lived branches, the other retires the short-lived branches that would otherwise keep pinning it.
flowchart LR
A["<b>Object Lifecycle</b><br/>expires objects by rule<br/>on target branches"] --> B["<b>Branch Lifecycle</b><br/>deletes stale branches<br/>still referencing them"]
B --> C["<b>Garbage Collection</b><br/>hard-deletes unreferenced<br/>objects from storage"]The three retention stages. Storage is reclaimed only after all three have done their part.
Where to start¶
If your goal is lowering storage cost, start with Garbage Collection, since it is the only feature that actually deletes data, and set retention rules that match how far back your teams need to travel. Then add Object Lifecycle rules for the committed prefixes that grow without bound, such as landing zones and staging paths that no pipeline deletes, and Branch Lifecycle policies for the ephemeral branch patterns your workflows create.
If your goal is meeting a retention obligation, such as guaranteeing that nothing lives longer than three years or that personal data is removed within a fixed window, start with Object Lifecycle. It is the feature that expresses the obligation as an auditable, repository-level rule, and every cleanup it performs is recorded as a commit and an audit log entry. Pair it with a Garbage Collection retention period short enough that the physical deletion also lands inside your window.