Object Lifecycle¶

Overview¶

Long-lived branches such as main, staging, and production are where data accumulates fastest. Ingestion pipelines write a fresh set of objects under a new dated prefix every time they execute and nothing removes the older ones, intermediate outputs get committed alongside the results they were meant to produce and then stay there, and sensitive data can outlive the retention window your compliance policy promises. Cleaning this up by hand means writing scripts that list and delete objects one by one, which is slow at data lake scale and leaves no durable record of what policy was applied or when.

Object Lifecycle turns that cleanup into repository-level rule. You define rules that say which objects to expire, on which branches, and after how long, and lakeFS applies them on a schedule. Objects selected by a rule are soft-deleted through a commit on the target branch, so the cleanup is versioned, reviewable, and reversible in the same way any other change to your data is. Once expired, those objects become eligible for permanent removal by Garbage Collection.

Object Lifecycle uses a rule-based configuration model similar to lifecycle configuration in Amazon S3, Azure Blob Storage, and Google Cloud Storage, applied to lakeFS branches instead of buckets.

Benefits¶

• Scale: Cleanup is fast whether a rule expires a hundred objects or ten million.
• Auditable: Every cleanup leaves a record of exactly what was expired, and by which rule.
• Reversible: The state before a cleanup stays reachable until Garbage Collection removes the data.

Use cases¶

• Compliance-driven expiration: Remove sensitive data under a prefix such as pii/ once it passes the retention period your policy allows, or cap retention across the whole repository with a single rule over all objects.
• Storage cost control: Expire committed data under prefixes such as raw/ or tmp/ that no pipeline deletes.

How it works¶

You define rules on a repository. A rule reads as a single sentence: expire objects under raw/ on main after 90 days.

rules:
  - id: raw-landing-expiry
    branches: ["main"]
    prefix: "raw/"
    expire_after_days: 90

An object is expired when its key starts with the rule's prefix and its age exceeds expire_after_days, measured from the object's last modification time as recorded in lakeFS rather than in the object store. Only committed objects are evaluated.

A scheduled job evaluates the rules and, for each target branch, removes all matching objects in a single commit. Evaluation is asynchronous, so a new or edited rule takes effect on the next cycle rather than immediately. The job runs daily by default, on a schedule your server administrator can change (see server configuration), and a single instance runs it at a time even when lakeFS is deployed with multiple replicas. A branch that has uncommitted changes is skipped for that cycle. Skipped branches, along with any branch the job fails to clean for another reason, are left untouched and picked up on the next cycle, so one branch never holds up the rest of the run.

Rule fields¶

Overlapping rules¶

Rules are evaluated independently, with no ordering or priority between them, so overlapping prefixes are allowed and nested prefixes behave predictably. An object matched by several rules is expired as soon as the shortest expire_after_days among them is reached, and the expiration is attributed to that rule. When two matching rules specify the same number of days, attribution goes to the first rule alphabetically by ID.

The cleanup commit¶

Each cycle produces at most one commit per branch, covering every rule that targets it:

• The committer is the dedicated system user object-lifecycle-manager.
• Commit metadata records the per-rule attribution and the instant the run evaluated against (see Monitoring cleanup runs).

Managing rules¶

Rules can be managed from the lakeFS UI, lakectl, or the API. To turn the feature off for a repository, clear its rules.

cat > rules.yaml <<'EOF'
rules:
  - id: "raw-staging-expiry"
    prefix: "raw/"
    branches: ["main"]
    expire_after_days: 90
    description: "Expire raw staging objects"
  - id: "temp-object-expiry"
    prefix: "tmp/"
    branches: ["main", "staging"]
    expire_after_days: 7
EOF

lakectl object-lifecycle set lakefs://my-repo -f rules.yaml

lakectl object-lifecycle get   lakefs://my-repo
lakectl object-lifecycle get   lakefs://my-repo --json
lakectl object-lifecycle clear lakefs://my-repo

{
  "rules": [
    {
      "id": "raw-landing-expiry",
      "prefix": "raw/",
      "branches": ["main"],
      "expire_after_days": 90,
      "description": "Expire raw landing data"
    },
    {
      "id": "max-retention",
      "prefix": "",
      "branches": ["main", "staging"],
      "expire_after_days": 1095,
      "description": "Compliance: expire everything older than 3 years"
    }
  ]
}

Permissions¶

The following RBAC actions gate the feature:

• retention:GetObjectLifecycleRules - read the repository's rules.
• retention:SetObjectLifecycleRules - create, update, or clear rules.

Both are scoped to the repository resource, arn:lakefs:fs:::repository/{repositoryId}.

Monitoring cleanup runs¶

Three places show what a cleanup did:

• The commit on the target branch. Its diff is the per-object list of what was expired, and its metadata carries the per-rule tally (.lakefs.object-lifecycle.expired_by_rule), the run total (expired_total), and the instant ages were judged against (evaluated_at).
• The audit log, where each cleanup commit is recorded with operation_id: CleanupCommit and service_name: object_lifecycle, attributed to the object-lifecycle-manager system user.
• The lakeFS server log, where each run records its duration, the branches and rules it evaluated, the objects expired, and the reason for any branch that was skipped or failed.