Staging Compaction¶

Listing a branch with a large staging area is slow, because uncommitted entries are flat keys in the key-value store while committed data is read from range files indexed by a meta-range. Staging compaction folds those entries into a meta-range, so the branch reads like a committed one without being committed or changed.

It suits long-lived branches that are written continuously and committed rarely, where staging grows into the millions of objects. If you commit regularly, you don't need it.

How staging compaction works¶

A compaction merges the branch's uncommitted entries with the commit they apply to, writes the result as a meta-range in the object store, and records it on the branch. Reads then use that meta-range instead of the staging entries it replaced.

The speedup comes from two things. A listing over a meta-range skips ranges outside its scope and seeks cheaply within a range, while a listing over staging has to scan. Range and meta-range files are also immutable, so lakeFS caches them on local disk and reuses them across reads, whereas staging entries are mutable and are read from the key-value store on every request. Delimited listings gain the most, since they seek past every common prefix, and recursive listings gain the least. See lakeFS Internals for how the two stores divide the work.

Get, List and the uncommitted diff return the same results as before, and the branch stays uncommitted. New writes keep going to staging exactly as they did before, during and after a compaction, and reads merge them with the compacted meta-range.

What clears a compacted meta-range¶

A compacted meta-range is tied to the commit it was built on, and two operations end it:

• Commit includes it in the new commit, along with anything staged since.
• Resetting all uncommitted changes discards it with the rest of staging.

Merge, revert, cherry-pick and hard reset require a clean branch, so they fail on a compacted branch until it is committed or reset.

Compacting branches automatically¶

Automatic compaction is off by default. Turn it on in the server configuration, described in the Enterprise configuration reference:

graveler:
  compaction:
    auto_trigger: true
    threshold: 50000

or with the equivalent environment variables:

LAKEFS_GRAVELER_COMPACTION_AUTO_TRIGGER=true
LAKEFS_GRAVELER_COMPACTION_THRESHOLD=50000

lakeFS then counts staged mutations per branch and compacts a branch once it crosses threshold, which defaults to 50,000. The count is approximate and shared across the cluster.

Compactions run on a background worker pool, so no request waits for one and writes continue while it runs. Only one compaction runs on a branch at a time, and read-only repositories are never compacted. The first compaction of a large staging area is the slowest, since later ones reuse the ranges no new key touches.

Compacting a branch on demand¶

You can compact a single branch through the API whether or not auto_trigger is on. The call requires the branches:CompactBranch permission:

curl -X POST -u "$LAKEFS_ACCESS_KEY_ID:$LAKEFS_SECRET_ACCESS_KEY" \
  "$LAKEFS_ENDPOINT/api/v1/repositories/<repo>/branches/<branch>/compact"

It returns 202 Accepted and runs the compaction in the background, or 503 Service Unavailable when no worker is free, in which case retry later. A compaction that is interrupted by a server restart is not resumed.

To check whether a branch holds a compacted meta-range:

curl -u "$LAKEFS_ACCESS_KEY_ID:$LAKEFS_SECRET_ACCESS_KEY" \
  "$LAKEFS_ENDPOINT/api/v1/repositories/<repo>/branches/<branch>/compact"

{"compacted": true, "meta_range_id": "..."}

This reports what the branch holds now, not whether a compaction is running.

Monitoring compaction¶

lakeFS exports graveler_compaction_seconds, a duration histogram labelled by outcome, alongside its other Prometheus metrics. Count compactions through its _count series:

rate(graveler_compaction_seconds_count{outcome="completed"}[5m])

Tuning the compaction threshold¶

Start at 50,000. Lower it if listings on a busy branch are still slow between compactions, and raise it if compactions run back to back with little to fold each time.