Data Curation

The Value of Curated Data¶

As a data lake grows, the useful data inside it becomes harder to find and harder to trust. A single analytics or machine learning project often draws on objects and tables spread across several repositories and prefixes, and the people who produce that data are rarely the people who consume it. Consumers end up needing to know exactly which repository, path, and commit holds the data they want, and that knowledge lives in tribal memory, wikis, or pipeline code rather than in the platform itself.

The common workaround is to copy the relevant data into a new location for each project, which creates duplicate storage, drifts out of sync with the source, and loses the lineage back to where the data came from. Governance suffers too, because access is tied to the physical layout of storage, so sharing a curated slice of data means granting access to whole repositories that also contain data the consumer should not see.

With lakeFS you can curate data into named, versioned, governed views called Datasets, giving consumers a stable, discoverable contract over exactly the data they need while the underlying data stays in place.

Curating Data with lakeFS Datasets¶

A dataset is a first-class entity that names a curated view over repository data. It is a set of pointers plus descriptive metadata, so no data is copied: the dataset definition stores references to objects, prefixes, and Iceberg tables in your source repositories, each pinned to a specific commit. This turns curation into a lightweight, repeatable act rather than an expensive copy, and it keeps the connection to the source data intact.

Discoverability¶

Because datasets sit alongside repositories at the top level of the installation and carry admin-defined metadata such as owner, classification, and tier, consumers can find data by dataset name and by its attributes without knowing which repository it physically lives in. A curated customer-360 dataset becomes the entry point to that data for the whole organization, rather than a path that only its producers remember.

Stable, reproducible contracts¶

Every dataset version is immutable and monotonically numbered, so customer-360@v7 always resolves to the same set of objects and tables no matter how the underlying repositories evolve afterward. Consumers and pipelines can pin to a specific version for a stable contract, or follow latest for the current curated state, and an auditor can point to the exact version that backed a decision. This builds directly on the same versioning that makes lakeFS data reproducible.

Governed sharing¶

A dataset is its own access boundary. A curator can only attach data they are allowed to read, and a consumer reading through the dataset needs permissions scoped to the dataset alone, not to the source repositories. You can therefore share a curated slice of data with a team without exposing the rest of the repository it came from, which keeps sharing governed and least-privilege by default.

Summary¶

Curating data with lakeFS Datasets lets you publish named, versioned, governed views over data that already lives in your lake, so consumers discover the right data, trust that it will not change underneath them, and access exactly the slice they are entitled to. Because datasets are pointers rather than copies, you get this curation without duplicating storage or losing lineage. To set up datasets and work with them in practice, see the Datasets guide, which also shows how datasets are addressable through standard lakeFS URIs and the Iceberg REST Catalog.