Unified Data Management

One Platform for All Your Data¶

Data platforms rarely deal with a single kind of data or a single place to keep it. A typical organization has structured tables sitting next to unstructured files such as images, documents, and model artifacts, spread across more than one storage system and often more than one cloud. Managing each of these with its own catalog, its own permission model, and its own access path leads to fragmented governance, duplicated effort, and data that is hard to move without losing consistency.

lakeFS Enterprise lets you manage all of that data as one versioned system. You get a single place to keep structured and unstructured data together, one access-control model that spans your storage backends and clouds, and data-access paths tuned for the way modern compute actually consumes data. The rest of this page walks through the three pillars that make this possible.

Multi-modal data in a single repository¶

A lakeFS repository is format-agnostic, so structured and unstructured data live side by side under the same versioning engine. You can manage Apache Iceberg tables through the built-in Iceberg REST Catalog while keeping unstructured objects such as images, audio, and documents in the same repository, and a single commit can capture a consistent snapshot across both. This means the tabular features that drive a model and the raw files those features were derived from are versioned together, rather than in two disconnected systems.

Because both kinds of data share the repository, Metadata Search can query across them from one interface. You can filter objects by system metadata such as path and size or by your own user-defined metadata, which is especially useful for machine learning workflows that need to select training data by labels and file attributes, and then join that selection with the structured tables that describe it. Managing structured and unstructured data in one repository, queryable through one search, removes the seams that normally force teams to stitch together separate catalogs.

Unified access control across backends and clouds¶

When data spans multiple storage systems, keeping permissions consistent is usually the hardest part. lakeFS Enterprise applies one Role-Based Access Control model over all the data it manages, so a single set of policies governs access regardless of where the underlying bytes physically live. You define who can read and write which resources once, and that decision holds across the whole installation.

This matters most when you use multiple storage backends to manage data across on-premises, hybrid, and multi-cloud environments. Rather than reconciling separate IAM systems for each cloud and each bucket, you govern everything through lakeFS, giving you one place to enforce data security and compliance while still keeping the data itself distributed across the locations that suit your infrastructure and cost model.

Optimized data access and retrieval¶

Managing data in one place is only useful if compute can reach it efficiently. lakeFS Mount optimizes this path by virtually mounting a repository as a local filesystem and fetching data lazily, so expensive compute such as GPU training jobs read files directly from the underlying object store through pre-signed URLs without routing bulk data through the lakeFS server. This keeps costly accelerators busy instead of idle and waiting on I/O, while your code keeps using ordinary filesystem calls.

For distributing data across regions or environments, transactional mirroring replicates repositories into consistent read-only copies in remote locations. Because the replica reflects committed, consistent states rather than a stream of partial changes, consumers in another region read a coherent view of the data rather than an in-progress one. Together, Mount and mirroring let you place data close to the compute that needs it, without giving up the consistency guarantees that make the data trustworthy.

Summary¶

Unified data management with lakeFS Enterprise means structured and unstructured data managed together and searchable through one interface, one access-control model that spans every storage backend and cloud, and data-access paths built for the demands of modern compute. Instead of operating a separate catalog, permission system, and delivery mechanism for each kind of data and each location, you manage it all as one versioned, governed platform. To go deeper on the building blocks, see the Iceberg REST Catalog, Metadata Search, RBAC, multiple storage backends, lakeFS Mount, and transactional mirroring.