- lakeFS Components
lakeFS is distributed as a single binary encapsulating several logical services:
The API Gateway implements lakeFS’ compatibility with S3. It implements a compatible subset of the S3 API to ensure most data systems can use lakeFS as a drop-in replacement for S3.
To achieve this, the gateway exposes an HTTP listener on a dedicated host and port.
See the S3 API Reference section for information on supported API operations.
The Swagger (OpenAPI) Server exposes the full set of lakeFS operations (see Reference). This includes basic CRUD operations against repositories and objects, as well as versioning related operations such as branching, merging, committing and reverting changes to data.
The S3 Storage Adapter is the component in charge of communication with the underlying S3 bucket. It is logically decoupled from the S3 Gateway to allow for future compatibility with other types of underlying storage such as HDFS or S3-Compatible storage providers.
See the roadmap for information on future plans for storage compatibility.
The Metadata index contains the versioning logic used inside lakeFS. It maps logical entities such as branches, commits and object paths to the relevant underlying storage.
The metadata itself is managed in PostgreSQL. This makes it relatively easy to maintain, is offered as a managed service by many providers and has a rock solid foundation.
The Auth service handles creation, management and validation of user credentials and RBAC policies.
Currently, the auth service manages its own database of users and credentials and does not use IAM in any way.
The UI layer is a simple browser-based client that uses the OpenAPI server. It allows management, exploration and data access to repositories, branches, commits and objects in the system.