Quickstart

Follow these quickstarts to try out lakeFS Enterprise.

⚠️ lakeFS Enterprise Quickstarts are not suitable for production use-cases.

lakeFS Enterprise Sample
Docker Quickstart
Kubernetes Helm Chart Quickstart

lakeFS Enterprise Sample

The lakeFS Enterprise Sample is the quickest way to experience the value of lakeFS Enterprise features in a containerized environment. This Docker-based setup is ideal if you want to easily interact with lakeFS without the hassle of integration and experiment with lakeFS without writing code.

By running the lakeFS Enterprise Sample, you will be getting a ready-to-use environment including the following containers:

lakeFS
Fluffy (includes lakeFS Enterprise features)
Postgres: used by lakeFS and Fluffy as a shared KV store
MinIO container: used as the storage connected to lakeFS
Jupyter notebooks setup: Pre-populated with notebooks that demonstrate lakeFS Enterprise’ capabilities
Apache Spark: this is useful for interacting with data you’ll manage with lakeFS

Checkout the RBAC demo notebook to see lakeFS Enterprise Role-Based Access Control capabilities in action.

Docker Quickstart

Prerequisites

You have installed Docker Compose version 2.23.1 or higher on your machine.
Access to download dockerhub/fluffy from Docker Hub. Contact us to gain access to Fluffy.
With the token you’ve been granted, login locally to Docker Hub with docker login -u externallakefs -p <TOKEN>.

The quickstart docker-compose files below create a lakeFS server that’s connected to a local blockstore and spin up the following containers:

lakeFS
Fluffy (includes lakeFS Enterprise features)
Postgres: used by lakeFS and Fluffy as a shared KV store

You can choose from the the following options:

Recommended: A fully functional lakeFS Enterprise setup without SSO support
Advanced: A fully functional lakeFS Enterprise setup including SSO support with OIDC integration configured

If you can postpone the evaluation of the SSO integration, we suggest starting without it to speed up overall testing. The SSO integration requires additional configurations and is best addressed later.

Recommended (SSO Disabled)
Advanced (SSO Enabled)

Create a docker-compose.yaml file with the following content
Run docker compose up in the same directory as the docker-compose.yaml file.
In your browser, go to http://localhost:8080 to access lakeFS UI.

version: "3"
services:
  lakefs:
    image: "treeverse/lakefs:1.25.0"
    command: "RUN"
    ports:
      - "8080:8080"
    depends_on:
      - "postgres"
    environment:
      - LAKEFS_LISTEN_ADDRESS=0.0.0.0:8080
      - LAKEFS_LOGGING_LEVEL=DEBUG
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY="random_secret"
      - LAKEFS_AUTH_API_ENDPOINT=http://fluffy:9000/api/v1
      - LAKEFS_AUTH_API_SUPPORTS_INVITES=true
      - LAKEFS_AUTH_UI_CONFIG_RBAC=internal
      - LAKEFS_AUTH_AUTHENTICATION_API_ENDPOINT=http://localhost:8000/api/v1
      - LAKEFS_AUTH_AUTHENTICATION_API_EXTERNAL_PRINCIPALS_ENABLED=true
      - LAKEFS_DATABASE_TYPE=postgres
      - LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=postgres://lakefs:lakefs@postgres/postgres?sslmode=disable
      - LAKEFS_BLOCKSTORE_TYPE=local
      - LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs
      - LAKEFS_BLOCKSTORE_LOCAL_IMPORT_ENABLED=true
    entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
    configs:
      - source: lakefs.yaml
        target: /etc/lakefs/config.yaml
  postgres:
    image: "postgres:11"
    ports:
      - "5433:5432"
    environment:
      POSTGRES_USER: lakefs
      POSTGRES_PASSWORD: lakefs

  fluffy:
    image: "${FLUFFY_REPO:-treeverse}/fluffy:${TAG:-0.4.4}"
    command: "${COMMAND:-run}"
    ports:
      - "8000:8000"
      - "9000:9000"
    depends_on:
      - "postgres"
    environment:
      - FLUFFY_LOGGING_LEVEL=DEBUG
      - FLUFFY_DATABASE_TYPE=postgres
      - FLUFFY_DATABASE_POSTGRES_CONNECTION_STRING=postgres://lakefs:lakefs@postgres/postgres?sslmode=disable
      - FLUFFY_AUTH_ENCRYPT_SECRET_KEY="random_secret"
      - FLUFFY_AUTH_SERVE_LISTEN_ADDRESS=0.0.0.0:9000
      - FLUFFY_LISTEN_ADDRESS=0.0.0.0:8000
      - FLUFFY_AUTH_SERVE_DISABLE_AUTHENTICATION=true
      - FLUFFY_AUTH_POST_LOGIN_REDIRECT_URL=http://localhost:8080/
    entrypoint: [ "/app/wait-for", "postgres:5432", "--", "/app/fluffy" ]

configs:
  lakefs.yaml:
    content: |
      auth:
        ui_config:
          login_cookie_names:
            - internal_auth_session

This setup uses OIDC as the SSO authentication method thus requiring a valid OIDC configuration.

Create a docker-compose.yaml with the content below.
Create a .env file with the configurations below in the same directory as the docker-compose.yaml, docker compose will automatically use that.
Run docker compose up in the same directory as the docker-compose.yaml file.
Validate the OIDC configuration:
- In your browser, go to http://localhost:8080 to access lakeFS UI
- Complete the Setup process, and login with your Admin credentials
- Logout and try to login again, you will be redirected to the OIDC login page.

.env

FLUFFY_AUTH_OIDC_CLIENT_ID=
FLUFFY_AUTH_OIDC_CLIENT_SECRET=
# The name of the query parameter that is used to pass the client ID to the logout endpoint of the SSO provider, i.e client_id
FLUFFY_AUTH_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER=
FLUFFY_AUTH_OIDC_URL=https://my-sso.com/
FLUFFY_AUTH_LOGOUT_REDIRECT_URL=https://my-sso.com/logout
# Optional: display a friendly name in the lakeFS UI by specifying which claim from the provider to show (i.e name, nickname, email etc)
LAKEFS_AUTH_OIDC_FRIENDLY_NAME_CLAIM_NAME=

docker-compose.yaml

version: "3"
services:
  lakefs:
    image: "treeverse/lakefs:1.25.0"
    command: "RUN"
    ports:
      - "8080:8080"
    depends_on:
      - "postgres"
    environment:
      - LAKEFS_LISTEN_ADDRESS=0.0.0.0:8080
      - LAKEFS_LOGGING_LEVEL=DEBUG
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY="random_secret"
      - LAKEFS_AUTH_API_ENDPOINT=http://fluffy:9000/api/v1
      - LAKEFS_AUTH_API_SUPPORTS_INVITES=true
      - LAKEFS_AUTH_UI_CONFIG_LOGIN_URL=http://localhost:8000/oidc/login
      - LAKEFS_AUTH_UI_CONFIG_LOGOUT_URL=http://localhost:8000/oidc/logout
      - LAKEFS_AUTH_UI_CONFIG_RBAC=internal
      - LAKEFS_AUTH_AUTHENTICATION_API_ENDPOINT=http://localhost:8000/api/v1
      - LAKEFS_AUTH_AUTHENTICATION_API_EXTERNAL_PRINCIPALS_ENABLED=true
      - LAKEFS_DATABASE_TYPE=postgres
      - LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=postgres://lakefs:lakefs@postgres/postgres?sslmode=disable
      - LAKEFS_BLOCKSTORE_TYPE=local
      - LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs
      - LAKEFS_BLOCKSTORE_LOCAL_IMPORT_ENABLED=true
      - LAKEFS_AUTH_OIDC_FRIENDLY_NAME_CLAIM_NAME=${LAKEFS_AUTH_OIDC_FRIENDLY_NAME_CLAIM_NAME}
    entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
    configs:
      - source: lakefs.yaml
        target: /etc/lakefs/config.yaml
  postgres:
    image: "postgres:11"
    ports:
      - "5433:5432"
    environment:
      POSTGRES_USER: lakefs
      POSTGRES_PASSWORD: lakefs

  fluffy:
    image: "${FLUFFY_REPO:-treeverse}/fluffy:${TAG:-0.4.4}"
    command: "${COMMAND:-run}"
    ports:
      - "8000:8000"
      - "9000:9000"
    depends_on:
      - "postgres"
    environment:
      - FLUFFY_LOGGING_LEVEL=DEBUG
      - FLUFFY_DATABASE_TYPE=postgres
      - FLUFFY_DATABASE_POSTGRES_CONNECTION_STRING=postgres://lakefs:lakefs@postgres/postgres?sslmode=disable
      - FLUFFY_AUTH_ENCRYPT_SECRET_KEY="random_secret"
      - FLUFFY_AUTH_SERVE_LISTEN_ADDRESS=0.0.0.0:9000
      - FLUFFY_LISTEN_ADDRESS=0.0.0.0:8000
      - FLUFFY_AUTH_SERVE_DISABLE_AUTHENTICATION=true
      - FLUFFY_AUTH_LOGOUT_REDIRECT_URL=${FLUFFY_AUTH_LOGOUT_REDIRECT_URL}
      - FLUFFY_AUTH_POST_LOGIN_REDIRECT_URL=http://localhost:8080/
      - FLUFFY_AUTH_OIDC_ENABLED=true
      - FLUFFY_AUTH_OIDC_URL=${FLUFFY_AUTH_OIDC_URL}
      - FLUFFY_AUTH_OIDC_CLIENT_ID=${FLUFFY_AUTH_OIDC_CLIENT_ID}
      - FLUFFY_AUTH_OIDC_CLIENT_SECRET=${FLUFFY_AUTH_OIDC_CLIENT_SECRET}
      - FLUFFY_AUTH_OIDC_CALLBACK_BASE_URL=http://localhost:8000
      - FLUFFY_AUTH_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER=${FLUFFY_AUTH_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER}
    entrypoint: [ "/app/wait-for", "postgres:5432", "--", "/app/fluffy" ]
    configs:
      - source: fluffy.yaml
        target: /etc/fluffy/config.yaml

 #This tweak is unfortunate but also necessary. logout_endpoint_query_parameters is a list
 #of strings which isn't parsed nicely as env vars.
configs:
  lakefs.yaml:
    content: |
      auth:
        ui_config:
          login_cookie_names:
            - internal_auth_session
            - oidc_auth_session
        oidc:
          # friendly_name_claim_name: "name"
          default_initial_groups:
            - Admins

  fluffy.yaml:
    content: |
      auth:
        oidc:
          logout_endpoint_query_parameters:
            - returnTo
            - http://localhost:8080/oidc/login

Kubernetes Helm Chart Quickstart

In order to use lakeFS Enterprise and Fluffy, we provided out of the box setup, see lakeFS Helm chart configuration.

The values below create a fully functional lakeFS Enterprise setup without SSO support. The created setup is connected to a local blockstore, and spins up the following pods:

lakeFS
Fluffy (includes lakeFS Enterprise features)
Postgres: used by lakeFS and Fluffy as a shared KV store

If you can postpone the evaluation of the SSO integration, we suggest starting without it to speed up overall testing. The SSO integration requires additional configurations and is best addressed later. To try lakeFS Enterprise SSO capability on a Kubernetes cluster, check out the production deployment guide.

Prerequisites

You have a Kubernetes cluster running in one of the platforms supported by lakeFS.
Helm is installed
Access to download dockerhub/fluffy from Docker Hub. Contact us to gain access to Fluffy.

Instructions

Add the lakeFS Helm repository with helm repo add lakefs https://charts.lakefs.io
Create a values.yaml file with the following content and make sure to replace <fluffy-docker-registry-token> with the token Docker Hub token you recieved, <lakefs.acme.com> and <ingress-class-name>.
In the desired K8S namespace run helm install lakefs lakefs/lakefs -f values.yaml
In your browser go to the Ingress host to access lakeFS UI.

lakefsConfig: |
  logging:
      level: "DEBUG"
  blockstore:
    type: local
ingress:
  enabled: true
  ingressClassName: <ingress-class-name>
  annotations: {}
  hosts:
    - host: <lakefs.acme.com>
      paths:
       - /
fluffy:
  enabled: true
  image:
    privateRegistry:
      enabled: true
      secretToken: <fluffy-docker-registry-token>
  fluffyConfig: |
    logging:
      level: "DEBUG"
  secrets:
    create: true
  sso:
    enabled: false
  rbac:
    enabled: true

# useDevPostgres is true by default and will override any other db configuration, set false for configuring your own db
useDevPostgres: true

Quickstart

Table of contents

lakeFS Enterprise Sample

Docker Quickstart

Prerequisites

Kubernetes Helm Chart Quickstart

Prerequisites

Instructions