lakeFS Mount¶

lakeFS Mount lets you virtually mount a remote lakeFS repository onto a local directory or within a Kubernetes environment, using the everest command-line tool. Once mounted, you can access data as if it resides on your local filesystem, using any tool, library, or framework.

Use Cases¶

• Simplified Data Loading: Use your existing tools to read and write files directly from the filesystem with no need for custom data loaders or SDKs.
• Seamless Scalability: Scale from a few local files to billions without changing your tools or workflow. Use the same code from experimentation to production.
• Enhanced Performance: lakeFS Mount supports billions of files and offers fast, lazy data fetching, making it ideal for optimizing GPU utilization and other performance-sensitive tasks.

Getting Started¶

This guide will walk you through setting up and using lakeFS Mount to mount a lakeFS repository on your local machine.

Prerequisites¶

• lakeFS Cloud account or lakeFS Enterprise Version 1.25.0 or higher.
• Supported OS: macOS (with NFS V3), Linux and Windows (using CFAPI).
• Get the lakeFS Mount Binary: lakeFS Mount is a self-contained binary with no installation required. Please contact us to get access.

Authentication & Configuration¶

lakeFS Mount uses the same configuration and authentication methods as lakectl. It discovers credentials and the server endpoint in the following order:

1. Command-Line Flags: --lakectl-access-key-id, --lakectl-secret-access-key, and --lakectl-server-url.
2. Environment Variables: LAKECTL_* or EVEREST_LAKEFS_* prefixed variables.
3. Configuration File: ~/.lakectl.yaml (or the file specified by --lakectl-config).

Authentication Methods

lakeFS Mount will attempt to authenticate in the following order:

1. Session Token: From EVEREST_LAKEFS_CREDENTIALS_SESSION_TOKEN or LAKECTL_CREDENTIALS_SESSION_TOKEN. If the token is expired, authentication will fail.
2. lakeFS Key Pair: Standard access key ID and secret access key (credentials are picked up from lakectl configuration if lakeFS Mount-specific credentials are not provided).

3. IAM Authentication: If your lakeFS environment is configured for AWS IAM Role Login, lakeFS Mount (≥ v0.4.0) can authenticate using your AWS environment (e.g., AWS_PROFILE). IAM authentication is only attempted when no static credentials are set. To enable this, configure your .lakectl.yaml with provider_type: aws_iam. The token is seamlessly refreshed as long as the AWS session remains valid.

   To configure IAM authentication using environment variables, use the EVEREST_LAKEFS_* or LAKECTL_* prefix:

   export EVEREST_LAKEFS_CREDENTIALS_PROVIDER_TYPE=aws_iam
   # or
   export LAKECTL_CREDENTIALS_PROVIDER_TYPE=aws_iam
   

export EVEREST_LAKEFS_CREDENTIALS_PROVIDER_AWS_IAM_CLIENT_LOG_PRE_SIGNING_REQUEST=true

Create Your First Mount¶

Let's mount a prefix from a lakeFS repository to a local directory. In read-only mode, lakeFS Mount pins a specific commit ID. If you provide a branch name, it will resolve to the HEAD commit at the time of mounting.

1. Mount the repository: This command mounts the datasets/pets/ prefix from the main branch of the image-repo repository into a new local directory named ./pets.

   everest mount "lakefs://image-repo/main/datasets/pets/" "./pets"
   

2. Explore the data: You can now use standard filesystem commands to interact with your data. Files are downloaded lazily only when you access their content.

   # List files - this only fetches metadata
   ls -l "./pets/dogs/"
   
   # Find files
   find ./pets -name "*.small.jpg"
   
   # Open a file - this triggers a download
   open -a Preview "./pets/dogs/golden_retrievers/cute.jpg"
   

3. Unmount the directory: When you are finished, unmount the directory.

   everest umount "./pets"
   

Core Concepts¶

This section will help you understand how lakeFS Mount manages performance, consistency, and caching in both local and Kubernetes deployments.

Cache Behavior¶

lakeFS Mount uses a local cache to improve performance when accessing files from lakeFS. Understanding how the cache works will help you optimize performance for your specific use case.

How Caching Works

When you access a file through a mounted lakeFS path, lakeFS Mount follows this process:

1. Lazy Fetching: Files are only downloaded when their content is accessed (e.g., reading a file, not just listing it with ls).
2. Cache Storage: When an object is not found in the local cache, lakeFS Mount fetches the data from the object store and stores it in the cache for subsequent access.
3. Cache Reuse: Subsequent reads of the same file are served directly from the cache, eliminating network requests and improving performance. Caches can't be shared between different instances of mount.

Default Cache Behavior

By default, lakeFS Mount creates a temporary cache directory when you run everest mount. This directory is automatically cleared when the mount is terminated via everest umount.

Key points:

• Each new mount creates a fresh cache directory.
• By default cache location is managed by lakeFS Mount and cleaned up automatically.
• The cache is ephemeral and does not persist between mount sessions. Unless you specify the cache directory.

Persistent Cache

To reuse cache data across multiple mount sessions, you can specify a custom cache directory using the --cache-dir flag:

everest mount lakefs://image-repo/main/datasets/ ./datasets --cache-dir ~/.everest-cache

Benefits of persistent cache:

• Faster startup times when remounting the same data.
• Reduced bandwidth usage by reusing previously downloaded files.
• Useful for iterative workflows where you repeatedly mount and unmount the same repository.

Cache Management

lakeFS Mount manages cached data based on the commit ID of the mounted reference:

• Commit-Based Caching: Each commit ID has its own cache namespace. This ensures that cached data always corresponds to the correct version of your files.
• Cache Invalidation on Commit: When you commit changes in write mode using everest commit, the mount point's source commit ID is updated to the new HEAD of the branch. As a result, the cache associated with the old commit ID is no longer used, and new data will be cached under the new commit ID.

Consistency & Data Behavior¶

File System Consistency

lakeFS Mount provides strong read-after-write consistency within a single mount point. Once a write operation completes, the data is guaranteed to be available for subsequent read operations on that same mount.

lakeFS Consistency

Local changes are reflected in lakeFS only after they are committed using the everest commit command. Until then:

• Changes are only visible within your local mount point
• Other users or mounts will not see your changes
• If two users mount the same branch, they will not see each other's changes until those changes are committed

Sync Operation

When you run everest diff or everest commit, lakeFS Mount performs a sync operation that uploads all local changes to a temporary location in lakeFS for processing. This ensures your changes are safely transferred before being committed to the branch.

See the Write-Mode Operations section for more details on working with writable mounts.

Performance Considerations¶

lakeFS Mount achieves high-performance data access through:

• Direct Object Store Access: By default, lakeFS Mount uses pre-signed URLs to read and write data directly to and from the underlying object store, bypassing the lakeFS server for data transfer. Only metadata operations go through the lakeFS server.
• Lazy Metadata Loading: Directory listings are fetched on-demand, allowing you to work with repositories containing billions of files without upfront overhead.
• Cache Sizing: Setting an appropriate --cache-size prevents frequent eviction and re-fetching. As a rule of thumb, size your cache to accommodate your working set.
• Network Bandwidth: Since data is fetched directly from object storage, ensure your network connection has adequate bandwidth for your workload.

Working with Data (Local Mount)¶

Read-Only Operations¶

Read-only mode is the default and is ideal for data exploration, analysis, and feeding data into local applications without the risk of accidental changes.

For information about how data is cached and accessed, see the Cache Behavior section.

everest mount lakefs://image-repo/main/datasets/pets/ ./pets

# Run a python script
pytorch_train.py --input ./pets

# Query data with DuckDB
duckdb "SELECT * FROM read_parquet('pets/labels.parquet')"

everest umount ./pets

Write-Mode Operations¶

By enabling write mode (--write-mode), you can modify, add, and delete files locally and then commit those changes back to the lakeFS branch. When running in write mode, the lakeFS URI must point to a branch, not a commit ID or a tag.

Example of changing data locally

1. Mount in write mode: Use the --write-mode flag to enable writes.

   everest mount lakefs://image-repo/main/datasets/pets/ ./pets --write-mode
   

2. Modify files: Make any changes you need using standard shell commands.

   # Add a new file
   echo "new data" > ./pets/birds/parrot/cute.jpg
   
   # Update an existing file
   echo "new data" >> ./pets/dogs/golden_retrievers/cute.jpg
   
   # Delete a file
   rm ./pets/cats/persian/cute.jpg
   

3. Review your changes: The diff command shows the difference between your local state and the branch's state at the time of mounting.

   everest diff ./pets
   # Output:
   # + added datasets/pets/birds/parrot/cute.jpg
   # ~ modified datasets/pets/dogs/golden_retrievers/cute.jpg
   # - removed datasets/pets/cats/persian/cute.jpg
   

4. Commit your changes: The commit command uploads your local changes and commits them to the source branch in lakeFS.

   everest commit ./pets -m "Updated pet images"
   

   After committing, your local mount will be synced to the new HEAD of the branch. Running diff again will show no changes.

5. Unmount when finished:

   everest umount ./pets
   

lakeFS Mount for Windows¶

lakeFS Mount is available for Windows, in read-only mode.

lakeFS Mount Behavior On Windows Operation System¶

CFAPI uses an OS-managed caching system optimized for cloud storage:

• Placeholder Files: Files initially appear as stubs containing only metadata, without actual content
• On-Demand Hydration: When accessed, files are "hydrated" - their content is fetched from lakeFS
• Local Cache: Subsequent reads are served directly from the local cache without reaching lakeFS
• Full Download: Currently, accessing any part of a file, triggers a full download
• Automatic Eviction: The OS "dehydrates" (clears content) under storage pressure. All data is deleted after unmount

Requirements¶

• lakeFS Mount supports the windows's native Cloud Filter API. No need in additional installations.
• CFAPI Support Starts at Windows 10, version 1709.
• Make sure your lakeFS Mount version is > 0.6.0

Skip Scan in Microsoft Protection Preferences (Windows Defender)¶

AV will try to fully scan all files contents, please make sure to must disable firewall for the mounted path. This can be done via the UI or in Powershell With an admin user:

Add-MpPreference -ExclusionPath "Path\To\Mount-Dir"

Verify exclusions:

Get-MpPreference | Select-Object -ExpandProperty ExclusionPath

Exclude Mount Directory from Windows Search Indexing¶

Windows Search automatically tries to index the newly available files recursively walking through the directory structure. For large repositories this can take a long time and impact performance, so it's recommended to exclude the mount directory from indexing.

How To Exclude Your Mount Folder (Recommended)

1. Open Settings > Privacy & security > Searching Windows
2. Under "Excluded Folders", click "Add an excluded folder"
3. Select your mount directory (e.g., C:\Users\me\mounted)
4. The indexer will stop trying to index that location

lakeFS Mount on Kubernetes (CSI Driver)¶

The lakeFS CSI (Container Storage Interface) Driver allows Kubernetes Pods to mount and interact with data in a lakeFS repository as if it were a local filesystem. The driver serves mounts from an unprivileged Pod on the node and reads lakeFS credentials per volume rather than from a single cluster-wide setting.

In this section:

• How it Works - Understanding the CSI driver architecture
• Status and Limitations - Supported platforms and current limitations
• Prerequisites - Requirements for deploying the CSI driver
• Deploy the CSI Driver - Installation instructions using Helm
• Authenticate to lakeFS - Choosing and configuring a credential source
• Use in Pods - How to mount lakeFS URIs in your Kubernetes workloads
• Troubleshooting - Common issues and debugging steps

How it Works¶

The driver has three moving parts. A controller Deployment watches for workload Pods that use a lakeFS volume and schedules a Mount Pod for them on the same node, recording which workload is attached to which Mount Pod in a cluster-scoped custom resource. A node DaemonSet implements the CSI node service, opens the kernel FUSE device, and passes the file descriptor to the Mount Pod over a Unix socket. The Mount Pod then runs everest mount-server against that descriptor as an unprivileged, non-root process in the everest-lakefs-csi namespace, and the node component bind mounts the result into each workload Pod that asked for it.

Mount Pods are shared, which keeps one FUSE process per mount on a node instead of one per workload. Two workloads share a Mount Pod only when everything about the mount matches: the node, the PersistentVolume, the volume ID, the mount options, the authentication source, and the workload's fsGroup, with the service account, namespace, and role ARN counting as well in pod authentication mode. When the last workload using a mount goes away, the controller marks the Mount Pod for a clean exit, and the node component unmounts the volume and removes the credential material it wrote.

Status and Limitations¶

• Kubernetes: Version >=1.30. On 1.30 and 1.31 the driver's custom resource installs without field selectors, which costs the controller some efficiency but works.
• Nodes: Linux nodes that expose the kernel FUSE device (/dev/fuse) and a readable kubelet path. The host distribution is not a factor, since everest runs inside the driver image rather than on the node, so Amazon Linux 2023, Bottlerocket, and the RHEL family all work.
• Architecture: linux/amd64, and linux/arm64 from chart 1.3.0 onwards.
• Provisioning: Static provisioning only.
• Access Modes: ReadOnlyMany for read-only mounts and ReadWriteMany for writable mounts.
• Writes: writes are staged in the Mount Pod and land in lakeFS only when you run everest commit against the mount directory, which uploads them to a hidden ephemeral branch, commits it, and merges the result into the source branch. Nothing is synced or committed on unmount, and the ephemeral branch is deleted when the mount shuts down, so anything not committed is lost. Staged writes occupy the Mount Pod's cache volume when cache is set and the container's writable layer otherwise, counting against cacheEmptyDirSizeLimit or the Pod's ephemeral storage budget, but never against everest's cache-size read cache.
• Security Context: Workload Pods may set their own securityContext, including runAsUser, because the driver mounts with allow_other. Files are served as root:root, so what your container sees follows its own security context rather than any ownership mount option, and the Mount Pod's security context is managed by the driver and is not configurable.

Prerequisites¶

1. lakeFS Cloud account or lakeFS Enterprise version 1.25.0 or higher, matching the lakeFS Mount prerequisites.
2. A Kubernetes cluster (>=1.30) with Helm installed and cluster admin permissions, since the chart creates a namespace, a priority class, cluster roles, and a cluster-scoped custom resource definition.
3. Network access from the cluster Pods to your lakeFS server.
4. A credential source for lakeFS, either a lakeFS access key pair or an AWS IAM identity, as described in Authenticate to lakeFS.

Deploy the CSI Driver¶

The driver is deployed using a Helm chart.

1. Add the lakeFS Helm repository:

   helm repo add lakefs https://charts.lakefs.io
   helm repo update lakefs
   

   Verify the chart is available and see the latest version:

   helm search repo lakefs/everest-lakefs-csi-driver
   

   To see all available chart versions, use the -l flag:

   helm search repo lakefs/everest-lakefs-csi-driver -l
   

2. Configure values.yaml:

   values.yaml example

   # Optional: only if you mirror the driver image to a registry that
   # requires authentication. The public image needs no credentials.
   # imagePullSecret:
   #   registry: https://index.docker.io/v1/
   #   username: <registry-user>
   #   token: <registry-token>
   
   node:
     # Logging verbosity (0-4 is normal, 5 is most verbose)
     logLevel: 4
     # Only set if your nodes use a non-standard kubelet directory
     # kubeletPath: /var/lib/kubelet
   
   mountpointPod:
     # Namespace the driver creates and runs Mount Pods in
     namespace: everest-lakefs-csi
   

3. Install the chart:

   helm install everest-lakefs-csi-driver lakefs/everest-lakefs-csi-driver \
     --namespace kube-system --version <chart-version>
   

   Add -f values.yaml to the command if you created a values file in the previous step.

4. Verify the rollout:

   kubectl rollout status daemonset/everest-lakefs-csi-node -n kube-system
   kubectl get pods -n kube-system -l app=everest-lakefs-csi-node
   

Authenticate to lakeFS¶

Each volume declares where its lakeFS credentials come from through the authenticationSource volume attribute, so different repositories can be mounted under different lakeFS identities on the same cluster. A volume that leaves the attribute unset falls back to driver.

kubectl create secret generic lakefs-creds \
  --namespace default \
  --from-literal=access_key_id=<lakefs-key-id> \
  --from-literal=secret_access_key=<lakefs-secret-key> \
  --from-literal=server_endpoint_url=https://lakefs.example.com

lakectl auth users aws-iam attach --id <lakefs-user> \
  --principal-id 'arn:aws:sts::<account>:assumed-role/<role>'

Use in Pods¶

To use the driver, you create a PersistentVolume (PV) and a PersistentVolumeClaim (PVC) that mount a lakeFS URI into your Pod.

• Static Provisioning: You must set storageClassName: "" in both the PV and the PVC. To ensure a PVC is bound to a specific PV, use a claimRef in the PV definition to create a one-to-one mapping.
• Mount URI: The lakeFSMountUri volume attribute is required and points at a repository, a ref, and an optional path, for example lakefs://my-repo/main/data/.
• Volume Handle: Each PV needs a unique volumeHandle, since Kubernetes processes a volume once per handle and a duplicate leaves your Pod stuck in ContainerCreating.
• Access Mode: ReadOnlyMany mounts read-only, and ReadWriteMany mounts read-write. Setting readOnly: true on the PV, the PVC, or the Pod's volume forces a read-only mount even when the access mode is ReadWriteMany, so leave it unset when you want writes to work.
• Mount Options: Entries under mountOptions are everest mount-server flags with the leading dashes removed, such as cache-size 1500000000 or log-level debug. The Command-Line Reference lists the full set under everest mount-server. Leave out the flags the driver sets for you: the protocol, the listen addresses, the cache directory, the credentials, and the choice between read-only and write mode. everest exits on a flag it does not recognise, and the Mount Pod then crash-loops.
• StatefulSets: Static provisioning creates no volumes on demand, so a StatefulSet needs one PersistentVolume per replica, prepared in advance. Its PVCs also outlive the StatefulSet itself and have to be deleted by hand.

Examples

The following examples demonstrate how to mount a lakeFS URI in different Kubernetes scenarios. Except for the pod-identity one, they all reference the lakefs-creds Secret created in Authenticate to lakeFS, and they all place their objects in the default namespace.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: everest-pv
spec:
  capacity:
    storage: 100Gi # Required by Kubernetes, but ignored by lakeFS Mount
  accessModes:
    - ReadOnlyMany
  storageClassName: "" # Required for static provisioning
  claimRef:
    namespace: default
    name: everest-claim
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-1 # Must be unique
    nodePublishSecretRef:
      name: lakefs-creds
      namespace: default
    volumeAttributes:
      # Replace with your lakeFS mount URI
      lakeFSMountUri: lakefs://<repo>/<ref>/<path>
      authenticationSource: secret
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: everest-claim
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: "" # Required for static provisioning
  resources:
    requests:
      storage: 5Gi # Required by Kubernetes, but ignored by lakeFS Mount
  volumeName: everest-pv
---
apiVersion: v1
kind: Pod
metadata:
  name: everest-app
  namespace: default
spec:
  containers:
    - name: app
      image: rockylinux/rockylinux
      command: ["/bin/sh", "-c", "ls /data/; tail -f /dev/null"]
      volumeMounts:
        - name: my-lakefs-data
          mountPath: /data
  volumes:
    - name: my-lakefs-data
      persistentVolumeClaim:
        claimName: everest-claim

apiVersion: v1
kind: PersistentVolume
metadata:
  name: everest-rw-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany # Writable mount
  storageClassName: ""
  claimRef:
    namespace: default
    name: everest-rw-claim
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-2 # Must be unique
    nodePublishSecretRef:
      name: lakefs-creds
      namespace: default
    volumeAttributes:
      lakeFSMountUri: lakefs://<repo>/<branch>/<path>
      authenticationSource: secret
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: everest-rw-claim
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: everest-rw-pv
---
apiVersion: v1
kind: Pod
metadata:
  name: everest-rw-app
  namespace: default
spec:
  containers:
    - name: app
      image: rockylinux/rockylinux
      command: ["/bin/sh", "-c", "echo hello > /data/hello.txt; cat /data/hello.txt; tail -f /dev/null"]
      volumeMounts:
        - name: my-lakefs-data
          mountPath: /data
  volumes:
    - name: my-lakefs-data
      persistentVolumeClaim:
        claimName: everest-rw-claim

apiVersion: v1
kind: PersistentVolume
metadata:
  name: multiple-pods-one-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  claimRef:
    namespace: default
    name: multiple-pods-one-claim
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-3 # Must be unique
    nodePublishSecretRef:
      name: lakefs-creds
      namespace: default
    volumeAttributes:
      lakeFSMountUri: lakefs://<repo>/<ref>/<path>
      authenticationSource: secret
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: multiple-pods-one-claim
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: multiple-pods-one-pv
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-pod-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: multi-pod-app
  template:
    metadata:
      labels:
        app: multi-pod-app
    spec:
      containers:
      - name: app
        image: rockylinux/rockylinux
        command: ["/bin/sh", "-c", "ls /data/; tail -f /dev/null"]
        volumeMounts:
        - name: lakefs-storage
          mountPath: /data
      volumes:
      - name: lakefs-storage
        persistentVolumeClaim:
          claimName: multiple-pods-one-claim

# PV 1
apiVersion: v1
kind: PersistentVolume
metadata:
  name: multi-mount-pv-1
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  claimRef:
    namespace: default
    name: multi-mount-claim-1
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-4 # Must be unique
    nodePublishSecretRef:
      name: lakefs-creds
      namespace: default
    volumeAttributes:
      lakeFSMountUri: lakefs://<repo>/<ref>/<path1>
      authenticationSource: secret
---
# PVC 1
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: multi-mount-claim-1
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: multi-mount-pv-1
---
# PV 2
apiVersion: v1
kind: PersistentVolume
metadata:
  name: multi-mount-pv-2
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  claimRef:
    namespace: default
    name: multi-mount-claim-2
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-5 # Must be unique
    nodePublishSecretRef:
      name: lakefs-creds
      namespace: default
    volumeAttributes:
      lakeFSMountUri: lakefs://<repo>/<ref>/<path2>
      authenticationSource: secret
---
# PVC 2
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: multi-mount-claim-2
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: multi-mount-pv-2
---
# Pod
apiVersion: v1
kind: Pod
metadata:
  name: multi-mount-pod
  namespace: default
spec:
  containers:
    - name: app
      image: rockylinux/rockylinux
      command: ["/bin/sh", "-c", "echo 'Path 1:'; ls /data1; echo 'Path 2:'; ls /data2; tail -f /dev/null"]
      volumeMounts:
        - name: lakefs-data-1
          mountPath: /data1
        - name: lakefs-data-2
          mountPath: /data2
  volumes:
    - name: lakefs-data-1
      persistentVolumeClaim:
        claimName: multi-mount-claim-1
    - name: lakefs-data-2
      persistentVolumeClaim:
        claimName: multi-mount-claim-2

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lakefs-app-sa
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/<role>
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: everest-pod-identity-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  claimRef:
    namespace: default
    name: everest-pod-identity-claim
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-7 # Must be unique
    volumeAttributes:
      lakeFSMountUri: lakefs://<repo>/<ref>/<path>
      authenticationSource: pod
      serverEndpointUrl: https://lakefs.example.com
      stsRegion: us-east-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: everest-pod-identity-claim
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
  volumeName: everest-pod-identity-pv
---
apiVersion: v1
kind: Pod
metadata:
  name: everest-pod-identity-app
  namespace: default
spec:
  serviceAccountName: lakefs-app-sa
  containers:
    - name: app
      image: rockylinux/rockylinux
      command: ["/bin/sh", "-c", "ls /data/; tail -f /dev/null"]
      volumeMounts:
        - name: my-lakefs-data
          mountPath: /data
  volumes:
    - name: my-lakefs-data
      persistentVolumeClaim:
        claimName: everest-pod-identity-claim

apiVersion: v1
kind: Pod
metadata:
  name: everest-non-root-app
  namespace: default
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 2000
    fsGroup: 2000
  containers:
    - name: app
      image: rockylinux/rockylinux
      command: ["/bin/sh", "-c", "id; ls -la /data/; tail -f /dev/null"]
      volumeMounts:
        - name: my-lakefs-data
          mountPath: /data
  volumes:
    - name: my-lakefs-data
      persistentVolumeClaim:
        claimName: everest-claim

apiVersion: v1
kind: PersistentVolume
metadata:
  name: options-demo-pv
spec:
  capacity:
    storage: 100Gi # ignored, required
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  mountOptions:
    # cap everest's read cache, below the volume limit set further down
    - cache-size 1500000000
    - log-level debug
  csi:
    driver: csi.everest.lakefs.io
    volumeHandle: everest-csi-driver-volume-6 # Must be unique
    nodePublishSecretRef:
      name: lakefs-creds
      namespace: default
    volumeAttributes:
      lakeFSMountUri: lakefs://<repo>/<ref>/<path>
      authenticationSource: secret
      # Local data cache backed by an emptyDir volume on the node
      cache: emptyDir
      cacheEmptyDirSizeLimit: 2Gi
      # Optional: back the cache with a tmpfs ramdisk instead of node storage
      # cacheEmptyDirMedium: Memory
      # Alternative: back the cache with a generic ephemeral volume instead
      # of an emptyDir. Both attributes are required with this cache type.
      # cache: ephemeral
      # cacheEphemeralStorageClassName: gp2
      # cacheEphemeralStorageResourceRequest: 2Gi
      # Optional: size the Mount Pod for heavy workloads
      # mountpointContainerResourcesRequestsMemory: 1Gi
---
# PVC and Pod definitions follow...

Troubleshooting¶

Mount failures surface on the workload Pod first, and the matching Mount Pod holds the detailed error, so work through them in that order.

1. Inspect the events and status of the workload Pod, the PV, and the PVC:

   kubectl describe pod <workload-pod> -n <namespace>
   kubectl get pv,pvc
   

2. Find the Mount Pod serving the volume and read its logs:

   kubectl get pods -n everest-lakefs-csi
   kubectl logs <mount-pod> -n everest-lakefs-csi
   

3. Check the driver components when no Mount Pod was created at all:

   kubectl logs -n kube-system -l app=everest-lakefs-csi-node
   kubectl logs -n kube-system -l app=everest-lakefs-csi-controller
   

4. Review the mount assignments the controller made, which tell you which workload was bound to which Mount Pod:

   kubectl get mountpoints3podattachments   # or: kubectl get s3pa
   

Common failures and what they mean:

• Secret is missing required key(s) means the Secret's keys are misspelled, and authenticationSource: secret requires a K8s Secret referenced via the PV's nodePublishSecretRef means the PV has no nodePublishSecretRef at all. The three keys are exactly access_key_id, secret_access_key, and server_endpoint_url.
• could not find access key returned from lakeFS means the keys are not lakeFS-issued. Generate them in the lakeFS UI under Administration → Credentials.
• A workload Pod stuck in ContainerCreating with several lakeFS volumes usually means two PVs share a volumeHandle, which Kubernetes processes only once.
• A Mount Pod stuck in Pending is normally a scheduling problem on the node, such as a taint, a node selector, or insufficient capacity, and kubectl describe pod <mount-pod> -n everest-lakefs-csi names the reason.
• driver name csi.everest.lakefs.io not found in the list of registered CSI drivers means a workload was scheduled before the driver finished registering with the kubelet on a fresh node. Taint joining nodes with csi.everest.lakefs.io/agent-not-ready:NoExecute; the driver removes the taint once it is ready.

Command-Line Reference¶

This section provides detailed documentation for all lakeFS Mount CLI commands. For conceptual information about how lakeFS Mount works, see the Core Concepts section.

everest mount¶

Mounts a lakeFS URI to a local directory.

everest mount <lakefs_uri> <mount_directory> [flags]

Tips:

• Since the server runs in the background, use --log-output /path/to/file to view logs.
• The optimal cache size is the size of the data you are going to read/write.
• To reuse the cache between restarts of the same mount, set the --cache-dir flag.
• In read-only mode, if you provide a branch or tag, lakeFS Mount will resolve and mount the HEAD commit. For a stable mount, use a specific commit ID in the URI.

Flags:

• --write-mode: Enable write mode (default: false).
• --cache-dir: Directory to cache files.
• --cache-size: Size of the local cache in bytes.
• --cache-create-provided-dir: If cache-dir is provided and does not exist, create it.
• --listen: Address for the mount server to listen on.
• --no-spawn: Do not spawn a new server; assume one is already running.
• --protocol: Protocol to use (default: nfs), for Windows use --cfapi.
• --log-level: Set logging level.
• --log-format: Set logging output format.
• --log-output: Set logging output(s).
• --presign: Use pre-signed URLs for direct object store access (default: true).

`everest umount`

Unmounts a lakeFS directory.

everest umount <mount_directory>

`everest diff` (Write Mode Only)

Shows the difference between the local mount directory and the source branch.

everest diff [mount_directory]

`everest commit` (Write Mode Only)

Commits local changes to the source lakeFS branch. The new commit is merged to the original branch using a source-wins strategy. After the commit succeeds, the mounted directory's source commit is updated to the new HEAD of the branch.

everest commit [mount_directory] -m <commit_message>

`everest mount-server` (Advanced)

Starts the mount server without performing the OS-level mount. This is intended for advanced use cases where you want to manage the server process and the OS mount command separately.

everest mount-server <remote_mount_uri> [flags]

Flags:

• --cache-dir: Directory to cache read files and metadata.
• --cache-create-provided-dir: Create the cache directory if it does not exist.
• --listen: Address to listen on.
• --protocol: Protocol to use (nfs | fuse | cfapi).
• --callback-addr: Callback address to report back to.
• --log-level: Set logging level.
• --log-format: Set logging output format.
• --log-output: Set logging output(s).
• --cache-size: Size of the local cache in bytes.
• --parallelism: Number of parallel downloads for metadata.
• --presign: Use presign for downloading.
• --write-mode: Enable write mode (default: false).
• --root: Directory to mount on the filesystem (Windows only)

Advanced Topics¶

Write Mode Limitations¶

When using write mode (--write-mode), be aware of the following limitations and modified behaviors. For more details on write mode operations, see the Write-Mode Operations section.

Unsupported Operations

• Rename: File and directory rename operations are not supported.
• Temporary Files: Temporary files are not supported.
• Hard/Symbolic Links: Hard links and symbolic links are not supported.
• POSIX File Locks: POSIX file locks (lockf) are not supported.
• POSIX Permissions: POSIX permissions are not supported. Default permissions are assigned to files and directories.

Modified Behavior

• Metadata Operations: Modifying file metadata (chmod, chown, chgrp, time attributes) results in a no-op. The file metadata will not be changed.
• Directory Removal: Calling remove on a directory is not supported. Use appropriate directory removal commands (e.g. rm -r) instead, and on the next commit, the directory will be deleted.

Functionality Limitations

• Empty Directories: Newly created empty directories will not reflect as directory markers in lakeFS.
• Path Conflicts: lakeFS allows having two path keys where one is a "directory" prefix of the other (e.g., both animals/cat.png and animals as an empty object are valid in lakeFS). However, since a filesystem cannot contain both a file and a directory with the same name, this will lead to undefined behavior depending on the filesystem type.

Integration with Git¶

It is safe to mount a lakeFS path inside a Git repository. lakeFS Mount automatically creates a virtual .gitignore file in the mount directory. This file instructs Git to ignore all mounted content except for a single file: .everest/source.

By committing the .everest/source file, which contains the lakefs:// URI, you ensure that anyone who clones your Git repository and uses lakeFS Mount will mount the exact same version of the data, making your project fully reproducible.

FAQ¶

How does data access work? Does it stream through the lakeFS server?¶

No. By default (--presign=true), lakeFS Mount uses pre-signed URLs to read and write data directly to and from the underlying object store, ensuring high performance. Metadata operations still go through the lakeFS server.

For more details, see Performance Considerations.

What happens if the lakeFS branch is updated after I mount it?¶

In read-only mode, your mount points to the commit that was at the HEAD of the branch at the time of mounting. It will not reflect subsequent commits to that branch unless you unmount and remount. In write mode, after a successful commit, the mount is updated to the new HEAD of the branch.

When are files downloaded?¶

lakeFS Mount uses a lazy fetching strategy. Files are only downloaded when their content is accessed (e.g., with cat, open, or reading in a script). Metadata-only operations like ls do not trigger downloads.

Downloaded files are cached locally for performance. See Cache Behavior for details on how caching works and how to configure it.

What are the RBAC permissions required for mounting?¶

You can use lakeFS's Role-Based Access Control to manage access.

Minimal Read-Only Permissions:

{
  "id": "MountReadOnlyPolicy",
  "statement": [
    {
      "action": ["fs:ReadObject"],
      "effect": "allow",
      "resource": "arn:lakefs:fs:::repository/<repo>/object/<prefix>/*"
    },
    {
      "action": ["fs:ListObjects", "fs:ReadCommit", "fs:ReadBranch", "fs:ReadTag", "fs:ReadRepository"],
      "effect": "allow",
      "resource": "arn:lakefs:fs:::repository/<repo>"
    },
    { "action": ["fs:ReadConfig"], "effect": "allow", "resource": "*" }
  ]
}

Minimal Write-Mode Permissions:

{
  "id": "MountWritePolicy",
  "statement": [
    {
      "action": ["fs:ReadObject", "fs:WriteObject", "fs:DeleteObject"],
      "effect": "allow",
      "resource": "arn:lakefs:fs:::repository/<repo>/object/<prefix>/*"
    },
    {
      "action": [
        "fs:ListObjects", "fs:ReadCommit", "fs:ReadBranch", "fs:ReadRepository",
        "fs:CreateCommit", "fs:CreateBranch", "fs:DeleteBranch", "fs:RevertBranch"
      ],
      "effect": "allow",
      "resource": "arn:lakefs:fs:::repository/<repo>"
    },
    { "action": ["fs:ReadConfig"], "effect": "allow", "resource": "*" }
  ]
}

Why use lakeFS Mount instead of lakectl local?¶

While both tools work with local data, they serve different needs. Use lakectl local for Git-like workflows where you need to pull and push entire directories. Use lakeFS Mount when you need immediate, on-demand access to a large repository without downloading it first, making it ideal for exploration, training ML models, or any task that benefits from lazy loading.