Upgrading lakeFS¶
This guide covers upgrading a self-managed lakeFS deployment. The database migration guidance applies to both lakeFS Community and lakeFS Enterprise, while Upgrading lakeFS Enterprise covers the Enterprise-specific steps, including migrating from the deprecated Fluffy service. If you run the hosted deployment, upgrades are handled for you.
Upgrading lakeFS from a previous version usually just requires re-deploying with the latest image, or downloading the latest version if you are using the binary. Before upgrading, check whether the release requires a migration.
When DB migrations are required¶
lakeFS 0.103.0 or greater¶
Version 0.103.0 added support for rolling KV upgrade. This means that users who already migrated to the KV ref-store (versions 0.80.0 and above) no longer have to pass through specific versions for migration.
This includes ACL migration which was introduced in lakeFS version 0.98.0.
Running lakefs migrate up on the latest lakeFS version will perform all the necessary migrations up to that point.
lakeFS 0.80.0 or greater (KV Migration)¶
Starting with version 0.80.2, lakeFS has transitioned from using a PostgreSQL based database implementation to a Key-Value datastore interface supporting multiple database implementations. More information can be found here. Users upgrading from a previous version of lakeFS must pass through the KV migration version (0.80.2) before upgrading to newer versions of lakeFS.
Important
Pre Migrate Requirements:
- Users using OS environment variables for database configuration must define the
connection_stringexplicitly or as environment variable before proceeding with the migration. - Database storage free capacity of at least twice the amount of the currently used capacity
- It is strongly recommended to perform these additional steps:
- Commit all uncommitted data on branches
- Create a snapshot of your database
- By default, old database tables are not being deleted by the migration process, and should be removed manually after a successful migration. To enable table drop as part of the migration, set the
database.drop_tablesconfiguration param totrue
Migration Steps¶
For each lakeFS instance currently running with the database
-
Modify the
databasesection under lakeFS configuration yaml:- Add
typefield with"postgres"as value - Copy the current configuration parameters to a new section called
postgres
- Add
-
Stop all lakeFS instances
- Using the
lakefsbinary for the new version (0.80.2), run the following: - lakeFS will run the migration process, which in the end should display the following message with no errors:
- It is now possible to remove the old database configuration. The updated configuration should look as such:
- Deploy (or run) the new version of lakeFS.
lakeFS 0.30.0 or greater¶
In case migration is required, you first need to stop the running lakeFS service.
Using the lakefs binary for the new version, run the following:
Deploy (or run) the new version of lakeFS.
Note that an older version of lakeFS cannot run on a migrated database.
Prior to lakeFS 0.30.0¶
Note
with lakeFS < 0.30.0, you should first upgrade to 0.30.0 following this guide. Then, proceed to upgrade to the newest version.
Starting version 0.30.0, lakeFS handles your committed metadata in a new way, which is more robust and has better performance. To move your existing data, you will need to run the following upgrade commands.
Verify lakeFS version == 0.30.0 (can skip if using Docker)
Migrate data from the previous format:
Or migrate using Docker image:
docker run --rm -it -e LAKEFS_DATABASE_CONNECTION_STRING=<database connection string> treeverse/lakefs:rocks-migrate migrate db
Once migrated, it is possible to now use more recent lakeFS versions. Please refer to their release notes for more information on ugrading and usage).
If you want to start over, discarding your existing data, you need to explicitly state this in your lakeFS configuration file. To do so, add the following to your configuration (relevant only for 0.30.0):
Data Migration for Version v0.50.0¶
Warning
If you are using a version before 0.50.0, you must first perform the previous upgrade to that version.
Upgrading lakeFS Enterprise¶
lakeFS Enterprise consolidates all enterprise functionality into a single binary. If you are upgrading an existing Enterprise deployment, the database migration steps above apply as they do for Community. The most significant Enterprise-specific change is the removal of the separate Fluffy service, described below.
Migrate From Fluffy to lakeFS Enterprise¶
The new lakeFS Enterprise integrates all enterprise features directly into a single binary, eliminating the need for the separate Fluffy service. This simplifies deployment, configuration, and maintenance.
Prerequisites¶
- You're using lakeFS enterprise binary or the image in Dockerhub treeverse/lakefs-enterprise with fluffy.
- Your lakeFS-Enterprise version is >= 1.63.0
- You possess a lakeFS Enterprise license.
Note
Contact us to gain access to lakeFS Enterprise. You will be granted a token that enables downloading dockerhub/lakeFS-Enterprise from Docker Hub, and a license to run lakeFS Enterprise.
To migrate from fluffy to lakeFS Enterprise, follow the steps below:
- Sanity Test (Optional): Install a new test lakeFS Enterprise before moving your current production setup. Make sure to include your lakeFS Enterprise license in the configuration before setup. Test the setup → login → create repository, etc. Once everything seems to work, delete and cleanup the test setup and we will move to the migration process.
- Update configuration: Unlike lakeFS + Fluffy, lakeFS Enterprise uses only one configuration file. See Configuration Changes, make sure to add the license to the configuration.
- Spin down lakeFS and fluffy, and run lakeFS Enterprise!
Warning
Please note that there will be a short downtime while replacing the lakeFS instances.
Configuration Changes¶
Authentication configuration¶
Most Fluffy auth.* settings migrate directly to lakeFS Enterprise with the same structure. Below are the differences between the configurations.
SAML
# fluffy.yaml
auth:
logout_redirect_url: https://lakefs.company.com
post_login_redirect_url: https://lakefs.company.com
saml:
enabled: true
sp_root_url: https://lakefs.company.com
sp_x509_key_path: dummy_saml_rsa.key
sp_x509_cert_path: dummy_saml_rsa.cert
sp_sign_request: true
sp_signature_method: http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
idp_metadata_url: https://my.saml-provider.com/federationmetadata/2007-06/federationmetadata.xml
# idp_authn_name_id_format: "urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified"
external_user_id_claim_name: samName
# idp_metadata_file_path:
# idp_skip_verify_tls_cert: true
# lakefs.yaml
auth:
logout_redirect_url: https://lakefs.company.com
cookie_auth_verification:
auth_source: saml
friendly_name_claim_name: displayName
persist_friendly_name: true
external_user_id_claim_name: samName
validate_id_token_claims:
department: r_n_d
default_initial_groups:
- "Developers"
ui_config:
login_url: https://lakefs.company.com/sso/login-saml
logout_url: https://lakefs.company.com/sso/logout-saml
login_cookie_names:
- internal_auth_session
- saml_auth_session
# lakefs.yaml
auth:
logout_redirect_url: https://lakefs.company.com/ # optional, URL to redirect to after logout
cookie_auth_verification:
auth_source: saml
friendly_name_claim_name: displayName
default_initial_groups: ["Admins"]
external_user_id_claim_name: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name
validate_id_token_claims:
department: r_n_d
providers:
saml:
# enabled: true # This field was dropped!
sp_root_url: https://lakefs.company.com
sp_x509_key_path: dummy_saml_rsa.key
sp_x509_cert_path: dummy_saml_rsa.cert
sp_sign_request: true
sp_signature_method: http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
idp_metadata_url: https://my.saml-provider.com/federationmetadata/2007-06/federationmetadata.xml
post_login_redirect_url: / # Where to redirect after successful SAML login
# external_user_id_claim_name: # This field was moved to auth.cookie_auth_verification
ui_config:
login_url: https://lakefs.company.com/sso/login-saml
logout_url: https://lakefs.company.com/sso/logout-saml
login_cookie_names:
- internal_auth_session
- saml_auth_session
OIDC + OIDC STS
# fluffy.yaml
auth:
post_login_redirect_url: /
logout_redirect_url: https://oidc-provider-url.com/logout/url
oidc:
enabled: true
url: https://oidc-provider-url.com/
client_id: <oidc-client-id>
client_secret: <oidc-client-secret>
callback_base_url: https://lakefs.company.com
is_default_login: true
logout_client_id_query_parameter: client_id
logout_endpoint_query_parameters:
- returnTo
- https://lakefs.company.com/oidc/login
# lakefs.yaml
auth:
logout_redirect_url: https://oidc-provider-url.com/logout/url # optional, URL to redirect to after logout
ui_config:
login_url: /oidc/login
logout_url: /oidc/logout
login_cookie_names:
- internal_auth_session
- oidc_auth_session
oidc:
friendly_name_claim_name: "nickname"
default_initial_groups: ["Admins"]
providers:
oidc:
# enabled: true # This field was dropped!
post_login_redirect_url: / # This field was moved here!
url: https://oidc-provider-url.com/
client_id: <oidc-client-id>
client_secret: <oidc-client-secret>
callback_base_url: https://lakefs.company.com
logout_client_id_query_parameter: client_id
logout_endpoint_query_parameters:
- returnTo
- http://lakefs.company.com/oidc/login
LDAP
# fluffy.yaml
auth:
post_login_redirect_url: /
ldap:
server_endpoint: ldaps://ldap.company.com:636
bind_dn: uid=<bind-user-name>,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com
bind_password: '<ldap password>'
username_attribute: uid
user_base_dn: ou=<some-ou>,o=<org-id>,dc=<company>,dc=com
user_filter: (objectClass=inetOrgPerson)
connection_timeout_seconds: 15
request_timeout_seconds: 7
# lakefs.yaml
auth:
ui_config:
logout_url: /logout
login_cookie_names:
- internal_auth_session
providers:
ldap:
server_endpoint: ldaps://ldap.company.com:636
bind_dn: uid=<bind-user-name>,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com
bind_password: '<ldap password>'
username_attribute: uid
user_base_dn: ou=<some-ou>,o=<org-id>,dc=<company>,dc=com
user_filter: (objectClass=inetOrgPerson)
connection_timeout_seconds: 15
request_timeout_seconds: 7
default_user_group: "Developers" # This field moved here!
AWS IAM
Authorization configuration¶
RBAC
Kubernetes: Migrating with Helm from Fluffy to new lakeFS Enterprise¶
Overview¶
Starting with lakeFS Helm chart version 1.5.0, the Fluffy authentication service has been deprecated and replaced with native lakeFS Enterprise authentication. This migration consolidates authentication into the main lakeFS application, simplifying deployment and maintenance.
What's Changing¶
When you upgrade to lakeFS Enterprise:
- Fluffy Deployment Removed: The separate Fluffy deployment, service, and associated Kubernetes resources are no longer needed
- Simplified Architecture: Authentication is now handled directly by lakeFS Enterprise, reducing the number of pods and services
- Streamlined Ingress: No more routing between Fluffy and lakeFS - all traffic goes directly to lakeFS
- Updated values.yaml Structure: Authentication configuration moves from
fluffy.*toenterprise.auth.*andlakefsConfig.auth.providers.*
Prerequisites¶
- Current lakeFS deployment using Fluffy authentication (chart version < 1.5.0)
- Access to update Helm values
- lakeFS Enterprise Docker Hub token
- Backup of your current values.yaml
Step-by-Step Migration Guide¶
Step 1: Update Helm Repository¶
Verify you have access to chart version 1.5.0 or later:
Step 2: Review New Chart Values¶
Examine all available configuration options in the new chart:
Step 3: Update Your Image Configuration¶
If you're overriding the image in your values.yaml, update it to use lakeFS Enterprise:
image:
repository: treeverse/lakefs-enterprise
tag: 1.63.0
privateRegistry:
enabled: true
secretToken: <your-dockerhub-token>
Note: If you're not overriding the image, the chart will automatically use the correct Enterprise image.
Step 3.5: License Configuration¶
Note
You can proceed without this step at the moment. However, licensing enforcement will be introduced soon. Contact support to receive your installation license.
lakeFS Enterprise requires a valid license to work.
In the helm chart the license is provided as a JWT token either from an existing secret or explicitly.
The following demonstrates how to configure the license in the values file:
Step 4: Migrate Your Authentication Configuration¶
Using the configuration examples below, update your values.yaml file:
1. Remove all fluffy.* configuration sections
2. Add the new enterprise.auth.* configuration for your authentication method
3. Move authentication settings to lakefsConfig.auth.providers.*
Refer to the complete examples in the lakeFS Helm chart repository.
Step 5: Validate with Dry Run¶
Before applying changes, validate your configuration:
helm upgrade <release-name> lakefs/lakefs \
--version 1.5.0 \
--namespace <namespace> \
--values <your-updated-values.yaml> \
--dry-run
Review the output to ensure: - No Fluffy resources are being created - lakeFS Enterprise deployment is configured correctly - Ingress configuration is simplified
Step 6: Perform the Upgrade¶
Once validated, perform the actual upgrade:
helm upgrade <release-name> lakefs/lakefs \
--version 1.5.0 \
--namespace <namespace> \
--values <your-updated-values.yaml>
Step 7: Verify the Migration¶
After the upgrade completes:
-
Check Pod Status:
-
Verify lakeFS Health:
-
Check Logs:
-
Test Authentication:
- Navigate to your lakeFS URL
- Verify SSO login works correctly
- Confirm RBAC permissions are preserved
-
Verify Fluffy Resources Removed:
Step 8: Rollback (if needed)¶
If you encounter issues, rollback to the previous version:
# Find the previous revision
helm history <release-name> -n <namespace>
# Rollback to previous revision
helm rollback <release-name> <previous-revision> -n <namespace>
Configuration Examples¶
Below are complete configuration examples for each authentication method, showing both the old (Fluffy) and new (Enterprise) configurations:
OIDC with Helm¶
OIDC with Helm
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
fluffy:
enabled: true
image:
privateRegistry:
enabled: true
secretToken: <dockerhub-token-fluffy-image>
fluffyConfig: |
auth:
logout_redirect_url: https://oidc-provider-url.com/logout/example
oidc:
enabled: true
url: https://oidc-provider-url.com/
client_id: <oidc-client-id>
callback_base_url: https://<lakefs.ingress.domain>
# the claim name that represents the client identifier in the OIDC provider (e.g Okta)
logout_client_id_query_parameter: client_id
# the query parameters that will be used to redirect the user to the OIDC provider (e.g Okta) after logout
logout_endpoint_query_parameters:
- returnTo
- https://<lakefs.ingress.domain>/oidc/login
secrets:
create: true
sso:
enabled: true
oidc:
enabled: true
# secret given by the OIDC provider (e.g auth0, Okta, etc)
client_secret: <oidc-client-secret>
rbac:
enabled: true
lakefsConfig: |
database:
type: local
blockstore:
type: local
auth:
ui_config:
login_cookie_names:
- internal_auth_session
- oidc_auth_session
oidc:
friendly_name_claim_name: <some-oidc-provider-claim-name>
default_initial_groups: ["Developers"]
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
enterprise:
enabled: true
auth:
oidc:
enabled: true
# secret given by the OIDC provider (e.g auth0, Okta, etc)
client_secret: <oidc-client-secret>
image:
privateRegistry:
enabled: true
secretToken: <dockerhub-token>
lakefsConfig: |
blockstore:
type: local
auth:
logout_redirect_url: https://oidc-provider-url.com/logout/example
oidc:
friendly_name_claim_name: <some-oidc-provider-claim-name>
default_initial_groups: ["Developers"]
providers:
oidc:
post_login_redirect_url: /
url: https://oidc-provider-url.com/
client_id: <oidc-client-id>
callback_base_url: https://<lakefs.ingress.domain>
# the claim name that represents the client identifier in the OIDC provider (e.g Okta)
logout_client_id_query_parameter: client_id
# the query parameters that will be used to redirect the user to the OIDC provider (e.g Okta) after logout
logout_endpoint_query_parameters:
- returnTo
- https://<lakefs.ingress.domain>/oidc/login
SAML with Helm¶
SAML with Helm
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
fluffy:
enabled: true
image:
privateRegistry:
enabled: true
secretToken: <dockerhub-token-fluffy-image>
fluffyConfig: |
auth:
# logout_redirect_url: https://<lakefs.ingress.domain>
# post_login_redirect_url: https://<lakefs.ingress.domain>
saml:
sp_sign_request: true
# depends on IDP
sp_signature_method: "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"
# url to the metadata of the IDP
idp_metadata_url: "https://<adfs-auth.company.com>/federationmetadata/2007-06/federationmetadata.xml"
# IDP SAML claims format default unspecified
# idp_authn_name_id_format: "urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified"
# claim name from IDP to use as the unique user name
external_user_id_claim_name: samName
# depending on IDP setup, if CA certs are self signed and not trusted by a known CA
idp_skip_verify_tls_cert: true
rbac:
enabled: true
secrets:
create: true
sso:
enabled: true
saml:
enabled: true
createSecret: true
lakeFSServiceProviderIngress: https://<lakefs.ingress.domain>
certificate:
saml_rsa_public_cert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
saml_rsa_private_key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
lakefsConfig: |
blockstore:
type: local
auth:
cookie_auth_verification:
# claim name to display user in the UI
friendly_name_claim_name: displayName
# claim name from IDP to use as the unique user name
external_user_id_claim_name: samName
default_initial_groups:
- "Developers"
ui_config:
login_cookie_names:
- internal_auth_session
- saml_auth_session
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
enterprise:
enabled: true
auth:
saml:
enabled: true
createCertificateSecret: true
certificate:
samlRsaPublicCert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
samlRsaPrivateKey: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
image:
privateRegistry:
enabled: true
secretToken: <dockerhub-token>
lakefsConfig: |
blockstore:
type: local
auth:
logout_redirect_url: https://<lakefs.ingress.domain>
cookie_auth_verification:
auth_source: saml
# claim name to display user in the UI
friendly_name_claim_name: displayName
# claim name from IDP to use as the unique user name
external_user_id_claim_name: samName
default_initial_groups:
- "Developers"
providers:
saml:
post_login_redirect_url: https://<lakefs.ingress.domain>
sp_root_url: https://<lakefs.ingress.domain>
sp_sign_request: true
# depends on IDP
sp_signature_method: "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"
# url to the metadata of the IDP
idp_metadata_url: "https://<adfs-auth.company.com>/federationmetadata/2007-06/federationmetadata.xml"
# IDP SAML claims format default unspecified
idp_authn_name_id_format: "urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified"
# depending on IDP setup, if CA certs are self signed and not trusted by a known CA
#idp_skip_verify_tls_cert: true
LDAP with Helm¶
LDAP with Helm
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
fluffy:
enabled: true
image:
privateRegistry:
enabled: true
secretToken: <dockerhub-token-fluffy-image>
fluffyConfig: |
auth:
post_login_redirect_url: /
ldap:
server_endpoint: ldaps://ldap.company.com:636
bind_dn: uid=<bind-user-name>,ou=Users,o=<org-id>,dc=<company>,dc=com
username_attribute: uid
user_base_dn: ou=Users,o=<org-id>,dc=<company>,dc=com
user_filter: (objectClass=inetOrgPerson)
connection_timeout_seconds: 15
request_timeout_seconds: 7
secrets:
create: true
sso:
enabled: true
ldap:
enabled: true
bind_password: <ldap bind password>
rbac:
enabled: true
lakefsConfig: |
blockstore:
type: local
auth:
remote_authenticator:
enabled: true
default_user_group: "Developers"
ui_config:
login_cookie_names:
- internal_auth_session
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
enterprise:
enabled: true
auth:
ldap:
enabled: true
bindPassword: <ldap bind password>
image:
privateRegistry:
enabled: true
secretToken: <dockerhub-token>
lakefsConfig: |
blockstore:
type: local
auth:
ui_config:
login_cookie_names:
- internal_auth_session
providers:
ldap:
server_endpoint: ldaps://ldap.company.com:636
bind_dn: uid=<bind-user-name>,ou=Users,o=<org-id>,dc=<company>,dc=com
username_attribute: uid
user_base_dn: ou=Users,o=<org-id>,dc=<company>,dc=com
user_filter: (objectClass=inetOrgPerson)
default_user_group: "Developers"
connection_timeout_seconds: 15
request_timeout_seconds: 7
AWS IAM with Helm¶
AWS IAM with Helm
lakefsConfig: |
auth:
authentication_api:
external_principals_enabled: true
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
fluffy:
enabled: true
image:
repository: treeverse/fluffy
pullPolicy: IfNotPresent
privateRegistry:
enabled: true
secretToken: <dockerhub-token-fluffy-image>
fluffyConfig: |
auth:
external:
aws_auth:
enabled: true
# the maximum age in seconds for the GetCallerIdentity request
#get_caller_identity_max_age: 60
# headers that must be present by the client when doing login request
required_headers:
# same host as the lakeFS server ingress
X-LakeFS-Server-ID: <lakefs.ingress.domain>
secrets:
create: true
sso:
enabled: true
rbac:
enabled: true
ingress:
enabled: true
ingressClassName: <class-name>
hosts:
# the ingress that will be created for lakeFS
- host: <lakefs.ingress.domain>
paths:
- /
lakefsConfig: |
auth:
external_aws_auth:
enabled: true
# the maximum age in seconds for the GetCallerIdentity request
#get_caller_identity_max_age: 60
# headers that must be present by the client when doing login request
required_headers:
# same host as the lakeFS server ingress
X-LakeFS-Server-ID: <lakefs.ingress.domain>
Important Notes¶
- Complete configuration examples for each authentication method are available in the lakeFS Helm chart repository
- The examples include local blockstore for quick-start - replace with S3/Azure/GCS for production deployments
- Configure the
image.privateRegistry.secretTokenwith your DockerHub token for accessing enterprise images - Update all placeholder values (marked with
<>) with your actual configuration
Troubleshooting¶
If you encounter issues during migration:
- Authentication Failures: Check that all authentication settings have been properly moved to the new configuration structure
- Image Pull Errors: Ensure your DockerHub token has access to the lakeFS Enterprise image
- Ingress Issues: Confirm that your ingress is pointing directly to lakeFS (not Fluffy)
For additional support, consult the lakeFS documentation or contact lakeFS support.