lakeFS Server Configuration¶

Configuring lakeFS is done using a YAML configuration file and/or environment variable. The configuration file's location can be set with the '--config' flag. If not specified, the first file found in the following order will be used:

1. ./config.yaml
2. $HOME/lakefs/config.yaml
3. /etc/lakefs/config.yaml
4. $HOME/.lakefs.yaml

Configuration items can each be controlled by an environment variable. The variable name will have a prefix of LAKEFS_, followed by the name of the configuration, replacing every . with a _.

This reference uses . to denote the nesting of values.

Reference¶

• listen_address (string : "0.0.0.0:8000") - A <host>:<port> structured string representing the address to listen on

logging¶

• logging.format (one of ["json", "text"] : "text") - Format to output log message in
• logging.level (one of ["TRACE", "DEBUG", "INFO", "WARN", "ERROR", "NONE"] : "INFO") - Logging level to output

• logging.audit_log_level (one of ["TRACE", "DEBUG", "INFO", "WARN", "ERROR", "NONE"] : "DEBUG") - Audit logs level to output.

  Note

  In case you configure this field to be lower than the main logger level, you won't be able to get the audit logs

• logging.output (string : "-") - A path or paths to write logs to. A - means the standard output, = means the standard error.

• logging.file_max_size_mb (int : 100) - Output file maximum size in megabytes.
• logging.files_keep (int : 0) - Number of log files to keep, default is all.

actions¶

• actions.enabled (bool : true) - Setting this to false will block hooks from being executed.
• actions.lua.net_http_enabled (bool : false) - Setting this to true will load the net/http package.
• actions.env.enabled (bool : true) - Environment variables accessible by hooks, disabled values evaluated to empty strings
• actions.env.prefix (string : "LAKEFSACTION_") - Access to environment variables is restricted to those with the prefix. When environment access is enabled and no prefix is provided, all variables are accessible.
• actions.network.blocked_addresses ([]string : []) - Blocks hooks (webhook, Airflow and the Lua net/http, aws, azure and databricks packages) from sending HTTP requests to these IP addresses or CIDR ranges. Enforced on every connection, including redirects, without using proxy environment variables. Empty by default, blocking nothing.

database¶

Configuration section for the lakeFS key-value store database.

• database.type (string ["postgres"|"dynamodb"|"cosmosdb"|"redis"|"local"] : ) - lakeFS database type

  Note

  The redis database type is available in lakeFS Enterprise

auth¶

• auth.login_duration (time duration : "168h") - The duration the login token is valid for.

• auth.login_max_duration (time duration : "336h") - The maximum duration user can ask for a login token.

  Deprecated

  Only logins through the external principals API could ask for a custom token duration, and lakeFS Community no longer serves that API, so the key is ignored and a warning is logged when it is set. lakeFS Enterprise documents it under auth.

auth.cache¶

• auth.cache.enabled (bool : true) - Whether to cache access credentials and user policies in-memory. Can greatly improve throughput when enabled.
• auth.cache.size (int : 1024) - How many items to store in the auth cache. Systems with a very high user count should use a larger value at the expense of ~1kb of memory per cached user.
• auth.cache.ttl (time duration : "20s") - How long to store an item in the auth cache. Using a higher value reduces load on the database, but will cause changes longer to take effect for cached users.
• auth.cache.jitter (time duration : "3s") - A random amount of time between 0 and this value is added to each item's TTL. This is done to avoid a large bulk of keys expiring at once and overwhelming the database.

auth.encrypt¶

• auth.encrypt.secret_key (string : "" - required) - A random (cryptographically safe) generated string that is used for encryption and HMAC signing.

  Warning

  It is best to keep this somewhere safe such as KMS or Hashicorp Vault, and provide it to the system at run time.

auth.api¶

auth.authentication_api¶

auth.remote_authenticator¶

auth.oidc¶

auth.cookie_auth_verification¶

auth.ui_config¶

• auth.ui_config.login_failed_message (string : "The credentials don't match.") - Custom error message displayed when authentication fails on the login form.

blockstore¶

• blockstore.type (one of ["local", "s3", "gs", "azure", "mem"] : required). Block adapter to use. This controls where the underlying data will be stored
• blockstore.default_namespace_prefix (string : ) - Use this to help your users choose a storage namespace for their repositories. If specified, the storage namespace will be filled with this default value as a prefix when creating a repository from the UI. The user may still change it to something else.
• blockstore.signing.secret_key (string : required) - A random generated string that is used for HMAC signing when using get/link physical address

graveler¶

• graveler.ensure_readable_root_namespace (bool: true) - When creating a new repository use this to verify that lakeFS has access to the root of the underlying storage namespace. Set false only if lakeFS should not have access (i.e pre-sign mode only).
• graveler.max_batch_delay (duration : 3ms) - Controls the server batching period for references store operations.
• graveler.background.rate_limit (int : 0) - Requests per seconds limit on background work performed (default: 0 - unlimited), like deleting committed staging tokens.

graveler.repository_cache¶

• graveler.repository_cache.size (int : 1000) - How many items to store in the repository cache.
• graveler.repository_cache.ttl (time duration : "5s") - How long to store an item in the repository cache.
• graveler.repository_cache.jitter (time duration : "2s") - A random amount of time between 0 and this value is added to each item's TTL.

graveler.commit_cache¶

• graveler.commit_cache.size (int : 50000) - How many items to store in the commit cache.
• graveler.commit_cache.ttl (time duration : "10m") - How long to store an item in the commit cache.
• graveler.commit_cache.jitter (time duration : "2s") - A random amount of time between 0 and this value is added to each item's TTL.

committed¶

• committed.block_storage_prefix (string : _lakefs) - Prefix for metadata file storage in each repository's storage namespace
• committed.sstable.memory.cache_size_bytes (int : 400_000_000) - maximal size of in-memory cache used for each SSTable reader.

committed.local_cache¶

An object describing the local (on-disk) cache of metadata from permanent storage.

• committed.local_cache.size_bytes (int : 1073741824) - bytes for local cache to use on disk. The cache may use more storage for short periods of time.
• committed.local_cache.dir (string, ~/lakefs/data/cache) - directory to store local cache.
• committed.local_cache.range_proportion (float : 0.9) - proportion of local cache to use for storing ranges (leaves of committed metadata storage).
• committed.local_cache.range.open_readers (int : 500) - maximal number of unused open SSTable readers to keep for ranges.
• committed.local_cache.range.num_shards (int : 30) - sharding factor for open SSTable readers for ranges. Should be at least sqrt(committed.local_cache.range.open_readers).
• committed.local_cache.metarange_proportion (float : 0.1) - proportion of local cache to use for storing metaranges (roots of committed metadata storage).
• committed.local_cache.metarange.open_readers (int : 50) - maximal number of unused open SSTable readers to keep for metaranges.
• committed.local_cache.metarange.num_shards (int : 10) - sharding factor for open SSTable readers for metaranges. Should be at least sqrt(committed.local_cache.metarange.open_readers).

committed.permanent¶

• committed.permanent.min_range_size_bytes (int : 0) - Smallest allowable range in metadata. Increase to somewhat reduce random access time on committed metadata, at the cost of increased committed metadata storage cost.
• committed.permanent.max_range_size_bytes (int : 20971520) - Largest allowable range in metadata. Should be close to the size at which fetching from remote storage becomes linear.
• committed.permanent.range_raggedness_entries (int : 50_000) - Average number of object pointers to store in each range (subject to min_range_size_bytes and max_range_size_bytes).

email¶

• email.smtp_host (string) - A string representing the URL of the SMTP host.
• email.smtp_port (int) - An integer representing the port of the SMTP service (465, 587, 993, 25 are some standard ports)
• email.use_ssl (bool : false) - Use SSL connection with SMTP host.
• email.username (string) - A string representing the username of the specific account at the SMTP. It's recommended to provide this value at runtime from a secret vault of some sort.
• email.password (string) - A string representing the password of the account. It's recommended to provide this value at runtime from a secret vault of some sort.
• email.local_name (string) - A string representing the hostname sent to the SMTP server with the HELO command. By default, "localhost" is sent.
• email.sender (string) - A string representing the email account which is set as the sender.
• email.limit_every_duration (duration : 1m) - The average time between sending emails. If zero is entered, there is no limit to the amount of emails that can be sent.
• email.burst (int: 10) - Maximal burst of emails before applying limit_every_duration. The zero value means no burst and therefore no emails can be sent.
• email.lakefs_base_url (string : "http://localhost:8000") - A string representing the base lakeFS endpoint to be directed to when emails are sent inviting users, reseting passwords etc.

gateways¶

• gateways.s3.domain_name (string : "s3.local.lakefs.io") - a FQDN representing the S3 endpoint used by S3 clients to call this server (*.s3.local.lakefs.io always resolves to 127.0.0.1, useful for local development, if using virtual-host addressing.
• gateways.s3.region (string : "us-east-1") - AWS region we're pretending to be in, it should match the region configuration used in AWS SDK clients
• gateways.s3.fallback_url (string) - If specified, requests with a non-existing repository will be forwarded to this URL. This can be useful for using lakeFS side-by-side with S3, with the URL pointing at an S3Proxy instance. When gateways.s3.resolve_tenant is enabled, only requests that resolve to the root tenant can reach the fallback, because a tenant-addressed bucket name is unambiguously a lakeFS address.
• gateways.s3.resolve_tenant (bool : false) - When enabled, the S3 gateway resolves the tenant from the bucket name: a name of the form {tenant}--{repository} addresses that repository inside that tenant, and a bare name addresses the reserved root tenant. While disabled, every bucket name is taken whole and every request operates on root. Requires lakeFS Enterprise with tenants licensed.
• gateways.s3.verify_unsupported (bool : true) - The S3 gateway errors on unsupported requests, but when disabled, defers to target-based handlers.

tls¶

• tls.enabled (bool :false) - Enable TLS listening. The listen_address will be used to serve HTTPS requests. (mainly for local development)
• tls.cert_file (string : ) - Server certificate file path used while serve HTTPS (.cert or .crt file - signed certificates).
• tls.key_file (string : ) - Server secret key file path used whie serve HTTPS (.key file - private key).

stats¶

• stats.enabled (bool : true) - Whether to periodically collect anonymous usage statistics
• stats.flush_interval (duration : 30s) - Interval used to post anonymous statistics collected
• stats.flush_size (int : 100) - A size (in records) of anonymous statistics collected in which we post

installation¶

• installation.user_name (string : ) - When specified, an initial admin user will be created when the server is first run. Works only when database.type is set to local. Requires installation.access_key_id and installation.secret_access_key.
• installation.access_key_id (string : ) - Admin's initial access key id (used once in the initial setup process)
• installation.secret_access_key (string : ) - Admin's initial secret access key (used once in the initial setup process)
• installation.allow_inter_region_storage (bool : true) - Allow storage in a different region than the one the server is running in.

usage_report¶

• usage_report.enabled (bool : true) - Store API and Gateway usage reports into key-value store.
• usage_report.flush_interval (duration : 5m) - Sets interval for flushing in-memory usage data to key-value store.

ui¶

• ui.enabled (bool: true) - Whether to serve the embedded UI from the binary

security¶

• security.audit_check_interval (duration : 24h) - Duration in which we check for security audit.

garbage collection¶

• ugc.prepare_max_file_size (int: 125829120) - Uncommitted garbage collection prepare request, limit the produced file maximum size
• ugc.prepare_interval (duraction: 1m) - Uncommitted garbage collection prepare request, limit produce time to interval

branch_lifecycle¶

See Branch Lifecycle for full feature details.

• branch_lifecycle.schedule (string : "0 * * * *") - Cron schedule for the job that applies repository branch lifecycle policies. The job is always wired; control whether it acts on a repository by setting or clearing that repository's policies.

object_lifecycle¶

See Object Lifecycle for full feature details.

• object_lifecycle.schedule (string : "0 2 * * *") - Cron schedule for the job that applies repository object lifecycle rules. The job is always wired; control whether it acts on a repository by setting or clearing that repository's rules.
• object_lifecycle.workers (int : 1) - Number of branches evaluated concurrently within a single run.

Using Environment Variables¶

All the configuration variables can be set or overridden using environment variables.

To set an environment variable, prepend LAKEFS_ to its name, convert it to upper case, and replace . with _:

For example, logging.format becomes LAKEFS_LOGGING_FORMAT, blockstore.s3.region becomes LAKEFS_BLOCKSTORE_S3_REGION, etc.

To set a value into a map[string]string type field, use the syntax key1=value1,key2=value2,...

Example Configurations¶

---
listen_address: "0.0.0.0:8000"

database:
type: "postgres"
postgres:
    connection_string: "postgres://localhost:5432/postgres?sslmode=disable"

logging:
format: text
level: DEBUG
output: "-"

auth:
encrypt:
    secret_key: "10a718b3f285d89c36e9864494cdd1507f3bc85b342df24736ea81f9a1134bcc"

blockstore:
type: local
local:
    path: "~/lakefs/dev/data"

gateways:
s3:
    region: us-east-1

---
logging:
format: json
level: WARN
output: "-"

database:
type: "dynamodb"
dynamodb:
    table_name: "kvstore"

auth:
encrypt:
    secret_key: "10a718b3f285d89c36e9864494cdd1507f3bc85b342df24736ea81f9a1134bcc"

blockstore:
type: s3
s3:
    region: us-east-1 # optional, fallback in case discover from bucket is not supported
    credentials_file: /secrets/aws/credentials
    profile: default

---
logging:
format: json
level: WARN
output: "-"

database:
type: "postgres"
postgres:
    connection_string: "postgres://user:pass@lakefs.rds.amazonaws.com:5432/postgres"

auth:
encrypt:
    secret_key: "10a718b3f285d89c36e9864494cdd1507f3bc85b342df24736ea81f9a1134bcc"

blockstore:
type: gs
gs:
    credentials_file: /secrets/lakefs-service-account.json

---
logging:
format: json
level: WARN
output: "-"

database:
type: "postgres"
postgres:
    connection_string: "postgres://user:pass@lakefs.rds.amazonaws.com:5432/postgres"

auth:
encrypt:
    secret_key: "10a718b3f285d89c36e9864494cdd1507f3bc85b342df24736ea81f9a1134bcc"

blockstore:
type: s3
s3:
    force_path_style: true
    endpoint: http://localhost:9000
    discover_bucket_region: false
    credentials:
        access_key_id: minioadmin
        secret_access_key: minioadmin

---
logging:
format: json
level: WARN
output: "-"

database:
type: "cosmosdb"
cosmosdb:
    key: "ExampleReadWriteKeyMD7nkPOWgV7d4BUjzLw=="
    endpoint: "https://lakefs-account.documents.azure.com:443/"
    database: "lakefs-db"
    container: "lakefs-container"

auth:
encrypt:
    secret_key: "10a718b3f285d89c36e9864494cdd1507f3bc85b342df24736ea81f9a1134bcc"

blockstore:
type: azure
azure:
    storage_account: exampleStorageAcount
    storage_access_key: ExampleAcessKeyMD7nkPOWgV7d4BUjzLw==