Using lakeFS with Dremio¶

Dremio is a next-generation data lake engine that liberates your data with live, interactive queries directly on cloud data lake storage, including S3 and lakeFS.

Iceberg REST Catalog¶

The recommended way to use lakeFS with Dremio is through the lakeFS Iceberg REST Catalog. lakeFS stays completely outside the data path, so data is read and written by Dremio executors directly to the underlying object store, while lakeFS versions the table metadata.

Configuration¶

To configure Dremio to work with the Iceberg REST Catalog, you need to configure the Iceberg REST Catalog in Dremio.

1. On the Datasets page, to the right of Sources in the left panel, click +

2. In the Add Data Source dialog, under Lakehouse Catalogs, select Iceberg REST Catalog Source. The New Iceberg REST Catalog Source dialog box appears, which contains the following tabs:

   1. In General →
      • Enter a name for your Iceberg REST Catalog source, specify the endpoint URI (i.e. https://lakefs.example.com/iceberg/api)
      • Uncheck "Use vended credentials"

   2. In Advanced Options → Catalog Properties, add the following key-value pairs (left = key, right = value):

      Key	Value	Notes
      oauth2-server-uri	https://lakefs.example.com/iceberg/api/v1/oauth/tokens	Your lakeFS OAuth2 token endpoint (not the catalog URL).
      credential	<lakefs_access_key>:<lakefs_secret_key>	Your lakeFS credentials.
      fs.s3a.aws.credentials.provider	org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider	Use static AWS credentials.
      fs.s3a.access.key	<aws_access_key_id>	AWS key with read/write access to your data bucket.
      fs.s3a.secret.key	<aws_secret_access_key>	AWS secret key.
      dremio.s3.list.all.buckets	false	Avoid listing all buckets during initialization.

3. Click Save to create the Iceberg REST Catalog source.

Data Bucket Permissions¶

The lakeFS Iceberg Catalog manages table metadata, while Dremio reads and writes data files directly from your underlying storage (for example, Amazon S3).

You must ensure that the IAM role or user Dremio uses has read/write access to your data bucket. The following AWS IAM policy provides the required permissions for direct access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DremioIcebergAccess",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<lakefs_repo_storage_namespace>/_managed/", 
                "arn:aws:s3:::<lakefs_repo_storage_namespace>/_managed/*"
            ]
        },
        {
            "Sid": "BucketLevelRequiredForDremio",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<lakefs_repo_storage_namespace_bucket_name>"
        }
    ]
}