Using lakeFS with Apache Iceberg¶

Apache Iceberg tables have no built-in way to branch, review, or roll back changes, so teams end up copying tables to test a migration or promote data between environments. lakeFS solves this by exposing a spec-compliant Apache Iceberg REST Catalog, so any standard Iceberg client can create, manage, and query tables through it while every table gains Git-like version control, letting you branch, commit, merge, and roll back your data.

Because it follows the Iceberg REST specification, lakeFS is a drop-in replacement for other Iceberg catalogs such as AWS Glue, Nessie, or Hive Metastore, as well as the deprecated lakeFS HadoopCatalog. It also stays completely outside the data path, so clients read and write data files directly to the underlying object store while lakeFS versions only the table metadata.

Quick Start¶

The catalog is served at /iceberg/api on your lakeFS server, with an OAuth2 token endpoint at /iceberg/api/v1/oauth/tokens. Authenticate with your lakeFS access key and secret, in the form <access_key_id>:<secret_access_key>.

Tables are addressed as <repository>.<branch>.<namespace>.<table>, so switching branches is as simple as changing the namespace:

from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(name="my_catalog", **{
    "prefix": "lakefs",
    "uri": "https://lakefs.example.com/iceberg/api",
    "oauth2-server-uri": "https://lakefs.example.com/iceberg/api/v1/oauth/tokens",
    "credential": "AKIAlakefs12345EXAMPLE:abc/lakefs/1234567bPxRfiCYEXAMPLEKEY",
})

# List namespaces in a branch
catalog.list_namespaces(("repo", "main"))

# Query a table
catalog.list_tables("repo.main.inventory")
table = catalog.load_table("repo.main.inventory.books")
arrow_df = table.scan().to_arrow()

spark.sql.catalog.lakefs=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lakefs.type=rest
spark.sql.catalog.lakefs.prefix=lakefs
spark.sql.catalog.lakefs.uri=https://lakefs.example.com/iceberg/api
spark.sql.catalog.lakefs.oauth2-server-uri=https://lakefs.example.com/iceberg/api/v1/oauth/tokens
spark.sql.catalog.lakefs.credential=AKIAlakefs12345EXAMPLE:abc/lakefs/1234567bPxRfiCYEXAMPLEKEY

spark.sql("USE lakefs.my_repo.main.inventory")

// List available tables
spark.sql("SHOW TABLES").show()

// Query data with branch isolation
spark.sql("SELECT * FROM books").show()

// Switch to a feature branch
spark.sql("USE lakefs.my_repo.new_branch.inventory")
spark.sql("SELECT * FROM books").show()

# example: /etc/trino/catalog/lakefs.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://lakefs.example.com/iceberg/api
iceberg.rest-catalog.nested-namespace-enabled=true
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.credential=${ENV:LAKEFS_CREDENTIALS}
iceberg.rest-catalog.oauth2.server-uri=https://lakefs.example.com/iceberg/api/v1/oauth/tokens

-- List tables in the iceberg catalog
USE "repo.main.inventory"; -- <repository>.<branch or reference>.<namespace>
SHOW TABLES;

-- Query a table
SELECT * FROM books LIMIT 100;

-- Switch to a different branch
USE "repo.new_branch.inventory";
SELECT * FROM books;

LOAD iceberg;
LOAD httpfs;

CREATE SECRET lakefs_credentials (
    TYPE ICEBERG,
    CLIENT_ID 'AKIAlakefs12345EXAMPLE',
    CLIENT_SECRET 'abc/lakefs/1234567bPxRfiCYEXAMPLEKEY',
    OAUTH2_SERVER_URI 'https://lakefs.example.com/iceberg/api/v1/oauth/tokens'
);

ATTACH 'lakefs' AS main_branch (
    TYPE iceberg,
    SECRET lakefs_credentials,
    -- scope the catalog to a repository and branch (see "Relative Namespace Support" below)
    ENDPOINT 'https://lakefs.example.com/iceberg/relative_to/my-repo.main/api'
);

USE main_branch.inventory;
SELECT * FROM books;

Looking for a different engine? See the dedicated guides for Dremio, Starburst Galaxy, Amazon Athena and AWS Glue Data Catalog.

Version Control for Iceberg Tables¶

Table and namespace changes made through the catalog (creating or dropping a table, committing new snapshots) are staged on the target branch, just like any other change in lakeFS. Commit them explicitly using lakeFS tools to record them in the branch history, and note that several Iceberg operations, potentially across multiple tables, can be grouped into a single lakeFS commit:

import lakefs

repo = lakefs.repository("repo")

# Create a branch to work in isolation
branch = repo.branch("new_branch").create(source_reference="main")

# The table is immediately accessible on the new branch
table = catalog.load_table(f"repo.{branch.id}.inventory.books")

# ... modify the table using any Iceberg client ...

# Commit the staged catalog changes to lakeFS
branch.commit(message="Update inventory.books")

# Merge the changes back into main
branch.merge_into("main")

# Changes are now visible in main
main_table = catalog.load_table("repo.main.inventory.books")

Use Cases¶

• Isolated data development: Create a branch to test schema changes, migrations or backfills across multiple tables, then merge safely with conflict detection.
• Multi-environment management: Represent environments (dev, staging, prod) as branches and promote table changes between them through merges, gated by automated testing.
• Collaboration: Multiple teams work on different table features simultaneously, and review changes to data and schema using pull requests.
• Governance and recovery: A built-in commit log captures who changed what and how; fine-grained RBAC policies control access; atomic rollback reduces time-to-recover.

Namespaces and Tables¶

Namespaces in the Iceberg Catalog follow the pattern "<repository>.<branch>.<namespace>(.<namespace>...)" where:

• <repository> must be a valid lakeFS repository name.
• <branch> must be a valid lakeFS branch name.
• <namespace> components can be nested using unit separator (e.g., inventory.books).

Examples:

• my-repo.main.inventory
• my-repo.feature-branch.inventory.books

The repository and branch components must already exist in lakeFS before using them in the Iceberg catalog.

The catalog supports the standard Iceberg namespace operations: create, list and drop namespaces, and list tables within namespaces.

Relative Namespace Support¶

Some Apache Iceberg clients do not support nested namespaces.

To support those, the lakeFS REST Catalog allows specifying relative namespaces: passing a partial namespace as part of the catalog URL endpoint (commonly, <repository>.<branch>):

https://lakefs.example.com/iceberg/relative_to/<repository>.<branch>/api

All namespaces passed by the client are then resolved relative to the namespace in the URL. For example, DuckDB allows limited nesting in the form of <database>.<schema>, so a repository-and-branch-scoped endpoint lets it address tables with just <namespace>.<table>, as shown in the DuckDB example above and the DuckDB integration guide.

Namespace Restrictions¶

• Repository and branch names must follow lakeFS naming conventions.
• Namespace components cannot contain special characters except dots (.) for nesting.
• The total namespace path length must be less than 255 characters.
• Namespaces are case-sensitive.
• Empty namespace components are not allowed.

Table Operations¶

The Iceberg Catalog supports all standard Iceberg table operations:

• Create tables with schemas and partitioning.
• Update table schemas and partitioning.
• Commit changes to tables.
• Delete tables.
• List tables in namespaces.

Authentication and Authorization¶

Authentication¶

Clients authenticate using the OAuth2 token endpoint at <lakefs-endpoint>/iceberg/api/v1/oauth/tokens. To authenticate, clients must provide their lakeFS access key and secret in the format access_key:secret as the credential.

Authorization¶

The Iceberg REST Catalog defines its own set of RBAC actions and resources, described below. When using lakeFS with Iceberg, permissions must be managed through Iceberg RBAC - repository or object-level permissions alone are not sufficient. Iceberg catalog permissions supersede any related object permissions. For example, having fs:ReadObject on a repository does not grant catalog:ReadTable on tables in that repository.

For general information about lakeFS RBAC, see the RBAC documentation.

Resources¶

Namespaces in the ARN use dot-separated notation (e.g., my.namespace). Wildcards (*) are supported for any path component.

Actions¶

List Operations Filtering¶

Namespace filtering:

For ListNamespaces, use wildcards in the resource ARN to control which namespaces are visible to the user.

Example - Allow listing only namespaces starting with analytics:

{
  "statement": [
    {
      "effect": "allow",
      "action": ["catalog:ListNamespaces"],
      "resource": "arn:lakefs:catalog:::namespace/my-repo/analytics*"
    }
  ]
}

This policy allows the user to see only namespaces starting with analytics in my-repo. Other namespaces in the same repository are filtered out of the response.

Table and view filtering (using conditions):

For ListTables and ListViews, the namespace ARN alone is not enough to filter individual tables or views within that namespace. Policies can use the StringLike condition operator with the catalog:TableName or catalog:ViewName fields to control which entities are visible.

Example - Allow listing only tables matching a pattern:

{
  "statement": [
    {
      "effect": "allow",
      "action": ["catalog:ListTables"],
      "resource": "arn:lakefs:catalog:::namespace/my-repo/analytics",
      "condition": {
        "StringLike": {
          "catalog:TableName": ["prod-*", "staging-*"]
        }
      }
    }
  ]
}

This policy allows the user to see only tables starting with prod- or staging- when listing tables in the analytics namespace of my-repo. Tables that do not match the pattern are filtered out of the response.

Example - Deny listing a specific view:

{
  "statement": [
    {
      "effect": "allow",
      "action": ["catalog:ListViews"],
      "resource": "arn:lakefs:catalog:::namespace/my-repo/my.namespace"
    },
    {
      "effect": "deny",
      "action": ["catalog:ListViews"],
      "resource": "arn:lakefs:catalog:::namespace/my-repo/my.namespace",
      "condition": {
        "StringLike": {
          "catalog:ViewName": ["secret_view"]
        }
      }
    }
  ]
}

This policy allows the user to list all views in the my.namespace namespace, except for secret_view which is explicitly denied.

Example Policies¶

Read-only access to all tables and namespaces:

{
  "statement": [
    {
      "effect": "allow",
      "action": [
        "catalog:ReadTable",
        "catalog:ReadTableData"
      ],
      "resource": "arn:lakefs:catalog:::table/*/*/*"
    },
    {
      "effect": "allow",
      "action": ["catalog:ListNamespaces", "catalog:GetNamespace"],
      "resource": "arn:lakefs:catalog:::namespace/*/*"
    }
  ]
}

catalog:ReadTableData is required to vend read-only credentials for a table's data. catalog:ReadTable alone grants access to table metadata but not to the vended data credentials. See Credentials Vending for details.

Read-write access to all tables in a repository (including data credentials):

{
  "statement": [
    {
      "effect": "allow",
      "action": [
        "catalog:ReadTable",
        "catalog:CreateTable",
        "catalog:UpdateTable",
        "catalog:ReadTableData",
        "catalog:WriteTableData"
      ],
      "resource": "arn:lakefs:catalog:::table/my-repo/*/*"
    }
  ]
}

Full access to tables in a specific repository:

{
  "statement": [
    {
      "effect": "allow",
      "action": ["catalog:*"],
      "resource": "arn:lakefs:catalog:::table/my-repo/*/*"
    },
    {
      "effect": "allow",
      "action": ["catalog:*"],
      "resource": "arn:lakefs:catalog:::namespace/my-repo/*"
    }
  ]
}

Allow reading all tables except a specific one:

{
  "statement": [
    {
      "effect": "allow",
      "action": ["catalog:ReadTable"],
      "resource": "arn:lakefs:catalog:::table/my-repo/*/*"
    },
    {
      "effect": "deny",
      "action": ["catalog:ReadTable"],
      "resource": "arn:lakefs:catalog:::table/my-repo/my.namespace/secret_table"
    }
  ]
}

Credentials Vending¶

lakeFS can vend short-lived, table-scoped object-store credentials to Iceberg REST Catalog clients. This removes the need to configure Spark, Trino, PyIceberg, or any other compatible client with permanent credentials for the repository's physical storage.

This solves the case where an Iceberg client knows only its lakeFS credentials, while the REST Catalog returns physical object-store locations that the client must read from and write to.

Credential vending is currently supported for S3, and support for additional storage backends is planned.

How it works¶

The client authenticates to the catalog with its lakeFS credentials. On S3, lakeFS then:

1. Authorizes access to the table's data.
2. Calls AWS STS AssumeRole.
3. Restricts the STS session to that table's data and lakeFS-managed metadata prefixes.
4. Returns the temporary S3 credentials and their expiration to the Iceberg client.
5. The client accesses the physical S3 endpoint directly.

The permissions carried by the vended credentials depend on the caller's access. Read-only sessions allow:

s3:GetObject
s3:ListBucket

Read-write sessions additionally allow:

s3:PutObject
s3:DeleteObject
s3:AbortMultipartUpload
s3:ListMultipartUploadParts

The generated session policy restricts these actions to the table's data and metadata prefixes.

Credential vending is best-effort. When credentials cannot be vended, for example because the caller lacks the relevant table-data permission, the location is outside the repository's storage namespace, or the backend does not support vending, the catalog operation still succeeds and returns the table metadata, only without credentials. The Iceberg client handles requesting and refreshing vended credentials on its own.

Limitations¶

• Vending is currently supported only for S3, with plans to support additional backends like Azure and GCS.
• Credentials are vended for locations within the repository's storage namespace. A table registered with a foreign or external location will not receive credentials for that location.

Configuration¶

Credential vending is a server configuration, defined per storage backend. To enable it for an S3 backend, set a dedicated role_arn in the following blockstore configuration:

blockstore:
  type: s3
  s3:
    credentials_vending:
      role_arn: arn:aws:iam::123456789012:role/lakefs-iceberg-vending
      session_duration: 1h
      external_id: optional-external-id
      endpoint: https://s3.customer-facing.example.com

For multi-storage backend installations, configure credentials_vending inside each relevant S3 entry under blockstores.stores[].s3. See the lakeFS Enterprise configuration reference for all credentials_vending fields and their equivalent LAKEFS_-prefixed environment variables.

AWS prerequisites (S3)¶

Setting role_arn enables vending, but the AWS side must also be set up. Complete the following steps:

1. Create or identify the IAM role referenced by role_arn, and grant its identity policy the S3 operations that vending needs: s3:GetObject and s3:ListBucket for reads, plus s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload, and s3:ListMultipartUploadParts for writes. The inline session policy that lakeFS generates only narrows these permissions; it cannot add permissions the role does not already have.
2. Allow the lakeFS server's AWS identity to assume the role by granting it sts:AssumeRole on role_arn.
3. Configure the role's trust policy to trust the lakeFS runtime identity. If external_id is configured, the trust-policy condition must match it.
4. Keep session_duration within limits so it does not exceed the role's MaxSessionDuration or any applicable STS role-chaining limits.

Also ensure network reachability: the lakeFS server can reach STS, and Iceberg clients can reach the physical/customer-facing S3 endpoint.

lakeFS authorization¶

Credential vending relies on table-data actions that are separate from the metadata actions used for normal catalog operations. The table-data actions determine whether read-only or read-write credentials are vended, while the metadata actions govern resolving the table and its physical location:

The user still needs the normal metadata actions for the requested catalog operation, such as catalog:ReadTable, catalog:CreateTable, or catalog:UpdateTable. See Authorization for example policies that include these data actions and for upgrade guidance on existing installations.

As with other catalog permissions, table resource ARNs are branch-agnostic.

Client configuration¶

Configure your Iceberg client to request vended credentials:

spark.sql.catalog.lakefs=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lakefs.type=rest
spark.sql.catalog.lakefs.uri=https://lakefs.example.com/iceberg/api
spark.sql.catalog.lakefs.oauth2-server-uri=https://lakefs.example.com/iceberg/api/v1/oauth/tokens
spark.sql.catalog.lakefs.credential=<lakefs-access-key>:<lakefs-secret-key>
spark.sql.catalog.lakefs.prefix=lakefs
spark.sql.catalog.lakefs.header.X-Iceberg-Access-Delegation=vended-credentials
spark.sql.catalog.lakefs.io-impl=org.apache.iceberg.aws.s3.S3FileIO

catalog = RestCatalog(name="lakefs", **{
    "uri": "https://lakefs.example.com/iceberg/api",
    "prefix": "lakefs",
    "oauth2-server-uri":
        "https://lakefs.example.com/iceberg/api/v1/oauth/tokens",
    "credential": "<lakefs-access-key>:<lakefs-secret-key>",
    "header.X-Iceberg-Access-Delegation": "vended-credentials",
})

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://lakefs.example.com/iceberg/api
iceberg.rest-catalog.prefix=lakefs
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.server-uri=https://lakefs.example.com/iceberg/api/v1/oauth/tokens
iceberg.rest-catalog.oauth2.credential=<lakefs-access-key>:<lakefs-secret-key>
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true

For S3-compatible services, keep the appropriate client S3 endpoint, region, and path-style options.

Troubleshooting¶

Table Maintenance¶

Compact Data Files¶

In the lakeFS catalog, data integrity is maintained after data file compaction operations (such as RewriteDataFiles), since data files are not deleted by such operations (only new snapshot and manifest files are created).

However, to avoid unnecessary merge conflicts, we recommend marking compaction operations using snapshotProperty(), so that lakeFS can automatically resolve conflicts when merging branches with compaction commits.

Requirements:

• Use the Spark Java API (not SQL procedures) with Iceberg v1.5 or newer
• Mark compaction operations with the following property and value using snapshotProperty():

SparkActions.get(spark)
    .rewriteDataFiles(table)
    .snapshotProperty("lakefs.compaction.operation-id", "rewrite-data-files")
    .execute();

Conflict Resolution:

When merging branches, lakeFS automatically resolves conflicts if both branches have compaction commits for the same table and at most one branch has non-compaction data changes. This allows safe merging of compacted branches without losing work.

For disabling the automatic conflict resolution, set the iceberg.ignore_compaction_commits configuration flag to false (it defaults to true).

Unsupported Operations¶

The following table maintenance operations are not supported in the current version:

• Drop table with purge
• Rewrite manifests
• Expire snapshots
• Remove old metadata files
• Delete orphan files

Limitations¶

1. Table Maintenance:
   • See Table Maintenance section for details
2. Advanced Features:
   • Server-side query planning
   • Table renaming
   • Updating table's location (using Commit)
3. lakeFS Iceberg REST Catalog is currently tested to work with Amazon S3 and Google Cloud Storage. Other storage backends, such as Azure or Local storage are currently not supported, but will be in future releases.
4. The Iceberg v3 table format is currently not supported (while the most common v2 and v1 are).

Roadmap¶

The following features are planned for future releases:

1. Table Import:
   • Support for importing existing Iceberg tables from other catalogs
   • Bulk import capabilities for large-scale migrations
2. Azure Storage Support
3. Advanced Features:
   • Views API support
   • Table transactions
4. Advanced versioning capabilities
   • merge non-conflicting table updates

How It Works¶

Under the hood, the lakeFS Iceberg REST Catalog keeps track of each table's metadata file. This is typically referred to as the table pointer.

This pointer is stored inside the repository's storage namespace.

When a request is made, the catalog would examine the table's fully qualified namespace: <repository>.<reference>.<namespace>.<table_name> to read that special pointer file from the given reference specified, and returns the underlying object store location of the metadata file to the client. When a table is created or updated, lakeFS would make sure to generate a new metadata file inside the storage namespace, and register that metadata file as the current pointer for the requested branch. These changes are staged in lakeFS, and may be committed to the branch.

This approach builds on Iceberg's existing metadata and the immutability of its snapshots: a commit in lakeFS captures a metadata file, which in turn captures manifest lists, manifest files and all related data files.

Besides simply avoiding "double booking" where both Iceberg and lakeFS would need to keep track of which files belong to which version, it also greatly improves the scalability and compatibility of the catalog with the existing Iceberg tool ecosystem.

Example: Reading an Iceberg Table¶

Here's a simplified example of what reading from an Iceberg table would look like:

sequenceDiagram
    Actor Iceberg Client
    participant lakeFS Catalog API
    participant lakeFS
    participant Object Store

    Iceberg Client->>lakeFS Catalog API: get table metadata("repo.branch.table")
    lakeFS Catalog API->>lakeFS: get('repo', 'branch', 'table')
    lakeFS->>lakeFS Catalog API: physical_address
    lakeFS Catalog API->>Iceberg Client: object location ("s3://.../metadata.json")
    Iceberg Client->>Object Store: GetObject
    Object Store->>Iceberg Client: table data

Example: Writing an Iceberg Table¶

Here's a simplified example of what writing to an Iceberg table would look like:

sequenceDiagram
    Actor Iceberg Client
    participant lakeFS Catalog API
    participant lakeFS
    participant Object Store
    Iceberg Client->>Object Store: write table data
    Iceberg Client->>lakeFS Catalog API: Iceberg commit
    lakeFS Catalog API->>lakeFS: stage('new table pointer')
    lakeFS->>Object Store: PutObject("metdata.json")
    lakeFS Catalog API->>Iceberg Client: Iceberg commit done
    Iceberg Client->>lakeFS: lakeFS commit('repo','branch', message)

Related Resources¶

Catalog Sync¶

Catalog Sync allows you to synchronize Iceberg tables between the lakeFS catalog and external catalogs such as AWS Glue Data Catalog or other Iceberg REST-compatible catalogs. This enables workflows where some users or tools need to access data through other external catalogs.

Use Cases¶

Collaboration with External Tools: Share data with users who rely on tools that only support specific catalogs. For example, data engineers can work with tables in lakeFS while data analysts query the same data through AWS Glue using Athena, all while maintaining isolation and version control.

sequenceDiagram
    actor Alice (Data Engineer)
    actor Bob (Data Analyst)
    participant lakeFS
    participant AWS Glue
    Alice (Data Engineer)->>lakeFS: Update tables on branch 'dev'
    Alice (Data Engineer)->>lakeFS: Push tables to Glue
    lakeFS->>AWS Glue: Register tables in 'dev' database
    Alice (Data Engineer)->>Bob (Data Analyst): Please review: glue/dev
    Bob (Data Analyst)->>AWS Glue: SELECT * FROM dev.table (via Athena)
    Bob (Data Analyst)->>Alice (Data Engineer): Approved!
    Alice (Data Engineer)->>lakeFS: Merge 'dev' into 'main'

Isolated Pipelines: Run data pipelines using tools that require external catalogs while maintaining isolation through lakeFS branches. Create a branch, push tables to an external catalog, run your pipeline, pull the changes back, and merge into main.

sequenceDiagram
    participant Orchestrator
    participant lakeFS
    participant External Catalog
    participant Pipeline Tool
    Orchestrator->>lakeFS: Create branch 'etl-2024-01-15'
    Orchestrator->>lakeFS: Push tables to external catalog
    lakeFS->>External Catalog: Register tables
    Orchestrator->>Pipeline Tool: Run ETL pipeline
    Pipeline Tool->>External Catalog: Read/write tables
    Orchestrator->>lakeFS: Pull updated tables
    lakeFS->>External Catalog: Read updated metadata
    Orchestrator->>lakeFS: Merge 'etl-2024-01-15' into 'main'

Configuration¶

Remote catalogs are configured in your lakeFS configuration file. Each remote catalog requires a unique identifier and type-specific connection properties.

AWS Glue Data Catalog¶

Configure an AWS Glue catalog by specifying the region and AWS credentials:

iceberg_catalog:
  remotes:
    - id: aws_glue_us_east_1
      type: glue
      glue:
        region: us-east-1
        access_key_id: <your-glue-key>
        secret_access_key: <your-glue-secret>

Iceberg REST Catalog¶

Configure a generic Iceberg REST catalog with basic authentication:

iceberg_catalog:
  remotes:
    - id: remote_catalog
      type: rest
      rest:
        uri: https://catalog.example.com/iceberg/api
        credential: <client-id>:<client-secret>

Or with OAuth2 client credentials flow:

iceberg_catalog:
  remotes:
    - id: remote_catalog
      type: rest
      rest:
        uri: https://catalog.example.com/iceberg/api
        credential: <client-id>:<client-secret>
        oauth_server_uri: https://auth.example.com/oauth/tokens
        oauth_scope: catalog:read catalog:write

Push to remote¶

Push operations register a table from lakeFS into a remote catalog. The table's metadata and data remain in lakeFS-managed storage, which are used as the pushed table's location.

API Endpoint: POST /iceberg/remotes/{catalog-id}/push

Parameters: - source: The lakeFS table location (repository, branch/reference, namespace, table name) - destination: The remote catalog location (namespace, table name) - force_update: (optional, default: false) Override the table if it already exists in the remote catalog - create_namespace: (optional, default: false) Create the namespace in the remote catalog if it doesn't exist

Example:

curl -X POST "https://lakefs.example.com/iceberg/remotes/aws_glue_us_east_1/push" \
  -H "Authorization: Bearer $LAKEFS_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "repository_id": "my-repo",
      "reference_id": "main",
      "namespace": ["default", "features"],
      "table": "image_properties"
    },
    "destination": {
      "namespace": ["main_features"],
      "table": "image_properties"
    },
    "create_namespace": true,
    "force_update": true
  }'

This example pushes the table my-repo.main.default.features.image_properties from lakeFS to the AWS Glue catalog as main_features.image_properties. It creates the remote namespace if needed (since create_namespace: true), and overwrites any existing table or possible recent updates committed to it (since force_update: true).

Pull from remote¶

Pull operations update a lakeFS table with changes from a remote catalog. This is useful after external tools have modified a table previously pushed from lakeFS.

API Endpoint: POST /iceberg/remotes/{catalog-id}/pull

Parameters: - source: The remote catalog location (namespace, table name) - destination: The lakeFS table location (repository, branch/reference, namespace, table name) - force_update: (optional, default: false) Override the table in lakeFS if metadata conflicts exist - create_namespace: (optional, default: false) Create the namespace in lakeFS if it doesn't exist

Example:

curl -X POST "https://lakefs.example.com/iceberg/remotes/aws_glue_us_east_1/pull" \
  -H "Authorization: Bearer $LAKEFS_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "namespace": ["main_features"],
      "table": "image_properties"
    },
    "destination": {
      "repository_id": "my-repo",
      "reference_id": "main",
      "namespace": ["default", "features"],
      "table": "image_properties"
    }
    "create_namespace": true,
    "force_update": true
  }'

This example pulls changes from the Glue table main_features.image_properties back into the lakeFS table my-repo.main.default.features.image_properties. It creates the namespace in lakeFS if needed (since create_namespace: true), and overwrites any existing table or possible recent updates committed to it (since force_update: true).

Important Notes¶

1. Storage Location: Pulled tables' metadata.json file must reside in a storage location to which lakeFS has read access to, or the pull operation will fail.

2. Atomicity: Push and pull operations are not atomic. If an operation fails partway through, manual intervention may be required (contact support in such a case if needed).

3. Authentication: Ensure the credentials configured for remote catalogs have appropriate permissions:

   • For AWS Glue: glue:CreateTable, glue:UpdateTable, glue:GetTable, glue:CreateDatabase (if create_namespace is used)
   • For REST catalogs: Appropriate OAuth scopes for table and namespace operations

4. Namespace Format: Namespaces are represented as arrays of strings to support nested namespaces (e.g., ["accounting", "tax"] represents accounting.tax).

Deprecated: Iceberg HadoopCatalog¶

Setup

<dependency>
<groupId>io.lakefs</groupId>
<artifactId>lakefs-iceberg</artifactId>
<version>0.1.4</version>
</dependency>

.config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.3.0,io.lakefs:lakefs-iceberg:0.1.4")

Configuration

.config("spark.sql.catalog.lakefs", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.lakefs.catalog-impl", "io.lakefs.iceberg.LakeFSCatalog") \
.config("spark.sql.catalog.lakefs.warehouse", f"lakefs://{repo_name}") \ 
.config("spark.sql.catalog.lakefs.cache-enabled", "false")

.config("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.endpoint", "https://example-org.us-east-1.lakefscloud.io") \
.config("spark.hadoop.fs.s3a.access.key", "AKIAIO5FODNN7EXAMPLE") \
.config("spark.hadoop.fs.s3a.secret.key", "wJalrXUtnFEMI/K3MDENG/bPxRfiCYEXAMPLEKEY") \
.config("spark.hadoop.fs.s3a.path.style.access", "true")

spark-shell --conf spark.sql.catalog.lakefs="org.apache.iceberg.spark.SparkCatalog" \
    --conf spark.sql.catalog.lakefs.catalog-impl="io.lakefs.iceberg.LakeFSCatalog" \
    --conf spark.sql.catalog.lakefs.warehouse="lakefs://example-repo" \
    --conf spark.sql.catalog.lakefs.cache-enabled="false" \
    --conf spark.hadoop.fs.s3.impl="org.apache.hadoop.fs.s3a.S3AFileSystem" \
    --conf spark.hadoop.fs.s3a.endpoint="https://example-org.us-east-1.lakefscloud.io" \
    --conf spark.hadoop.fs.s3a.access.key="AKIAIO5FODNN7EXAMPLE" \
    --conf spark.hadoop.fs.s3a.secret.key="wJalrXUtnFEMI/K3MDENG/bPxRfiCYEXAMPLEKEY" \
    --conf spark.hadoop.fs.s3a.path.style.access="true"

Using Iceberg tables with HadoopCatalog

Create a table

To create a table on your main branch, use the following syntax:

CREATE TABLE lakefs.main.db1.table1 (id int, data string);

Insert data into the table

INSERT INTO lakefs.main.db1.table1 VALUES (1, 'data1');
INSERT INTO lakefs.main.db1.table1 VALUES (2, 'data2');

Create a branch

We can now commit the creation of the table to the main branch:

lakectl commit lakefs://example-repo/main -m "my first iceberg commit"

Then, create a branch:

lakectl branch create lakefs://example-repo/dev -s lakefs://example-repo/main

Make changes on the branch

We can now make changes on the branch:

INSERT INTO lakefs.dev.db1.table1 VALUES (3, 'data3');

Query the table

If we query the table on the branch, we will see the data we inserted:

SELECT * FROM lakefs.dev.db1.table1;

Results in:

+----+------+
| id | data |
+----+------+
| 1  | data1|
| 2  | data2|
| 3  | data3|
+----+------+

However, if we query the table on the main branch, we will not see the new changes:

SELECT * FROM lakefs.main.db1.table1;

Results in:

+----+------+
| id | data |
+----+------+
| 1  | data1|
| 2  | data2|
+----+------+

Migrating an existing Iceberg table to the Hadoop Catalog

This is done through an incremental copy from the original table into lakeFS.

1. Create a new lakeFS repository lakectl repo create lakefs://example-repo <base storage path>

2. Initiate a spark session that can interact with the source iceberg table and the target lakeFS catalog. Here's an example of Hadoop and S3 session and lakeFS catalog with per-bucket config:

   SparkConf conf = new SparkConf();
   conf.set("spark.hadoop.fs.s3a.path.style.access", "true");
   
   // set hadoop on S3 config (source tables we want to copy) for spark
   conf.set("spark.sql.catalog.hadoop_prod", "org.apache.iceberg.spark.SparkCatalog");
   conf.set("spark.sql.catalog.hadoop_prod.type", "hadoop");
   conf.set("spark.sql.catalog.hadoop_prod.warehouse", "s3a://my-bucket/warehouse/hadoop/");
   conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions");
   conf.set("spark.hadoop.fs.s3a.bucket.my-bucket.access.key", "<AWS_ACCESS_KEY>");
   conf.set("spark.hadoop.fs.s3a.bucket.my-bucket.secret.key", "<AWS_SECRET_KEY>");
   
   // set lakeFS config (target catalog and repository)
   conf.set("spark.sql.catalog.lakefs", "org.apache.iceberg.spark.SparkCatalog");
   conf.set("spark.sql.catalog.lakefs.catalog-impl", "io.lakefs.iceberg.LakeFSCatalog");
   conf.set("spark.sql.catalog.lakefs.warehouse", "lakefs://example-repo");
   conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions");
   conf.set("spark.hadoop.fs.s3a.bucket.example-repo.access.key", "<LAKEFS_ACCESS_KEY>");
   conf.set("spark.hadoop.fs.s3a.bucket.example-repo.secret.key", "<LAKEFS_SECRET_KEY>");
   conf.set("spark.hadoop.fs.s3a.bucket.example-repo.endpoint"  , "<LAKEFS_ENDPOINT>");
   

   3. Create Schema in lakeFS and copy the data

   Example of copy with spark-sql:

   -- Create Iceberg Schema in lakeFS
   CREATE SCHEMA IF NOT EXISTS <lakefs-catalog>.<branch>.<db>
   -- Create new iceberg table in lakeFS from the source table (pre-lakeFS)
   CREATE TABLE IF NOT EXISTS <lakefs-catalog>.<branch>.<db> USING iceberg AS SELECT * FROM <iceberg-original-table>