Link Search Menu Expand Document

Using lakeFS with Dremio

Dremio is a next-generation data lake engine that liberates your data with live, interactive queries directly on cloud data lake storage, including S3 and lakeFS.

Configuration

Starting from version 3.2.3, Dremio supports Minio as an experimental S3-compatible plugin. Similarly, you can connect lakeFS with Dremio.

Suppose you already have both lakeFS and Dremio deployed, and want to use Dremio to query your data in the lakeFS repositories. You can follow the steps listed below to configure on Dremio UI:

  1. click Add Data Lake.
  2. Under File Stores, choose Amazon S3.
  3. Under Advanced Options, check Enable compatibility mode (experimental).
  4. Under Advanced Options > Connection Properties, add fs.s3a.path.style.access and set the value to true.
  5. Under Advanced Options > Connection Properties, add fs.s3a.endpoint and set lakeFS S3 endpoint to the value.
  6. Under the General tab, specify the access_key_id and secret_access_key provided by lakeFS server.
  7. Click Save, and now you should be able to browse lakeFS repositories on Dremio.