Using lakeFS with Dremio
Dremio is a next-generation data lake engine that liberates your data with live, interactive queries directly on cloud data lake storage, including S3 and lakeFS.
Configuration
Starting from version 3.2.3, Dremio supports Minio as an experimental S3-compatible plugin. Similarly, you can connect lakeFS with Dremio.
Suppose you already have both lakeFS and Dremio deployed, and want to use Dremio to query your data in the lakeFS repositories. You can follow the steps listed below to configure on Dremio UI:
- click
Add Data Lake
. - Under
File Stores
, chooseAmazon S3
. - Under
Advanced Options
, checkEnable compatibility mode (experimental)
. - Under
Advanced Options
>Connection Properties
, addfs.s3a.path.style.access
and set the value to true. - Under
Advanced Options
>Connection Properties
, addfs.s3a.endpoint
and set lakeFS S3 endpoint to the value. - Under the
General
tab, specify theaccess_key_id
andsecret_access_key
provided by lakeFS server. - Click
Save
, and now you should be able to browse lakeFS repositories on Dremio.