Configuring lakeFS to use presigned URLs¶
With lakeFS, you can access data directly from the storage and not through lakeFS using a presigned URL. Based on the user's access to an object in the object store, the presigned URL will get read or write access.
The presign support is enabled for block adapter that supports it (AWS, GCP, Azure), and can be disabled by the configuration (blockstore.<blockstore_type>.disable_pre_signed). Note that the UI support is disabled by default.
It is possible to override the default pre-signed URL endpoint in AWS by setting the configuration (blockstore.s3.pre_signed_endpoint).
This is useful, for example, when you wish to define a VPC endpoint access for the pre-signed URL.
Using presigned URLs in the UI¶
To use presigned URLs in the UI:
- Enable the presigned URL support UI in the lakeFS configuration:
setblockstore.<blockstore_type>.disable_pre_signed_ui=false. - Add CORS (Cross-Origin Resource Sharing) permissions to the bucket for the UI to fetch objects using a presigned URL (instead of through lakeFS).
Warning
Currently DuckDB fetching data from lakeFS does not support fetching data using presigned URL.
Examples¶
Client-side multipart upload on S3¶
Available in lakeFS Team and lakeFS Enterprise from v1.107.0, on S3 only. Start a free trial or contact us.
With client-side multipart upload, the client sends every request of a large upload directly to S3, including the requests that create, complete and abort it. lakeFS only signs those requests and records the finished object on the branch.
lakectl fs upload and lakectl local commit use it automatically for files of 5 MiB or more when presigned URLs are in use and the server supports it. Otherwise they fall back to other upload methods.
It is enabled by default on the S3 blockstore. To turn it off, set blockstore.s3.disable_pre_signed_multipart in the configuration.