Using lakeFS with AWS CLI

The AWS Command Line Interface (CLI) is a unified tool for managing your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

You can use the file commands for S3 to access lakeFS.

Table of contents

1. Configuration
2. Path convention
3. Usage
4. Examples
   1. List directory
   2. Copy from lakeFS to lakeFS
   3. Copy from lakeFS to a local path
   4. Copy from a local path to lakeFS
   5. Delete file
   6. Delete directory
5. Adding an alias

Configuration

You would like to configure an AWS profile for lakeFS.

To configure the lakeFS credentials, run:

aws configure --profile lakefs

You will be prompted to enter AWS Access Key ID and AWS Secret Access Key.

It should look like this:

aws configure --profile lakefs
# output:  
# AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE    
# AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# Default region name [None]: 
# Default output format [None]:

Path convention

When accessing objects in S3, you will need to use the lakeFS path convention: s3://[REPOSITORY]/[BRANCH]/PATH/TO/OBJECT

Usage

After configuring the credentials, this is what a command should look:

aws s3 --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  ls s3://example-repo/main/example-directory

You can use an alias to make it shorter and more convenient.

Examples

List directory

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 ls s3://example-repo/main/example-directory

Copy from lakeFS to lakeFS

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 cp s3://example-repo/main/example-file-1 s3://example-repo/main/example-file-2

Copy from lakeFS to a local path

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 cp s3://example-repo/main/example-file-1 /path/to/local/file

Copy from a local path to lakeFS

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 cp /path/to/local/file s3://example-repo/main/example-file-1

Delete file

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 rm s3://example-repo/main/example-directory/example-file

Delete directory

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 rm s3://example-repo/main/example-directory/ --recursive

Adding an alias

To make the command shorter and more convenient, you can create an alias:

alias awslfs='aws --endpoint https://lakefs.example.com --profile lakefs'

Now, the ls command using the alias will be as follows:

awslfs s3 ls s3://example-repo/main/example-directory