Using lakeFS with AWS CLI

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

We could use the file commands for S3 to access lakeFS

Table of contents

1. Configuration
2. Path convention
3. Usage
4. Examples
   1. List directory
   2. Copy from lakeFS to lakeFS
   3. Copy from lakeFS to a local path
   4. Copy from a local path to lakeFS
   5. Delete file
   6. Delete directory
5. Adding an alias

Configuration

We would like to configure an AWS profile for lakeFS.

In order to configure the lakeFS credentials run:

aws configure --profile lakefs

we will be prompted to enter AWS Access Key ID , AWS Secret Access Key

It should look like this:

aws configure --profile lakefs
# output:  
# AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE    
# AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# Default region name [None]: 
# Default output format [None]:

Path convention

When accessing objects in s3 we will need to use the lakeFS path convention s3://[REPOSITORY]/[BRANCH]/PATH/TO/OBJECT

Usage

After configuring the credentials, This is how a command should look:

aws s3 --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  ls s3://example-repo/main/example-directory

We could use an alias to make it shorter and more convenient.

Examples

List directory

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 ls s3://example-repo/main/example-directory

Copy from lakeFS to lakeFS

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 cp s3://example-repo/main/example-file-1 s3://example-repo/main/example-file-2

Copy from lakeFS to a local path

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 cp s3://example-repo/main/example-file-1 /path/to/local/file

Copy from a local path to lakeFS

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 cp /path/to/local/file s3://example-repo/main/example-file-1

Delete file

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 rm s3://example-repo/main/example-directory/example-file

Delete directory

aws --profile lakefs \
  --endpoint-url https://lakefs.example.com \
  s3 rm s3://example-repo/main/example-directory/ --recursive

Adding an alias

In order to make the command shorter and more convenient we can create an alias:

alias awslfs='aws --endpoint https://lakefs.example.com --profile lakefs'

Now, the ls command using the alias will be:

awslfs s3 ls s3://example-repo/main/example-directory