Link Search Menu Expand Document

Authenticate to lakeFS with AWS IAM Roles

lakeFS Enterprise

External principals API is available for lakeFS Enterprise. If you’re using the open-source version you can check the pluggable APIs.

Table of contents

  1. Overview
  2. Server Configuration
  3. Administration of IAM Roles in lakeFS
  4. Get lakeFS API Token

Overview

lakeFS supports authenticating users programmatically using AWS IAM roles instead of using static lakeFS access and secret keys. The method enables you to bound IAM principal ARNs to lakeFS users. A single lakeFS user may have many AWS’s principle ARNs attached to it. When a client is authenticating to a lakeFS server with an AWS’s session, the actions performed by the client are on behalf of the user attached to the ARN.

Using Session Names

The bound ARN can be attached to a single lakeFS user with or without SessionName, serving different users. For example, consider the following mapping:

Principal ARN lakeFS User
arn:aws:sts::123456:assumed-role/Dev foo
arn:aws:sts::123456:assumed-role/Dev/john@acme.com john

if the bound ARN were arn:aws:sts::123456:assumed-role/Dev/<SessionName> it would allow any principal assuming Dev role in AWS account 123456 to login to it. If the SessionName is john@acme.com then lakeFS would return token for john user

How AWS authentication works

The AWS STS API includes a method, sts:GetCallerIdentity, which allows you to validate the identity of a client. The client signs a GetCallerIdentity query using the AWS Signature v4 algorithm and sends it to the lakeFS server.

The GetCallerIdentity query consists of four pieces of information: the request URL, the request body, the request headers and the request method. The AWS signature is computed over those fields. The lakeFS server reconstructs the query using this information and forwards it on to the AWS STS service. Depending on the response from the STS service, the server authenticates the client.

Notably, clients don’t need network-level access themselves to talk to the AWS STS API endpoint; they merely need access to the credentials to sign the request. However, it means that the lakeFS server does need network-level access to send requests to the STS endpoint.

Each signed AWS request includes the current timestamp to mitigate the risk of replay attacks. In addition, lakeFS allows you to require an additional header, X-LakeFS-Server-ID (added by default), to be present to mitigate against different types of replay attacks (such as a signed GetCallerIdentity request stolen from a dev lakeFS instance and used to authenticate to a prod lakeFS instance).

It’s also important to note that Amazon does NOT appear to include any sort of authorization around calls to GetCallerIdentity. For example, if you have an IAM policy on your credential that requires all access to be MFA authenticated, non-MFA authenticated credentials will still be able to authenticate to lakeFS using this method.

Server Configuration

Note: lakeFS Helm chart supports the configuration since version 1.2.11 - see usage values.yaml example.

  • in lakeFS auth.authentication_api.external_principals_enabled must be set to true in the configuration file, other configuration (auth.authentication_api.*) can be found at at configuration reference

fluffy server configuration reference:

  • auth.external.aws_auth.enabled (bool : false) - If true, external principals API will be enabled, e.g auth service and login api’s.
  • auth.external.aws_auth.get_caller_identity_max_age (duration : 15m) - The maximum age in seconds for the GetCallerIdentity request to be valid, the max is 15 minutes enforced by AWS, smaller TTL can be set.
  • auth.authentication_api.external_principals_enabled (bool : false) - If true, external principals API will be enabled, e.g auth service and login api’s.
  • auth.external.aws_auth.valid_sts_hosts ([]string) - The default are all the valid AWS STS hosts (sts.amazonaws.com, sts.us-east-2.amazonaws.com etc).
  • auth.external.aws_auth.required_headers (map[string]string : ) - Headers that must be present by the client when doing login request (e.g X-LakeFS-Server-ID: <lakefs.ingress.domain>).
  • auth.external.aws_auth.optional_headers (map[string]string : ) - Optional headers that can be present by the client when doing login request.
  • auth.external.aws_auth.http_client.timeout (duration : 10s) - The timeout for the HTTP client used to communicate with AWS STS.
  • auth.external.aws_auth.http_client.skip_verify (bool : false) - Skip SSL verification with AWS STS.

By default lakeFS clients will add the parameter X-LakeFS-Server-ID: <lakefs.ingress.domain> to the initial login request for STS.

Example configuration with required headers:

Configuration for lakefs.yaml:

auth:
  authentication_api:
    endpoint: http://<fluffy-sso>/api/v1
    external_principals_enabled: true
  api:
    endpoint: http://<fluffy-rbac>/api/v1

Configuration for fluffy.yaml:

# fluffy address for lakefs auth.authentication_api.endpoint
# used by lakeFS to login and get the token
listen_address: <fluffy-sso>
auth:
  # fluffy address for lakeFS auth.api.endpoint 
  # used by lakeFS to manage the lifecycle attach/detach of the external principals
  serve_listen_address: <fluffy-rbac>
  external:
    aws_auth:
      enabled: true
      # headers that must be present by the client when doing login request
      required_headers:
        # same host as the lakeFS server ingress
        X-LakeFS-Server-ID: <lakefs.ingress.domain>

Administration of IAM Roles in lakeFS

Administration referes to the management of the IAM roles that are allowed to authenticate to lakeFS. Operations such as attaching and detaching IAM roles to a user, listing the roles attached to a user, and listing the users attached to a role. Currently this is done through the lakeFS External Principals API and generated clients.

Example of attaching an IAM roles to a user:

import lakefs_sdk as lakefs  

configuration = lakefs.Configuration(host = "...",username="...",password="...")
username = "<lakefs-user>"
api = lakefs.ApiClient(configuration)
auth_api = lakefs.AuthApi(api)
# attach the role(s)to a lakeFS user
auth_api.create_user_external_principal(user_id=username, principal_id='arn:aws:sts::<id>:assumed-role/<role A>/<optional session name>')
auth_api.create_user_external_principal(user_id=username, principal_id='arn:aws:sts::<id>:assumed-role/<role B>')
# list the roles attached to the user
resp = auth_api.list_user_external_principals(user_id=args.user)
for p in resp.results:
    # do something

Get lakeFS API Token

The login to lakeFS is done by calling the login API with the GetCallerIdentity request signed by the client. Currently, the login operation is supported out of the box in lakeFS Hadoop FileSystem version 0.2.4, see Spark usage. Other clients (i.e HTTP, Python etc) can use the login endpoint to authenticate to lakeFS but, you will have to build the request input.