Link Search Menu Expand Document

Using lakeFS with Kafka

Apache Kafka provides a unified, high-throughput, low-latency platform for handling real-time data feeds.

Different distributions of Kafka have different methods for exporting data to s3, called Kafka Sink Connectors.

Most commonly used for S3 is Confluent’s S3 Sink Connector.

Add the following to connector.properties file for lakeFS support:

# Your lakeFS repository
s3.bucket.name=example-repo

# Your lakeFS S3 endpoint and credentials
store.url=https://s3.lakefs.example.com
aws.access.key.id=AKIAIOSFODNN7EXAMPLE
aws.secret.access.key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# master being the branch we want to write to
topics.dir=master/topics