Publish to MSK Kafka using AWS Lambda Python

Apache Kafka is a distributed streaming platform designed to handle real-time data streams efficiently. It uses a publish-subscribe model, where producers publish messages to topics, and consumers subscribe to those topics to receive the messages. Amazon MSK simplifies the deployment and management of Kafka clusters on AWS, providing a fully managed and scalable solution. You can use AWS lambda in python to publish to MSK topic.

Dependencies

First we need to build a lambda layer which will install dependencies for the lambda function:

Bash

Open the AWS Management Console and navigate to the AWS Lambda service.

Create a new layer: Click on "Layers" in the left navigation menu and then click the "Create layer" button.

Configure the layer: Provide a name for your layer, and optionally, a description and compatible runtimes. Select the appropriate runtime based on the code files included in your layer.

Upload the zip file: In the "Code entry type" section, select "Upload a file from Amazon S3" or "Upload a file from your computer", depending on where your zip file is located. If it's on your local machine, choose the latter option. Then, click the "Upload" button and select your zip file.

Create the layer: Once the zip file is uploaded, click the "Create" button to create the Lambda layer.

Add the layer to your Lambda function: After the layer is created, you can add it to your Lambda function. Open the function's configuration, scroll down to the "Layers" section, and click on "Add a layer". Select the layer you created from the list and click "Add".

Kafka Producer

We will write a producer to publish to MSK topic. Environment variables passed are:

kafka_topic. Producers write data to a specific topic, and consumers read from one or more topics.
Kafka broker. Brokers receive messages from producers and serve them to consumers. Multiple brokers form a Kafka cluster, and they collaborate to provide fault tolerance, scalability, and high throughput.
sasl username. SASL (Simple Authentication and Security Layer) is a framework used for authentication and data security in distributed systems.
sasl password. Secret credential associated with a SASL username.

Python

Kafka Consumer

Similarly, we can write a consumer to subscribe to the topic:

Python