life is too short for a diary

Posts Tagged: aws

Empty s3 bucket and Delete using Jenkins and AWS CLI

Amazon Simple Storage Service (s3) is a scalable object storage service provided by AWS. It allows users to store and retrieve vast amount of data, make it a fundamental building block for many applications hosted on AWS. Deleting a s3 bucket is a two step process: empty a s3 bucket and then delete a s3 bucket...

Continue reading → jenkins aws

Publish to MSK Kafka using AWS Lambda Python

Apache Kafka is a distributed streaming platform designed to handle real-time data streams efficiently. It uses a publish-subscribe model, where producers publish messages to topics, and consumers subscribe to those topics to receive the messages. Amazon MSK simplifies the deployment and management of Kafka clusters on AWS, providing a fully managed and scalable solution. You can use AWS lambda in python to publish to MSK topic...

Continue reading → kafka aws python

Fix SonarQube Error of Public Writable Directories in AWS lambda python

In AWS Lambda functions, it's common to use temporary directories for tasks like generating temporary files, storing intermediate data, or processing data securely. However, improperly managing publicly writable directories can lead to security vulnerabilities. In this article, we'll explore how to address this issue and ensure safe usage of temporary directories in AWS Lambda functions using Python's tempfile module...

Continue reading → sonarqube python aws

Unit Test AWS Lambda Function in Python

AWS lambda function are event driven serverless code. To follow TDD, we should write unit test our lambda functions...

Continue reading → aws python

AWS CLI to stop running instances of statemachine

I could not delete a statemachine that had running instances stuck in Progress. Since the number of instances were humongous, it was impossible to manually delete them using UI...

Continue reading → aws statemachine

Resolve the Error CROSSSLOT Keys in request don't hash to the same slot

AWS offers managed service Amazon MemoryDB which is redis compatible. The entire keyspace in Redis cluster is divided into hash slots and these slots are assigned to multiple nodes. In redis, getting a single value is straightforward...

Continue reading → python redis memorydb aws

Cross Region AWS Glue Data Catalog access with Glue ETL

AWS Glue job are fully managed ETL service. Glue job uses hive-compatible metastore called Glue Data catalog. ..

Continue reading → aws glue

Create a Lambda Function to run SQL queries in Redshift Cluster

Amazon Redshift is fully managed warehouse service provided by AWS. The Amazon Redshift engine is a SQL-compliant, massively-parallel, query processing and database management system designed to support analytics workload...

Continue reading → python sql lambda function aws redshift

Creating Python AWS lambda layer with Docker

When you develop AWS lambda functions, you might feel the need to install additional `python` libraries. This can be achieved using Lambda layers that can be included in any lambda function...

Continue reading → docker aws python

Automate creation of AWS Stack

AWS CDK is a great framework to programmatically deploy cloudformation stack. If you are unfamiliar with AWS CDK, I would recommend first to check out [Getting started with AWS CDK](

Continue reading → aws cdk cloudformation typescript tutorial

Getting Started with AWS CDK

Cloudformation service in AWS allows you to describe an entire set of resources required to make a pipeline. The cloudformation template can be described in JSON or YAML format. Updating the cloudformation template was not a pleasant experience. I wanted to discover alternative ways to generate cloudformation template programmatically...

Continue reading → aws cdk cloudformation typescript python

Export DynamoDB Table to S3 Bucket Using Lambda Function

Dynamodb is a great NoSQL service by AWS. One of the most common use case is to export data to s3. There are multiple ways to export data to s3. For this tutorial we will leverage lambda function to achieve our goal...

Continue reading → aws lambda function dynamodb s3