life is too short for a diary




Cross Region AWS Glue Data Catalog access with Glue ETL

Tags: aws glue

AWS Glue job are fully managed ETL service. Glue job uses hive-compatible metastore called Glue Data catalog.

So one way to create dynamic frame in AWS Glue (Pyspark) is

This Glue job reads Glue database defined in same region. As if now, there's no way to access cross-region database in Glue catalog.

However, if the s3 bucket is global, you can directly create dataframe from s3 bucket.


comments powered by Disqus