#Lakehouse architecture code#

You can dynamically scale EMR clustersto meet the varying resource needs of big data pipelines that can handle up to petabytes of data. Additionally, AWS Glue provides triggers and workflow capabilities that you can use to build a multi-step end-to-end data processing pipeline that includes job dependencies as well as running parallel steps. AWS Glue ETL provides step-by-step processing of partitioned data.

AWS Glue ETL jobs can reference both tables hosted on Amazon Redshift and Amazon S3 by accessing them through the common Lake Formation catalog (which the AWS Glue crawler populates by collecting data from Amazon S3 as well as Amazon Redshift). In the same workflow, AWS Glue can load and process Amazon Redshift data stored in flat table format as well as datasets stored on S3 data lake hosted using popular open source formats like CSV, JSON, Parquet and Avro. AWS Glue provides built-in capabilities for processing data stored in Amazon Redshift as well as the S3 data lake.

#Lakehouse architecture code#

To speed ETL development, AWS Glue automatically generates ETL code and provides commonly used data structures and ETL transformation (to validate, clean, transform, and flatten data). These same jobs can store processed data sets back to the S3 data lake, Amazon Redshift data warehouse, or both in the Lakehouse storage layer.ĪWS Glue provides serverless, pay-per-use, per-use ETL to enable ETL pipelines to handle tens of terabytes of data, without the need to manage servers or clusters. These jobs can use open source connectors as well as Spark native to access and combine relational data stored in Amazon Redshift with complex flat or hierarchical structured data stored in Amazon S3. Big data processingįor integrated processing of large volumes of semi-structured, unstructured, or highly structured data stored on the Lakehouse storage layer (Amazon S3 and Amazon Redshift), you can build big data jobs using Apache Spark and run them on AWS Glue or Amazon EMR. As a final step, data processing pipelines can insert curated, enriched, and modeled data into Amazon Redshift internal tables or external tables stored in Amazon S3. Data is stored in data lake using open source file formats such as JSON, Avro, Parquet and ORCįor data enrichment steps, these pipelines may include SQL statements that join internal dimension tables to large fact tables stored in data lake S3 (using the Redshift Spectrum class).Flat structured data delivered directly to Amazon Redshift staging tables by AWS DMS or Amazon AppFlow.Similar stored procedure-based ELT pipelines on Amazon Redshift can transform the following: These ELT pipelines can use batch parallel processing (MPP) in Amazon Redshift and the ability in Redshift Spectrum to create thousands of temporary nodes to scale processing to petabytes of data. To transform structured data in the Lakehouse storage layer, you can build powerful ELT pipelines using familiar SQL semantics. Near-real-time streaming data processing using Spark Streaming on Amazon EMR.Near-real-time streaming data processing using Spark Streaming on AWS Glue.Near-real-time streaming data processing using Amazon Kinesis.Big data processing using AWS Glue or Amazon EMR.SQL-based ELT using Amazon Redshift (with Redshift Spectrum).You can use purpose-built components to build data transformation pipelines: You can add metadata from the resulting datasets to a central Lake Formation catalog using the AWS Glue data collector or the Lake Formation API. Amazon Redshift SQL (with Redshift Spectrum).We can use process layer components to build data processing jobs that can read and write data stored in both data warehouse and data lake storage using the following interfaces: Each component can read and write data to both Amazon S3 and Amazon Redshift (collectively, storage of Lakehouse architecture). The processing layer of the Lakehouse Architecture provides a variety of purpose-built components that enable a wide variety of data processing and use cases, in a single structure (flat, hierarchical, or unstructured tabular) architecture) and the speed (batch or streaming) of the dataset in Lakehouse. See also part 2 on the Ingestion Layer & Storage Layer.

See also part 1 on the Lakehouse Approach. Part 2 went into depth about the Ingestion Layer and Storage Layer, part 3 will clarify the remaining two important layers, completing the overview of the Lakehouse Reference Architecture. Following part 1, the following section will introduce a reference architecture that uses AWS services to create each layer described in the Lakehouse architecture.