AWS Glue Architecture — A Deep Dive into the Internal Components

Understanding the AWS Glue architecture is essential for designing performant, cost-effective ETL pipelines. This document breaks down every component, how they interconnect, and what role each plays in the data integration lifecycle.

High-Level Architecture Flow

The overall data flow through AWS Glue follows this pattern:

graph LR
    A["Data Sources<br/>(S3, JDBC, DynamoDB)"] -->|Scan| B["AWS Glue Crawlers"]
    B -->|Register Metadata| C["AWS Glue Data Catalog"]
    C -->|Read Schema| D["AWS Glue ETL Jobs<br/>(Apache Spark)"]
    D -->|Write Transformed Data| E["Target Data Stores<br/>(S3, Redshift, RDS)"]
    F["Triggers & Workflows"] -->|Orchestrate| B
    F -->|Orchestrate| D
    G["Job Bookmarks"] -->|Track State| D

Let's explore each building block in detail.

1. The AWS Glue Data Catalog

The Data Catalog is the heart of AWS Glue. It is a fully managed, Apache Hive Metastore-compatible metadata repository that stores structural and operational metadata for all your data assets.

graph TD
    DC["AWS Glue Data Catalog"] --> DB["Databases<br/>Logical grouping containers"]
    DC --> CN["Connections<br/>JDBC creds & URLs"]
    DB --> TB["Tables<br/>Metadata pointers to data"]
    TB --> SC["Schemas<br/>Column names & types"]
    TB --> PT["Partitions<br/>Sub-divisions for pruning"]

What Does the Data Catalog Store?

Key Properties of the Data Catalog

• Hive Metastore Compatible: The Data Catalog is a drop-in replacement for the Apache Hive Metastore. Any tool that speaks the Hive Metastore Protocol (like Athena, EMR, Redshift Spectrum) can natively query it.
• Schema Registry: Supports schema versioning and evolution, so you can track how your table schemas change over time.
• Cross-Account Access: The Data Catalog can be shared across multiple AWS accounts using AWS Lake Formation or IAM resource policies.
• Single Source of Truth: All AWS analytics services (Athena, EMR, Redshift Spectrum, Glue ETL) read from the same catalog, ensuring consistency.

Data Catalog Hierarchy

AWS Glue Data Catalog
│
├── Database: raw_events_db
│   ├── Table: user_clickstream
│   │   ├── Schema: [user_id: string, event: string, ts: timestamp]
│   │   ├── Location: s3://data-lake/raw/clickstream/
│   │   └── Partitions:
│   │       ├── year=2025/month=12/
│   │       ├── year=2026/month=01/
│   │       └── year=2026/month=02/
│   │
│   └── Table: server_logs
│       ├── Schema: [ip: string, path: string, status: int, ts: timestamp]
│       └── Location: s3://data-lake/raw/logs/
│
├── Database: analytics_db
│   └── Table: daily_user_metrics
│       ├── Schema: [user_id: string, total_clicks: bigint, revenue: double]
│       └── Location: s3://data-lake/analytics/daily_metrics/
│
└── Connection: prod_mysql_rds
    ├── Type: JDBC
    ├── URL: jdbc:mysql://prod-db.abc123.us-east-1.rds.amazonaws.com:3306/production
    └── Credentials: Stored in AWS Secrets Manager

2. AWS Glue Crawlers

A Crawler is an automated agent that connects to a data store, samples the data, infers its schema and structure, and writes the resulting metadata into the Data Catalog.

How Does a Crawler Work?

Step 1: Connect to Data Source
    │  (S3 bucket, JDBC database, DynamoDB table)
    ▼
Step 2: Sample Data
    │  (Reads a configurable sample of files/rows)
    ▼
Step 3: Infer Schema
    │  (Determines column names, data types, file format)
    ▼
Step 4: Detect Partitions
    │  (Identifies directory structures like year=2026/month=05/)
    ▼
Step 5: Write Metadata to Data Catalog
    │  (Creates or updates Database → Table → Schema → Partitions)
    ▼
Step 6: Handle Schema Changes
       (Adds new columns, updates types, logs deletions)

Crawler Configuration Options

When to Use Crawlers vs. Manual Table Definitions

3. AWS Glue ETL Engine

The ETL Engine is where the actual data processing happens. Under the hood, every Glue ETL Job runs on a managed Apache Spark cluster.

Glue Job Types

Glue Versions & Spark Versions

DPU (Data Processing Unit) — The Compute Currency

A DPU is the unit of compute in AWS Glue:

1 DPU = 4 vCPUs + 16 GB Memory

GlueContext & DynamicFrame — Glue's Custom Abstractions

AWS Glue extends the standard PySpark API with two custom abstractions:

GlueContext

A wrapper around SparkContext that adds Glue-specific capabilities:

from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

DynamicFrame

An extension of Spark's DataFrame specifically designed for semi-structured and schema-inconsistent data:

# Read from the Data Catalog using DynamicFrame
dynamic_frame = glueContext.create_dynamic_frame.from_catalog(
    database="raw_events_db",
    table_name="user_clickstream"
)

# DynamicFrame handles schema inconsistencies automatically!
# e.g., a column that is "string" in some records and "int" in others
# is represented as a ChoiceType rather than throwing an error.

Key Differences: DynamicFrame vs DataFrame

4. AWS Glue Job Bookmarks

Job Bookmarks enable incremental data processing — they track which data a job has already processed, so subsequent runs only process new or modified data.

How Bookmarks Work

Run 1 (Day 1):
    ├── Reads files: file_001.parquet, file_002.parquet, file_003.parquet
    ├── Processes and transforms all 3 files
    └── Bookmark saved: "Last processed = file_003.parquet"

Run 2 (Day 2):
    ├── New files arrived: file_004.parquet, file_005.parquet
    ├── Bookmark check: "Already processed up to file_003.parquet"
    ├── Processes ONLY: file_004.parquet, file_005.parquet  ← Incremental!
    └── Bookmark updated: "Last processed = file_005.parquet"

Bookmark Modes

5. AWS Glue Triggers & Workflows

Triggers

A Trigger starts the execution of one or more Crawlers or ETL Jobs:

Workflows

A Workflow is a visual orchestration graph that chains together Crawlers, Jobs, and Triggers into a multi-step pipeline:

              ┌──────────┐
              │  Trigger  │  (Scheduled: Daily 2 AM)
              └─────┬────┘
                    │
              ┌─────▼─────┐
              │  Crawler   │  (Scan s3://raw/ → Update Catalog)
              └─────┬─────┘
                    │ (On Success)
          ┌─────────┴──────────┐
          │                    │
    ┌─────▼─────┐       ┌─────▼─────┐
    │  ETL Job  │       │  ETL Job  │
    │  (Clean)  │       │  (Enrich) │
    └─────┬─────┘       └─────┬─────┘
          │                    │
          └─────────┬──────────┘
                    │ (Both Succeed)
              ┌─────▼─────┐
              │  ETL Job  │
              │ (Aggregate)│
              └─────┬─────┘
                    │
              ┌─────▼─────┐
              │  Crawler   │  (Update Catalog for analytics tables)
              └───────────┘

6. Security Architecture

IAM Roles & Policies

Every Glue component (Crawler, Job, Dev Endpoint) assumes an IAM Role to access AWS resources:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-data-lake-bucket",
        "arn:aws:s3:::my-data-lake-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetTable",
        "glue:GetDatabase",
        "glue:GetPartitions"
      ],
      "Resource": "*"
    }
  ]
}

Network Security

• VPC Connectivity: Glue Jobs can run inside a VPC to access private resources (RDS in a private subnet, on-prem databases via VPN/Direct Connect).
• Encryption: Data at rest is encrypted using AWS KMS. Data in transit uses TLS.
• AWS Lake Formation: Fine-grained column-level and row-level access control on Data Catalog tables.

Architecture Summary — Component Interaction Map

Understanding this architecture is the foundation for designing robust, scalable AWS Glue pipelines. In the next section, we'll put this knowledge into practice by creating and running a real Glue ETL Job.