Airflow as the Control Plane

Architecture Analogy

Think of a modern data stack as an airport:

graph TB
    subgraph "✈️ CONTROL PLANE — Air Traffic Control (Airflow)"
        ATC["Airflow Scheduler<br/>Decides WHEN things run"]
        RADAR["Airflow Webserver<br/>Monitors everything"]
        COMM["Airflow Workers<br/>Dispatches commands"]
    end

    subgraph "🛫 DATA PLANE — The Airlines & Runways"
        SPARK["Apache Spark<br/>Heavy data processing"]
        SNOW["Snowflake<br/>Data warehousing"]
        DBT["dbt<br/>SQL transformations"]
        S3["AWS S3<br/>Data lake storage"]
        KAFKA["Apache Kafka<br/>Event streaming"]
    end

    ATC --> COMM
    RADAR --> ATC
    COMM -->|"Submit Spark job"| SPARK
    COMM -->|"Execute SQL"| SNOW
    COMM -->|"Run dbt model"| DBT
    COMM -->|"Transfer files"| S3
    COMM -->|"Trigger pipeline"| KAFKA

    style ATC fill:#017cee,stroke:#015bb5,color:#fff
    style RADAR fill:#00c7d4,stroke:#009ea8,color:#fff
    style COMM fill:#00ad46,stroke:#008a38,color:#fff
    style SPARK fill:#E25A1C,stroke:#c04a14,color:#fff
    style SNOW fill:#29B5E8,stroke:#1a8fbf,color:#fff
    style DBT fill:#FF694B,stroke:#e05535,color:#fff
    style S3 fill:#FF9900,stroke:#cc7a00,color:#fff
    style KAFKA fill:#231F20,stroke:#000,color:#fff

What Happens During a DAG Run?

Here's what actually happens when Airflow executes a typical ETL pipeline:

sequenceDiagram
    participant S as Scheduler
    participant W as Worker
    participant API as External API
    participant SP as Apache Spark
    participant DW as Snowflake

    S->>W: Task 1: extract_data
    W->>API: HTTP GET /api/v1/sales
    API-->>W: JSON response (2 MB)
    W->>W: Save to S3 staging

    S->>W: Task 2: transform_data
    W->>SP: spark-submit transform.py
    SP->>SP: Process 500 GB data
    SP-->>W: Job completed

    S->>W: Task 3: load_to_warehouse
    W->>DW: COPY INTO sales_table FROM @stage
    DW-->>W: 10M rows loaded

    S->>W: Task 4: quality_check
    W->>DW: SELECT COUNT(*) FROM sales_table
    DW-->>W: Row count matches

    Note over S,DW: Total Airflow resource usage: ~100 MB RAM<br/>Total data processed: 500 GB (by Spark & Snowflake)

The Control Plane Principle in Practice

Anti-Pattern: Processing Data in Airflow

# ❌ BAD — Processing data inside Airflow worker
@task()
def bad_transform():
    import pandas as pd
    # This loads 50 GB into the Airflow worker's memory!
    df = pd.read_parquet("s3://bucket/huge_dataset.parquet")
    result = df.groupby("category").sum()
    result.to_parquet("s3://bucket/output.parquet")

# ✅ GOOD — Delegating to an external processing engine
@task()
def good_transform():
    from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
    # Airflow only sends the command, Spark does the heavy lifting
    return SparkSubmitOperator(
        task_id="run_spark_transform",
        application="/spark/jobs/transform.py",
        conn_id="spark_cluster",
    ).execute(context={})

When Airflow is the Right Choice

This content is reserved for Premium Members.