Airflow Executors — Detailed Comparison

Executor Comparison Matrix

SequentialExecutor

flowchart LR
    S["Scheduler"] --> E["SequentialExecutor"] --> T1["Task 1\nruns to completion"] --> T2["Task 2\nruns to completion"] --> T3["Task 3\nruns to completion"]
    style S fill:#017cee,stroke:#015bb5,color:#fff
    style E fill:#F44336,stroke:#D32F2F,color:#fff

• Tasks run one at a time, in a single process
• Uses SQLite as the database (no external DB needed)
• Only for development and testing — never use in production

[core]
executor = SequentialExecutor

[database]
sql_alchemy_conn = sqlite:///airflow.db

LocalExecutor

graph LR
    S["Scheduler"] --> E["LocalExecutor"]
    E --> P1["Process 1<br/>Task A"]
    E --> P2["Process 2<br/>Task B"]
    E --> P3["Process 3<br/>Task C"]
    E --> P4["Process N<br/>Task D"]
    style S fill:#017cee,stroke:#015bb5,color:#fff
    style E fill:#FFC107,stroke:#F9A825,color:#333
    style P1 fill:#4CAF50,stroke:#388E3C,color:#fff
    style P2 fill:#4CAF50,stroke:#388E3C,color:#fff
    style P3 fill:#4CAF50,stroke:#388E3C,color:#fff
    style P4 fill:#4CAF50,stroke:#388E3C,color:#fff

• Spawns multiple processes on a single machine
• Requires PostgreSQL or MySQL
• Great for small to medium deployments (< 50 concurrent tasks)

[core]
executor = LocalExecutor
parallelism = 32

[database]
sql_alchemy_conn = postgresql+psycopg2://airflow:password@localhost:5432/airflow

CeleryExecutor

graph TB
    S["Scheduler"] --> E["CeleryExecutor"]
    E --> Q[("Message Queue<br/>(Redis / RabbitMQ)")]
    Q --> W1["Worker 1<br/>(Machine A)"]
    Q --> W2["Worker 2<br/>(Machine B)"]
    Q --> W3["Worker N<br/>(Machine C)"]
    W1 --> DB[("Metadata DB")]
    W2 --> DB
    W3 --> DB
    style S fill:#017cee,stroke:#015bb5,color:#fff
    style E fill:#4CAF50,stroke:#388E3C,color:#fff
    style Q fill:#FF9800,stroke:#F57C00,color:#fff
    style W1 fill:#2196F3,stroke:#1976D2,color:#fff
    style W2 fill:#2196F3,stroke:#1976D2,color:#fff
    style W3 fill:#2196F3,stroke:#1976D2,color:#fff
    style DB fill:#e43921,stroke:#c02a10,color:#fff

• Tasks are distributed to remote workers via a message broker (Redis or RabbitMQ)
• Workers can run on separate machines — horizontal scaling
• Best for large teams with stable, predictable workloads

[core]
executor = CeleryExecutor

[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:password@postgres:5432/airflow
worker_concurrency = 16

KubernetesExecutor

graph TB
    S["Scheduler"] --> E["KubernetesExecutor"]
    E --> K8S["Kubernetes API Server"]
    K8S --> P1["Pod 1<br/>(Task A)<br/>2 CPU, 4 GB"]
    K8S --> P2["Pod 2<br/>(Task B)<br/>1 CPU, 2 GB"]
    K8S --> P3["Pod 3<br/>(Task C)<br/>4 CPU, 8 GB"]
    P1 -->|"exits"| X1["Cleaned up"]
    P2 -->|"exits"| X2["Cleaned up"]
    P3 -->|"exits"| X3["Cleaned up"]
    style S fill:#017cee,stroke:#015bb5,color:#fff
    style E fill:#2196F3,stroke:#1976D2,color:#fff
    style K8S fill:#326CE5,stroke:#2457B5,color:#fff
    style P1 fill:#4CAF50,stroke:#388E3C,color:#fff
    style P2 fill:#FF9800,stroke:#F57C00,color:#fff
    style P3 fill:#9C27B0,stroke:#7B1FA2,color:#fff

• Each task runs in its own Kubernetes pod with custom resources
• Pods are created on-demand and destroyed after completion — zero waste
• Best for cloud-native deployments with variable workloads and strict isolation requirements

# pod template.yaml — Custom resources per task
apiVersion: v1
kind: Pod
metadata:
  name: airflow-worker
spec:
  containers:

    - name: base
      image: apache/airflow:2.10.0
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "2"
          memory: "4Gi"

Decision Flowchart: Which Executor Should I Use?

graph TD
    START["How will you<br/>deploy Airflow?"] --> Q1{"Running on<br/>Kubernetes?"}
    Q1 -->|"Yes"| K8S["KubernetesExecutor"]
    Q1 -->|"No"| Q2{"Need distributed<br/>workers?"}
    Q2 -->|"Yes"| CELERY["CeleryExecutor"]
    Q2 -->|"No"| Q3{"Production<br/>or Dev?"}
    Q3 -->|"Production"| LOCAL["LocalExecutor"]
    Q3 -->|"Development"| SEQ["SequentialExecutor"]

    style START fill:#607D8B,stroke:#455A64,color:#fff
    style K8S fill:#2196F3,stroke:#1976D2,color:#fff
    style CELERY fill:#4CAF50,stroke:#388E3C,color:#fff
    style LOCAL fill:#FFC107,stroke:#F9A825,color:#333
    style SEQ fill:#F44336,stroke:#D32F2F,color:#fff

This content is reserved for Premium Members.