Hands-On Exercises — Architecture

Exercise 1: Draw the Architecture

On paper or in a diagramming tool, draw the Airflow architecture for a CeleryExecutor deployment with:

• 2 Scheduler instances (HA)
• 1 Webserver
• 3 Celery Workers
• PostgreSQL metadata database
• Redis message broker

Label each connection and describe what data flows between components.

Exercise 2: Choose the Right Executor

For each scenario, recommend the best executor and explain why:

1. A startup with 5 DAGs and 2 data engineers
2. An enterprise running 500 DAGs with 200 concurrent tasks
3. A machine learning team where each task needs different GPU resources
4. A developer testing a new DAG on their laptop

Exercise 3: Install Airflow Locally

Using Docker Compose, set up Airflow on your machine and:

1. Access the webserver at http://localhost:8080
2. Enable one of the example DAGs
3. Trigger a DAG Run manually
4. Navigate to the Graph view and identify all tasks
5. Check the logs of a successful task

This content is reserved for Premium Members.