Data Engineering Path · Airflow
Hands-On Exercises
Practice What You've Learned
Complete these exercises to reinforce your understanding of Airflow's core concepts. Each exercise builds on the previous one.
Exercise 1: Create Your First DAG
Create a DAG called hello_airflow that runs three tasks in sequence:
# TODO: Complete this DAG
# 1. Task 1: Print "Hello from Airflow!"
# 2. Task 2: Print the current date and time
# 3. Task 3: Print "Pipeline completed successfully!"
# Schedule: Run every hour
# Start date: January 1, 2024
from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator
from datetime import datetime
# Your code here...
📘 Hint
Use
Use
PythonOperator with python_callable for each task. Define three simple Python functions and set up the dependency chain with >>.
Exercise 2: TaskFlow API Conversion
Convert the DAG from Exercise 1 to use the TaskFlow API with @dag and @task decorators.
Exercise 3: Identify the Anti-Patterns
Review this DAG and identify at least 3 problems:
from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator
from datetime import datetime
import pandas as pd
with DAG("bad_pipeline", schedule="* * * * *", start_date=datetime(2020, 1, 1)) as dag:
def process_all():
# Read 50GB file
df = pd.read_csv("s3://bucket/huge_file.csv")
# Transform
result = df.groupby("col").sum()
# Write back
result.to_csv("s3://bucket/output.csv")
# Send email
import smtplib
smtplib.SMTP("mail.server.com").sendmail("a@b.com", "c@d.com", "Done!")
do_everything = PythonOperator(task_id="do_everything", python_callable=process_all)
🔍 Click to reveal the anti-patterns
1. **Processing data inside Airflow** — 50 GB CSV should be processed by Spark/BigQuery, not in the worker
2. **`schedule="* * * * *"`** — Runs every minute! This is almost certainly wrong
3. **`start_date=datetime(2020, 1, 1)` with no `catchup=False`** — Airflow will try to create DAG Runs for every minute since 2020
4. **Single monolithic task** — Violates the principle of atomicity. Extract, transform, load, and notify should be separate tasks
5. **No retries configured** — If it fails, it just fails. No automatic recovery
6. **Hardcoded credentials** — SMTP server details shouldn't be in the code. Use Connections
7. **No error handling** — No try/except, no alerting configuration