Theoretical Quiz — Module 6

An idempotent task produces the same result regardless of how many times it runs for the same logical date. This is critical because Airflow supports retries and manual re-runs. Without idempotency, re-running a task could duplicate data. The pattern is: DELETE existing data for the date, then INSERT fresh data.

1. Always set catchup=False and max_active_runs=1
2. Use Connections for all credentials (never hardcode)
3. Set retries, timeouts, and SLA alerts on every task
4. Never process large data inside Airflow workers
5. Run DAG validation tests in CI/CD before deploying

The scheduler re-parses DAG files every 30 seconds (configurable). Any code at the top level (outside of tasks) runs during every parse cycle. This means API calls, database queries, or expensive computations at the top level will execute thousands of times per day, potentially causing rate limiting, credential issues, and performance degradation.

1. Task-level: email_on_failure=True in default_args for immediate notification
2. DAG-level: on_failure_callback to send Slack/PagerDuty alerts
3. SLA monitoring: sla=timedelta(hours=1) for time-sensitive pipelines
4. DAG Run timeout: dagrun_timeout=timedelta(hours=3) to catch stuck pipelines
5. Meta-monitoring: A separate "health check" DAG that monitors other DAGs