Backfilling — Running Historical Dates
Running the Past, Not Just the Present
catchup=False has been the default recommendation throughout this course - start processing from now, ignore historical dates. Backfilling is the deliberate, opposite operation: intentionally running a DAG for a range of past dates, most often after fixing a bug that affected historical data or adding a brand-new DAG that needs to process months of past data once.
catchup=True vs a Manual Backfill
These solve different problems, despite both touching historical dates:
catchup=True |
airflow dags backfill |
|
|---|---|---|
| When it runs | Automatically, the moment the DAG is unpaused | On demand, whenever you explicitly invoke it |
| Scope | Every missed interval since start_date |
Exactly the date range you specify |
| Typical use | A DAG that should always be fully caught up | A one-time re-run after a bug fix, for a known range |
Running a Backfill
airflow dags backfill sales_etl \
--start-date 2026-01-01 \
--end-date 2026-01-31
This creates and runs a DAG run for every scheduled interval between the two dates — 31 daily runs for a schedule="@daily" DAG, exactly as if each of those days had originally been processed on time.
Why This Needs execution_date/logical_date-Aware Code
A backfilled run for January 5th has its {{ ds }} (and every Jinja date variable covered in the Step-by-Step module) set to 2026-01-05, not today's actual date. Any task that hardcodes "today" instead of using the templated logical date will backfill wrong — reprocessing the same day's data 31 times instead of each historical day correctly:
# Wrong - ignores the backfill's actual target date entirely
def extract_wrong():
today = datetime.now().strftime("%Y-%m-%d") # always "today", even during a Jan 5th backfill run
...
# Correct - uses the DAG run's actual logical date
def extract_correct(ds=None, **context):
... # ds is "2026-01-05" for that specific backfilled run
Backfilling Safely
Check idempotency first (covered in the Production Best Practices module) — a backfill that re-runs a load task without a delete-then-insert pattern will duplicate every row instead of replacing it.
Watch for max_active_runs. A DAG with max_active_runs=1 backfilling 31 days runs them one at a time, sequentially — potentially very slow. Consider temporarily raising it for the backfill if the underlying data source can handle concurrent historical requests without collision.
Use --rerun-failed-tasks for resuming an interrupted backfill:
airflow dags backfill sales_etl \
--start-date 2026-01-01 --end-date 2026-01-31 \
--rerun-failed-tasks
This re-runs only the runs/tasks that actually failed on a previous backfill attempt, rather than restarting the entire range from scratch.
Backfilling January while the DAG's normal schedule is also actively running for the current date means two sets of runs writing to related tables simultaneously. For anything touching shared state, backfill during a maintenance window or against an isolated copy of the destination, not blindly alongside live traffic.