Pools & Resource Management
How Many Tasks Can Actually Run at Once?
Airflow has several overlapping concurrency limits, and Pools are the one most people never configure - right up until fifty tasks all try to hit the same downstream database at once. This page is exactly where all of that is controlled.
The Three Levels of Concurrency Control
Airflow doesn't have one concurrency setting — it has three, stacked on top of each other:
| Level | Setting | Controls |
|---|---|---|
| Whole deployment | [core] parallelism in airflow.cfg |
The absolute ceiling — total task instances running anywhere, across every DAG, at once |
| Per DAG | max_active_tasks (DAG parameter) |
How many tasks from this one DAG can run concurrently |
| Per Pool | Slots, set in the UI or CLI | How many tasks using this specific pool can run concurrently, regardless of which DAG they belong to |
Pools are the one that matters most in practice, because they're the only mechanism that limits concurrency across different DAGs — max_active_tasks only sees its own DAG.
The Problem Pools Solve
Say ten different DAGs each have a task that submits a Spark job to the same small cluster. Without a Pool, all ten could fire at once — nothing in Airflow's default configuration stops them, and the cluster falls over. A Pool puts a hard cap on that specific kind of work, independent of how many DAGs happen to need it.
Creating a Pool
Via the UI (Admin → Pools) or the CLI:
airflow pools set spark_pool 5 "Limits concurrent Spark-heavy tasks to 5"
That's a real pool, created by that exact command, visible immediately in the admin page:
Figure — every pool tracks live slot usage in four states. When Queued Slots climbs and stays there, that pool is the bottleneck.
Assigning Tasks to a Pool
from airflow.decorators import dag, task
@dag(schedule=None, start_date=..., catchup=False)
def multi_source_spark_pipeline():
@task(pool="spark_pool", pool_slots=1)
def run_spark_job_a():
...
@task(pool="spark_pool", pool_slots=1)
def run_spark_job_b():
...
@task() # no pool= specified -> uses default_pool, unaffected by spark_pool's limit
def unrelated_lightweight_task():
...
[run_spark_job_a(), run_spark_job_b(), unrelated_lightweight_task()]
Run for real: run_spark_job_a and run_spark_job_b both draw from spark_pool's 5 slots, while the unrelated task runs independently in default_pool:
Tasks that don't set
pool= aren't unlimited — they're just in default_pool (128 slots out of the box), which is itself a real, adjustable limit. There's no "no pool" state.
pool_slots — Weighting Tasks Differently
Not every task in a pool needs to count the same. A task processing 10x the data of another can claim more of the pool's capacity:
@task(pool="spark_pool", pool_slots=3) # counts as 3 of spark_pool's 5 slots
def run_heavy_spark_job():
...
With pool_slots=3, this single task leaves only 2 slots for everything else in spark_pool while it runs — useful when one job is known to be disproportionately resource-hungry and you want the pool's slot count to reflect real load, not just task count.
In the Pools admin page, a pool with a consistently high Queued Slots count is your actual throughput bottleneck — raising
[core] parallelism won't help if a specific pool is what's actually capping things.