ECS & AWS Batch Operators & Hooks
When the Work Is "Run This Container"
Not every task fits Lambda's 15-minute ceiling or needs Spark's distributed compute. For "run this specific Docker image, give it real memory/CPU, for as long as it needs" — ECS and AWS Batch are the two container-native ways Airflow can hand off that work, similar in spirit to KubernetesPodOperator but native to AWS's own container orchestration.
Same constraint as EMR/Redshift/Glue: a real ECS task needs a running cluster, a task definition, a container image in a registry, and networking (VPC/subnets/security groups); AWS Batch needs a compute environment and job queue on top of that. That's substantial one-time infrastructure setup disproportionate to a demo screenshot, so - like this course's KubernetesPodOperator page - this one shows correct, real code without a live run.
ECS: Running a Task
from airflow.providers.amazon.aws.operators.ecs import EcsRunTaskOperator
run_container_job = EcsRunTaskOperator(
task_id="run_report_generator",
cluster="analytics-cluster",
task_definition="report-generator:latest",
launch_type="FARGATE",
overrides={
"containerOverrides": [{
"name": "report-generator",
"command": ["--date", "{{ ds }}"],
}],
},
network_configuration={
"awsvpcConfiguration": {
"subnets": ["subnet-0123456789abcdef0"],
"securityGroups": ["sg-0123456789abcdef0"],
"assignPublicIp": "DISABLED",
}
},
aws_conn_id="aws_default",
awslogs_group="/ecs/report-generator",
awslogs_stream_prefix="ecs/report-generator",
)
With launch_type="FARGATE", there's no EC2 instance to manage at all — AWS runs the container on serverless compute, similar in spirit to how KubernetesPodOperator can target a serverless Fargate profile.
AWS Batch: Running a Job
Batch adds its own queueing/scheduling layer on top of ECS or EKS — better suited than raw ECS when you have many jobs competing for constrained compute and want AWS to handle the queueing and prioritization.
from airflow.providers.amazon.aws.operators.batch import BatchOperator
run_batch_job = BatchOperator(
task_id="run_bulk_reprocessing_job",
job_name="reprocess-historical-orders",
job_queue="analytics-job-queue",
job_definition="reprocess-orders-job-def",
overrides={
"command": ["--start-date", "2020-01-01", "--end-date", "2020-12-31"],
},
aws_conn_id="aws_default",
)
BatchOperator polls for completion by default; for finer control over that polling (separate retry/backoff behavior), use BatchSensor after a fire-and-forget submission instead.
The Hooks Directly
from airflow.providers.amazon.aws.hooks.ecs import EcsHook
from airflow.providers.amazon.aws.hooks.batch_client import BatchClientHook
def check_ecs_task_exit_code(cluster: str, task_arn: str):
hook = EcsHook(aws_conn_id="aws_default")
client = hook.get_conn()
description = client.describe_tasks(cluster=cluster, tasks=[task_arn])
return description["tasks"][0]["containers"][0]["exitCode"]
Same underlying idea (run a container, get the result) across three different services. Pick Lambda for small/fast/stateless (under 15 minutes, modest memory). Pick ECS/Batch when the team's infrastructure is already AWS-native and container-based. Pick KubernetesPodOperator when the team already runs Kubernetes and wants every workload - Airflow tasks included - going through the same k8s-native tooling, regardless of cloud.