Data Engineering Path · Airflow
AWS CORE PLATFORM CASE STUDY
S3 Operators & Hooks
S3 Is Airflow's Most Common Neighbor
Almost every real pipeline touches S3 somewhere — landing raw files, archiving processed output, or acting as the handoff point between two systems that can't talk to each other directly. This page covers the operators, the sensor, and the Hook underneath all of them.
The Operators
| Operator | Purpose |
|---|---|
S3CreateBucketOperator / S3DeleteBucketOperator |
Create or delete a bucket |
S3CreateObjectOperator |
Write an object directly from a string/bytes payload |
S3CopyObjectOperator |
Copy an object within or between buckets |
S3DeleteObjectsOperator |
Delete one or more keys |
S3ListOperator |
List keys under a prefix, returned via XCom |
S3FileTransformOperator |
Download → run a transform script → re-upload, in one task |
A Real, Complete Flow
from airflow.providers.amazon.aws.operators.s3 import (
S3CreateObjectOperator,
S3CopyObjectOperator,
S3DeleteObjectsOperator,
)
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
BUCKET = "my-data-lake"
upload_raw = S3CreateObjectOperator(
task_id="upload_raw_file",
s3_bucket=BUCKET,
s3_key="raw/sales_2026_09_04.csv",
data=csv_payload,
replace=True,
aws_conn_id="aws_default",
)
wait_for_file = S3KeySensor(
task_id="wait_for_raw_file",
bucket_name=BUCKET,
bucket_key="raw/sales_2026_09_04.csv",
aws_conn_id="aws_default",
poke_interval=30,
timeout=600,
)
archive_copy = S3CopyObjectOperator(
task_id="archive_raw_file",
source_bucket_name=BUCKET,
source_bucket_key="raw/sales_2026_09_04.csv",
dest_bucket_name=BUCKET,
dest_bucket_key="archive/sales_2026_09_04.csv",
aws_conn_id="aws_default",
)
cleanup_raw = S3DeleteObjectsOperator(
task_id="cleanup_raw_file",
bucket=BUCKET,
keys=["raw/sales_2026_09_04.csv"],
aws_conn_id="aws_default",
)
upload_raw >> wait_for_file >> archive_copy >> cleanup_raw
This exact pipeline, run for real against a live S3 bucket:
Figure — every task shows its real operator class name beneath it (S3CreateObjectOperator, S3KeySensor, S3CopyObjectOperator, S3DeleteObjectsOperator), confirming this is the exact code above, not a simplified stand-in.
Why an S3KeySensor between upload and copy, if the upload task already succeeded?
On a real pipeline, the "producer" of
On a real pipeline, the "producer" of
raw/sales_2026_09_04.csv is often a different DAG or an external system — not this DAG's own upload task. The sensor pattern here is what you'd actually use when you don't control the producer; when you do (as in this toy example), the direct dependency already guarantees ordering and the sensor is technically redundant but shown for completeness.
S3Hook Directly
Reach for the Hook directly when you need something the operators don't expose as a parameter — checking existence without failing the task, listing with pagination control, or generating a presigned URL.
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
def check_and_branch(**context):
hook = S3Hook(aws_conn_id="aws_default")
if hook.check_for_key(key="raw/sales_2026_09_04.csv", bucket_name="my-data-lake"):
return "process_file"
return "skip_no_file"
| S3Hook Method | Use For |
|---|---|
check_for_key() |
Boolean existence check, no exception on missing |
list_keys() |
List with prefix/delimiter, returns a plain list |
read_key() |
Read small text objects directly into memory |
load_file() |
Upload a local file (streams, doesn't load fully into memory first) |
generate_presigned_url() |
Give a downstream consumer temporary direct access without AWS credentials |
Note
Every operator on this page accepts
Every operator on this page accepts
aws_conn_id — pointing at an Airflow Connection of type Amazon Web Services (Admin → Connections), which stores the access key/secret (or, in production, is usually left blank so the Worker's IAM role is used instead — never hardcode long-lived AWS keys in a Connection running on infrastructure that already has an IAM role attached).
arrow_back Previous Topic
Custom Operators Hooks Sensors
Next Topic arrow_forward
SNS and SQS Operators and Hooks