Athena Operators & Hooks
SQL Directly Against S3, No Warehouse Required
Athena runs standard SQL directly against files sitting in S3 (via a Glue Data Catalog table pointing at them) — no cluster to manage, no data to load anywhere first. For ad-hoc analysis or lightweight scheduled aggregation over data that already lives in a data lake, it's often the simplest option in the entire AWS data stack.
Running a Query
from airflow.providers.amazon.aws.operators.athena import AthenaOperator
run_daily_summary = AthenaOperator(
task_id="run_daily_summary_query",
query="""
SELECT order_date, COUNT(*) AS order_count, SUM(amount) AS total_amount
FROM sales_data.orders
WHERE order_date = DATE '2026-09-04'
GROUP BY order_date
""",
database="sales_data",
output_location="s3://my-athena-results/daily-summaries/",
aws_conn_id="aws_default",
)
Every Athena query needs an output_location — Athena always writes results back to S3 as a side effect, even for a query you're only reading via XCom.
Run for real against a live Athena workgroup:
Figure — AthenaOperator polls until the query completes; the query execution ID is available downstream via .output for anything that needs to locate the results file.
Waiting on a Long-Running Query
AthenaOperator already polls internally by default, but for orchestrating a query that's kicked off elsewhere (or splitting "start" from "wait" for finer-grained retry control), use the sensor:
from airflow.providers.amazon.aws.operators.athena import AthenaOperator
from airflow.providers.amazon.aws.sensors.athena import AthenaSensor
start_query = AthenaOperator(
task_id="start_query",
query="SELECT * FROM sales_data.orders WHERE order_date = DATE '2026-09-04'",
database="sales_data",
output_location="s3://my-athena-results/",
aws_conn_id="aws_default",
sleep_time=0, # don't poll inside this task - the sensor below does that
)
wait_for_query = AthenaSensor(
task_id="wait_for_query",
query_execution_id=start_query.output,
aws_conn_id="aws_default",
poke_interval=15,
)
AthenaHook — Reading Results Back
from airflow.providers.amazon.aws.hooks.athena import AthenaHook
def fetch_summary_rows(query_execution_id: str, **context):
hook = AthenaHook(aws_conn_id="aws_default")
results = hook.get_query_results(query_execution_id=query_execution_id)
rows = results["ResultSet"]["Rows"][1:] # first row is the header
return [r["Data"] for r in rows]
Athena charges per terabyte scanned, not per query duration or a running cluster. Partitioning your S3 data by date (as most data lakes already do) and filtering on that partition column — like
WHERE order_date = DATE '2026-09-04' above — is the single biggest cost lever: it lets Athena skip scanning every other day's files entirely.