Great Expectations — Data Quality as Code
Beyond a Handful of assert Statements
Earlier in this course, data quality checks looked like a single assert row_count > 0 inside a Python task. That's fine for one or two checks. Great Expectations is what to reach for once a table has a dozen real constraints worth checking — nulls, ranges, uniqueness, referential rules — and you want them declared as data, not scattered across ad hoc code.
Same honest treatment as EMR/Redshift/Glue/ECS-Batch/KubernetesPodOperator: code-only, no live run. Great Expectations' dependency footprint (pandas, numpy, scipy and their exact pinned versions) conflicted with this shared Airflow sandbox's own pinned versions badly enough that installing it broke pandas outright - not worth destabilizing every other real screenshot on this page for one more.
Declaring Expectations
An "Expectation" is one declarative rule about the data:
import great_expectations as gx
context = gx.get_context()
validator = context.sources.pandas_default.read_csv("orders.csv")
validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_between("amount", min_value=0, max_value=100000)
validator.expect_column_values_to_be_unique("order_id")
validator.save_expectation_suite()
Each expect_* call is both documentation ("this column should never be null") and an executable check — the same statement serves both purposes, instead of a comment next to an assert.
Running Validation Inside a DAG
from airflow.decorators import dag, task
import great_expectations as gx
@dag(schedule="@daily", start_date=..., catchup=False)
def sales_pipeline_with_quality_gate():
@task()
def extract_and_load():
...
@task()
def validate_orders_quality():
context = gx.get_context()
result = context.run_checkpoint(checkpoint_name="orders_checkpoint")
if not result["success"]:
raise ValueError("Data quality validation failed - see GE Data Docs for details")
return result["success"]
@task()
def load_to_warehouse():
...
extract_and_load() >> validate_orders_quality() >> load_to_warehouse()
run_checkpoint() returns a result object with success=False rather than raising by default - the DAG code above explicitly raises to turn that into a failed, retryable Airflow task. Without that explicit check, a failed validation would silently report as green, since nothing else in the task actually errored.
Data Docs — Human-Readable Validation Reports
Great Expectations generates browsable HTML reports ("Data Docs") for every checkpoint run — a shareable audit trail of exactly which checks ran, on what data, with what result, independent of anything in the Airflow UI itself. Point stakeholders who don't use Airflow at the Data Docs site instead of asking them to read task logs.
If the data already lives in a Glue Catalog table, the
GlueDataQualityOperator covered in the AWS operators section is AWS's own managed equivalent. Reach for Great Expectations instead when the data quality layer needs to work identically across multiple sources (S3, a warehouse, a plain CSV) rather than being tied to one AWS service.