Secrets Backends — Connections Without a Database
Every Credential Covered So Far Lived in Airflow's Own Database
Every Connection created throughout this course was added straight into Airflow's metadata database. That's fine for a lot of teams - Connections are encrypted at rest there. But once a company already has AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager as its single source of truth for credentials, duplicating them into Airflow's own DB becomes a second place secrets can drift out of sync or get missed during a rotation.
Wiring this up live requires restarting the Airflow scheduler/webserver with real AWS credentials injected as container environment variables (the secrets backend needs to authenticate to AWS itself, separately from any Connection stored inside Airflow) - out of scope for a quick screenshot in this shared sandbox without disrupting everything else already running in it. Code-only.
The Core Idea
A Secrets Backend is checked first, before Airflow's own metadata database, whenever a DAG asks for a Connection or Variable by name. If the backend has it, Airflow uses that value; if not, it falls back to its own database. Nothing about DAG code changes — PostgresHook(postgres_conn_id="warehouse") looks identical whether warehouse lives in Airflow's DB or in Secrets Manager.
AWS Secrets Manager
# airflow.cfg
[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}
With this configured, a Connection named warehouse is looked up as the secret airflow/connections/warehouse in Secrets Manager — created there like any other secret:
aws secretsmanager create-secret \
--name "airflow/connections/warehouse" \
--secret-string '{"conn_type": "postgres", "host": "warehouse.internal", "login": "airflow", "password": "...", "schema": "analytics", "port": 5432}'
HashiCorp Vault
[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {"connections_path": "connections", "variables_path": "variables", "url": "https://vault.internal:8200"}
GCP Secret Manager
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
backend_kwargs = {"connections_prefix": "airflow-connections", "project_id": "my-gcp-project"}
All three follow the identical shape: a backend class, and backend_kwargs telling it where under that system's own naming convention to look.
What Doesn't Change
DAG code is completely unaffected — a Hook or Operator that takes conn_id="warehouse" works exactly the same regardless of which backend eventually resolves that name:
from airflow.providers.postgres.hooks.postgres import PostgresHook
def run_query(**context):
# Identical code whether "warehouse" lives in Airflow's DB,
# AWS Secrets Manager, Vault, or GCP Secret Manager
hook = PostgresHook(postgres_conn_id="warehouse")
return hook.get_first("SELECT COUNT(*) FROM orders")
A secrets backend doesn't have to hold every Connection. Common real-world setup: only the genuinely sensitive production credentials live in Secrets Manager/Vault, while low-stakes dev/test Connections stay in Airflow's own database - the fallback behavior (check backend first, then Airflow's DB) makes this mix-and-match approach work without any special DAG-level handling.