The Modern Data Engineer: Roles & Practices in the AI Age
The Identity Crisis of Data Engineering
AI created an identity crisis in the profession. In 2019, the job was clear: build pipelines, manage schemas, ensure data freshness. By 2024, the boundaries had blurred dramatically:
- ML Engineers were writing pipelines
- Data Scientists were building feature stores
- Analytics Engineers were owning transformations (dbt)
- Data Engineers were expected to understand LLM APIs
This convergence created a new archetype: the AI-Native Data Engineer.
Practice 1: Data Contracts Replace Ad-Hoc Schemas
The Old Way: Trust and Hope
Engineers consumed upstream data without formal agreements. When the upstream team changed a column type, downstream pipelines broke silently.
The New Way: Data Contracts
A data contract is a formal, schema-versioned agreement between data producers and consumers.
# data contract.yaml - The new standard
apiVersion: "0.9.3"
kind: "DataContract"
id: "urn:datacontract:ecommerce:orders:v2"
info:
title: "Orders Dataset Contract"
version: "2.1.0"
owner: "platform-team@company.com"
status: "active"
models:
orders:
type: table
description: "Core orders table - source of truth for revenue metrics"
fields:
order_id:
type: string
required: true
unique: true
description: "UUID for the order"
customer_id:
type: string
required: true
references: "customers.customer_id"
revenue:
type: decimal
precision: 10
scale: 2
minimum: 0
description: "Gross revenue in USD"
order_date:
type: date
required: true
quality:
type: SodaCL
specification:
checks for orders:
- row_count > 0
- missing_count(customer_id) = 0
- duplicate_count(order_id) = 0
- min(revenue) >= 0
servicelevels:
freshness:
description: "Data must be updated within 1 hour of transaction"
threshold: "1h"
availability:
percentage: 99.9
Why This Matters for AI: LLMs consuming your data via Text-to-SQL need reliable, well-documented schemas. Without data contracts, AI-generated queries produce wrong answers.
Practice 2: dbt Becomes the Universal Transformation Layer
dbt (data build tool) emerged as the industry-standard transformation framework because it:
- Version-controls all SQL transformations (Git-native)
- Auto-generates documentation and lineage
- Builds in testing as a first-class concept
- Integrates with AI tools (dbt Copilot, dbt Cloud AI)
-- models/marts/fct_daily_revenue.sql
-- dbt model with built-in documentation and tests
{{
config(
materialized='incremental',
unique_key='date_order_id',
on_schema_change='sync_all_columns',
tags=['finance', 'daily', 'critical']
)
}}
with orders as (
select * from {{ ref('stg_orders') }}
{% if is_incremental() %}
where order_date >= (select max(order_date) from {{ this }})
{% endif %}
),
customers as (
select * from {{ ref('dim_customers') }}
),
final as (
select
o.order_date,
o.order_id,
o.order_id || '_' || o.order_date::varchar as date_order_id,
c.customer_segment,
c.region,
o.revenue,
o.revenue * 0.8 as net_revenue, -- after 20% platform fee
o.is_refunded
from orders o
left join customers c using (customer_id)
where not o.is_test_order
)
select * from final
# schema.yml - tests co-located with models
models:
- name: fct_daily_revenue
description: "Daily grain revenue fact table for finance reporting"
meta:
owner: "analytics-eng@company.com"
sla: "available by 6am UTC"
columns:
- name: date_order_id
tests:
- unique
- not_null
- name: revenue
tests:
- not_null
- dbt_utils.accepted_range:
min_value: 0
max_value: 1000000
- name: customer_segment
tests:
- accepted_values:
values: ['enterprise', 'smb', 'consumer', 'unknown']
Practice 3: The Lakehouse Architecture Wins
The debate between data lakes and data warehouses ended in a draw — the Lakehouse won. Open table formats like Delta Lake, Apache Iceberg, and Apache Hudi brought ACID transactions and schema evolution to the data lake.
Traditional Stack (2015):
Data Lake (S3) → ETL → Data Warehouse (Redshift) → BI
Modern Lakehouse Stack (2024):
All Sources → Data Lake (S3/ADLS/GCS)
│
▼
Open Table Format (Iceberg/Delta)
┌──────────────────────────────┐
│ ACID Transactions │
│ Schema Evolution │
│ Time Travel │
│ Row-level deletes (GDPR) │
└──────────────────────────────┘
│
┌─────────┼──────────┐
▼ ▼ ▼
SQL ML Training AI Agents
(Athena) (SageMaker) (Bedrock)
Practice 4: Metadata as a First-Class Product
The Old View: Metadata is an Afterthought
Engineers wrote code first, documented later (or never). Metadata lived in stale wikis.
The New View: Metadata Powers AI
# Modern metadata-driven pipeline with OpenLineage
from openlineage.client import OpenLineageClient
from openlineage.client.run import RunEvent, RunState, Run, Job
from openlineage.client.facet import (
SchemaDatasetFacet,
SchemaField,
DataQualityMetricsInputDatasetFacet
)
client = OpenLineageClient.from_environment()
# Emit lineage BEFORE running the job
run_event = RunEvent(
eventType=RunState.START,
eventTime=datetime.now().isoformat(),
run=Run(runId=str(uuid.uuid4())),
job=Job(namespace="ecommerce", name="daily_revenue_aggregation"),
inputs=[{
"namespace": "s3://raw-data",
"name": "orders/2024-01-15/",
"facets": {
"schema": SchemaDatasetFacet(fields=[
SchemaField("order_id", "STRING"),
SchemaField("revenue", "DECIMAL(10,2)"),
SchemaField("order_date", "DATE")
])
}
}],
outputs=[{
"namespace": "s3://curated",
"name": "daily_revenue/",
}]
)
client.emit(run_event)
Why This Matters: Data catalogs (Datahub, Amundsen, Collibra) consume this metadata to power AI-driven data discovery. "Find me all tables related to customer churn" becomes possible.
Practice 5: Observability Replaces Monitoring
Old Model: Alert When It Breaks
Engineers set up cron-job health checks and email alerts. By the time they knew about a problem, business was already impacted.
New Model: Anomaly-Based Observability
# Monte Carlo / Great Expectations style observability
import great_expectations as gx
context = gx.get_context()
# Define expectations as code
suite = context.add_expectation_suite("orders_freshness_suite")
validator = context.get_validator(
datasource_name="s3_datasource",
data_asset_name="orders"
)
# ML-powered anomaly detection on volume
validator.expect_table_row_count_to_be_between(
min_value={"$PARAMETER": "PREVIOUS_DAY_COUNT * 0.7"},
max_value={"$PARAMETER": "PREVIOUS_DAY_COUNT * 1.3"}
)
# Freshness check
validator.expect_column_max_to_be_between(
column="order_date",
min_value={"$PARAMETER": "NOW() - INTERVAL '25 HOURS'"},
max_value={"$PARAMETER": "NOW()"}
)
# Distribution shift detection
validator.expect_column_mean_to_be_between(
column="revenue",
min_value={"$PARAMETER": "HISTORICAL_MEAN * 0.85"},
max_value={"$PARAMETER": "HISTORICAL_MEAN * 1.15"}
)
results = validator.validate()
The New Data Engineering Competency Map
Level 1 - Foundation (Still Required)
├── SQL (Advanced)
├── Python / PySpark
├── Cloud Storage (S3/GCS/ADLS)
└── Git / Version Control
Level 2 - Modern Tooling (Expected by 2024)
├── dbt (Transformations)
├── Airflow / Dagster (Orchestration)
├── Delta Lake / Iceberg (Open Table Formats)
└── Terraform (Infrastructure as Code)
Level 3 - AI-Era Skills (Competitive Differentiator)
├── LLM API Integration (Bedrock, OpenAI)
├── Vector Databases (Pinecone, pgvector, Weaviate)
├── Data Contract Design
├── AI Pipeline Observability
└── Prompt Engineering for Data Tools
Level 4 - Frontier (Emerging)
├── Agentic Data Systems Design
├── RAG Pipeline Architecture
├── AI Feature Store Engineering
└── Multi-modal Data Processing
Summary
The practices of data engineering didn't die — they elevated. SQL is still king. Python is still the glue. But the engineer who only knows these will be outcompeted by the one who also understands how to:
- Design data schemas that AI systems can understand
- Build pipelines that produce AI-ready outputs
- Govern AI-generated data with the same rigor as human-authored data
Find this content helpful?
☕ Buy me a coffee