Introduction - Spark Structured Streaming
"Master Apache Spark and Big Data Engineering from first principles."
What You'll Master
Streaming Sources & Sinks
Setting up active stream reads (Kafka, File, Rate) and writing to Sinks (Console, Parquet, Kafka) using spark.readStream.
Output Modes & Triggers
Differentiating execution modes (Append, Update, Complete) and controlling trigger execution intervals.
Event-Time Watermarking
Aggregating data based on event generation times with tumbling and sliding windows, and dropping old state safely with watermarks.
Real-World Case Study
An IoT Sensor Streaming case study covering real-time JSON directory streams, sliding windows, and watermarking.
Learning Path & Course Syllabus
An O'Reilly case study explaining real-time JSON directory streams, sliding windows, and watermarking.
Differentiating execution modes (Append, Update, Complete) and controlling trigger execution intervals.
Setting up active stream reads (Kafka, File, Rate) and writing to Sinks (Console, Parquet, Kafka) using spark.readStream.
Aggregating data based on event generation times, tumbling and sliding windows, and executing watermarks to drop old state safely.
A hands-on workbook applying streaming sources, output modes, and watermarking to real sliding-window scenarios.
Scenario questions covering streaming output modes, source/sink wiring, and watermark-based state eviction.
What's Included in This Module
| Component | Coverage Details |
|---|---|
| Core Topics | Driver & Executor Architecture, Cluster Managers, Datasets |
| Practical Exercises | Interactive Hands-on Labs & Spark Tasks |
| Assessments | 1 Practical Assignment + 1 System Design Interview Quiz |