Scenario: Streaming Kafka Ingestion

Scenario: Real-Time Fraud Detection via Kafka Ingestion

The Challenge

A data engineering team needs to run a Spark Structured Streaming application that ingests continuous payment streams from an Apache Kafka cluster, joins them with static blacklist databases, flags suspicious transactions, and writes the flagged outputs to a downstream real-time analytics dashboard database. The job must run 24/7 without memory leaks, resource starvation, or latency spikes.

1. Optimal Spark-Submit Configuration

spark-submit \
    --master yarn \
    --deploy-mode cluster \
    --name "Realtime-Kafka-Fraud-Detector" \
    --num-executors 6 \
    --executor-cores 3 \
    --executor-memory 8g \
    --driver-memory 4g \
    --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0 \
    --conf spark.streaming.backpressure.enabled=true \
    --conf spark.streaming.kafka.maxRatePerPartition=5000 \
    --conf spark.sql.streaming.forceDeleteTempCheckpointLocation=false \
    --conf spark.cleaner.referenceTracking.cleanCheckpoints=true \
    hdfs:///scripts/realtime_fraud_detector.py

2. Parameter Explanations & Rationale

• --packages org.apache.spark:spark-sql-kafka-...: Critical for dependency loading. Instead of manually downloading jar packages, Spark automatically downloads the Spark-Kafka integration connector from Maven Central and distributes it to all worker nodes.
• Moderate Resources (--num-executors 6 --executor-cores 3 --executor-memory 8g): Streaming applications don't process giant historical batches. Instead, they handle small micro-batches every few seconds. Allocating moderate resources prevents wasting expensive cluster capacity on a job that runs 24/7.
• spark.streaming.backpressure.enabled=true: Vital for pipeline stability. If a sudden event occurs (e.g., Black Friday traffic) and Kafka receives an enormous spike of messages, backpressure automatically slows down Spark ingestion rates to match Spark's processing capacity, preventing the executors from running out of memory.
• spark.streaming.kafka.maxRatePerPartition=5000: Sets a safety limit on the maximum number of messages Spark will ingest per Kafka partition per second during startup or recovery, preventing the system from being overwhelmed.
• spark.cleaner.referenceTracking.cleanCheckpoints=true: Enables automatic metadata and checkpoint garbage collection. This is critical for streaming jobs running for months on end to prevent memory leaks in the Driver JVM.