Scenario: Kubernetes Cloud Analytics

Scenario: Serverless Cloud Customer Cohort Analysis

The Challenge

A data analytics team needs to analyze clickstream events stored in an S3 bucket to calculate user cohort retention rates. The processing must execute inside a modern Kubernetes (K8s) cluster. The pipeline needs to use containerized dependencies and scale up dynamically on-demand using cloud virtual machines, then terminate to minimize computing costs.

1. Optimal Spark-Submit Configuration

spark-submit \
    --master k8s://https://kubernetes.default.svc.cluster.local:443 \
    --deploy-mode cluster \
    --name "k8s-clickstream-cohorts" \
    --conf spark.kubernetes.container.image=888888888888.dkr.ecr.us-east-1.amazonaws.com/spark-py-jobs:v1.2 \
    --conf spark.kubernetes.namespace=data-platform \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator \
    --conf spark.executor.instances=15 \
    --conf spark.kubernetes.executor.request.cores=4 \
    --conf spark.executor.cores=4 \
    --conf spark.executor.memory=12g \
    --conf spark.driver.memory=6g \
    --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
    --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider \
    local:///opt/spark/work-dir/cohort_analysis.py \
    --s3_input "s3a://analytics-lakehouse/events/2026/05/"

2. Parameter Explanations & Rationale

• --master k8s://...: Submits the application directly to the Kubernetes API server endpoint to handle resource orchestrations.
• --conf spark.kubernetes.container.image=...: Points to a pre-built Docker image stored in ECR containing the correct operating system libraries, PySpark binaries, custom Python scripts (cohort_analysis.py), and packages (like pandas or scipy).
• --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator: Connects the Driver pod to a specific Kubernetes service account. This gives the Driver pod permission to create and delete Executor pods inside the K8s namespace.
• local:///opt/spark/...: Notice the local:/// prefix instead of hdfs:// or s3://. This indicates that the PySpark script is already baked directly inside the Docker image container files, avoiding the need to download it at runtime.
• WebIdentityTokenCredentialsProvider: Implements IAM Roles for Service Accounts (IRSA). This allows the pods to securely authenticate with AWS S3 using Kubernetes service account annotations instead of hardcoding dangerous AWS access keys.