Bootstrap EMRFS and Serverless

To run cost-efficient, production-grade workloads on Amazon EMR, you must master its advanced configurations and feature integrations. This guide covers Bootstrap Actions, EMRFS, and the modern EMR Serverless deployment model.

1. Bootstrap Actions

A Bootstrap Action is a custom shell script that EMR runs on all cluster nodes (Primary, Core, and Task) before Hadoop/Spark services start.

Common Use Cases:

• Installing operating system libraries (yum install ...).
• Installing Python packages (pip install pandas scikit-learn pg8000).
• Setting environment variables or writing custom cluster configuration files.
• Downloading custom connection certificates or database drivers.

Example Bootstrap Script (install_packages.sh)

Upload this file to an S3 bucket before launching the cluster:

#!/bin/bash
set -ex

# Update pip
sudo python3 -m pip install --upgrade pip

# Install required python packages for PySpark job execution
sudo python3 -m pip install \
    pandas==1.5.3 \
    numpy==1.23.5 \
    boto3==1.26.0 \
    requests==2.28.2 \
    pg8000==1.29.4

Note: Make sure to start the script with #!/bin/bash and ensure your S3 bucket permissions allow the EMR EC2 instance profile to read the script.

2. EMRFS (EMR File System)

EMRFS is an implementation of the Hadoop File System (HDFS) that allows Amazon EMR clusters to read and write data directly to and from Amazon S3 as if it were a local HDFS file system.

Why Use EMRFS Instead of Traditional HDFS?

1. Decoupled Compute and Storage: You don't need to keep a giant Core node cluster running simply to preserve HDFS storage. You store data cheaply on S3 and scale compute nodes (Task nodes) dynamically.
2. Persistence: Since the data is in S3, you can safely terminate your EMR cluster when your jobs finish, without losing any output.
3. High Durability: S3 provides 99.999999999% (11 9s) durability natively.
4. Data Sharing: Other services (like Athena, SageMaker, or Redshift Spectrum) can query S3 data concurrently.

3. EMRFS Object Store Caching (EMR 6.x+)

Starting with EMR 6.x, EMR introduced Object Store Caching for EMRFS.

• How it works: It automatically caches frequently accessed S3 data in the local NVMe SSDs or RAM of your Core and Task compute nodes.
• Benefits: Can speed up subsequent Spark queries on the same S3 tables by up to 10x, while significantly reducing Amazon S3 API transaction costs.

4. EMR Serverless (Modern Option)

For developers looking to avoid the complexities of cluster sizing, bootstrap scripts, and node management, AWS introduced EMR Serverless.

• What it is: An on-demand serverless engine running Apache Spark and Hive.
• Workflow:
• You create an EMR Serverless "Application" (specifying the Spark/Hive version).
• You submit a job run specifying your PySpark script and arguments.
• AWS automatically starts worker instances, executes the script, scales up workers as demanded by Spark's dynamic allocation, and stops them when the script finishes.
• Pricing: Billed per-second based on the total vCPU, Memory, and Storage resources consumed by your active workers. No charge when jobs are idle.