Creating EMR and PaaS API

There are three primary ways to provision an Amazon EMR cluster: the AWS Management Console (Web UI), the AWS CLI, and programmatically using the AWS SDK (Boto3)—often referred to as EMR's PaaS API.

1. Prerequisites for EMR Cluster Creation

Before launching an EMR cluster, ensure you have configured:

1. EMR Service Role (EMR_DefaultRole): Gives EMR permission to provision and manage AWS resources (like EC2 instances) on your behalf.
2. EC2 Instance Profile (EMR_EC2_DefaultRole): Assigned to the EC2 instances in your cluster, allowing them to access S3 buckets, Glue catalogs, and CloudWatch logs.
3. VPC and Subnets: EMR runs inside a VPC. Make sure your subnets have routes to access S3 (via VPC S3 Endpoint) and other systems.
4. Key Pair: An Amazon EC2 key pair for SSH access (optional but highly recommended for debugging).

2. Option A: Creating EMR via the AWS Console

1. Navigate to Amazon EMR in the AWS Console.
2. Click Create cluster.
3. Software Configuration: Select your EMR release (e.g., emr-6.10.0 or newer) and the applications you need (e.g., Spark, Hadoop, Tez, Hive).
4. Hardware Configuration:
5. Select VPC and Subnet.
6. Choose Instance Groups or Instance Fleets.
7. Select Instance types for Primary, Core, and Task nodes.
8. Security Configuration: Select your EC2 key pair, EMR service role, and instance profile.
9. Click Create cluster.

3. Option B: Creating EMR via the AWS CLI

You can launch a cluster using a single command in your terminal. Here is an example of creating a transient cluster that installs Spark and runs a step:

aws emr create-cluster \
    --name "My Spark Cluster" \
    --release-label emr-6.10.0 \
    --applications Name=Spark Name=Hadoop \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole,KeyName=my-ec2-keypair \
    --instance-groups \
        InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m5.xlarge \
        InstanceGroupType=CORE,InstanceCount=2,InstanceType=m5.xlarge \
    --use-default-roles \
    --auto-terminate

Note: --auto-terminate ensures the cluster automatically shuts down once all steps have finished executing.

4. Option C: Creating EMR Programmatically via Python PaaS API (Boto3)

In enterprise workflows, clusters are frequently created programmatically inside Python microservices, AWS Lambda functions, or orchestration tools.

Here is the production-grade script to create an EMR cluster using the Python Boto3 library:

create_emr_cluster.py

import boto3
import logging

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

def launch_emr_cluster():
    emr_client = boto3.client('emr', region_name='us-east-1')

    try:
        response = emr_client.run_job_flow(
            Name='Production-Spark-ETL-Cluster',
            ReleaseLabel='emr-6.10.0',
            Instances={
                'InstanceGroups': [
                    {
                        'Name': 'Master Node',
                        'Market': 'ON_DEMAND',
                        'InstanceRole': 'MASTER',
                        'InstanceType': 'm5.xlarge',
                        'InstanceCount': 1,
                    },
                    {
                        'Name': 'Core Nodes',
                        'Market': 'ON_DEMAND', # Or 'SPOT'
                        'InstanceRole': 'CORE',
                        'InstanceType': 'm5.xlarge',
                        'InstanceCount': 2,
                    },
                    {
                        'Name': 'Task Compute Nodes',
                        'Market': 'SPOT',  # Using spot to optimize costs
                        'InstanceRole': 'TASK',
                        'InstanceType': 'r5.xlarge',
                        'InstanceCount': 2,
                    }
                ],
                'Ec2KeyName': 'my-ec2-keypair',
                'KeepJobFlowAliveWhenNoSteps': False, # Terminate cluster when finished
                'TerminationProtected': False,
                'Ec2SubnetId': 'subnet-0bb123456789abcde', # Put your subnet ID here
            },
            Applications=[
                {'Name': 'Spark'},
                {'Name': 'Hadoop'}
            ],
            Configurations=[
                {
                    'Classification': 'spark',
                    'Properties': {
                        'maximizeResourceAllocation': 'true' # Dynamic Spark resource optimization
                    }
                },
                {
                    'Classification': 'spark-defaults',
                    'Properties': {
                        'spark.serializer': 'org.apache.spark.serializer.KryoSerializer',
                        'spark.dynamicAllocation.enabled': 'true'
                    }
                }
            ],
            Steps=[
                {
                    'Name': 'Run PySpark ETL Job',
                    'ActionOnFailure': 'TERMINATE_CLUSTER', # Terminate cluster if task fails
                    'HadoopJarStep': {
                        'Jar': 'command-runner.jar',
                        'Args': [
                            'spark-submit',
                            '--deploy-mode', 'cluster',
                            's3://my-etl-scripts-bucket/spark_jobs/sample_pyspark_job.py',
                            '--input', 's3://my-data-bucket/input/',
                            '--output', 's3://my-data-bucket/output/'
                        ]
                    }
                }
            ],
            BootstrapActions=[
                {
                    'Name': 'Install Custom Python Packages',
                    'ScriptBootstrapAction': {
                        'Path': 's3://my-etl-scripts-bucket/bootstrap/install_packages.sh'
                    }
                }
            ],
            ServiceRole='EMR_DefaultRole',
            JobFlowRole='EMR_EC2_DefaultRole',
            LogUri='s3://my-emr-logs-bucket/logs/'
        )

        cluster_id = response['JobFlowId']
        logger.info(f"Successfully launched EMR Cluster. Cluster ID: {cluster_id}")
        return cluster_id

    except Exception as e:
        logger.error(f"Failed to launch EMR Cluster: {str(e)}")
        raise e

if __name__ == "__main__":
    launch_emr_cluster()

Key API Parameters Explained:

• KeepJobFlowAliveWhenNoSteps:
• If set to True, the cluster remains running indefinitely after your jobs complete. Perfect for persistent interactive environments.
• If set to False, the cluster automatically terminates when all processing steps finish. Recommended for automated batch ETL to save money.
• Steps: An array of commands to execute. EMR uses a special built-in command-runner.jar file to execute generic commands like spark-submit on the cluster's primary node.
• Configurations: Allows you to pass complex configurations to Spark or Hadoop components directly at launch, avoiding the need to edit configuration files on nodes manually.