Conventional ETL to Big Data Evolution
Understand the architectural shift from early centralized business mainframes and single-node SMP (Symmetric Multiprocessing) ETL engines to horizontally scaled distributed big data clusters.
The Monolithic Mainframe Era
In the early decades of enterprise data processing, centralized mainframes dominated data execution:
The Monolithic Model
A single massive mainframe frame housed all processing compute registers, memory channels, and storage arrays centrally.
Vertical Scaling Limit
Scaling up required purchasing expensive proprietary hardware components (RAM, CPU blocks) to fit into the single machine framework.
Exponential Cost Curves
Scaling vertically yields diminishing returns; physical heat boundaries and RAM speed latency make doubling capacity cost up to 10x more.
The Conventional ETL Era
As relational databases (RDBMS) and Data Warehouses (EDW) matured, specialized single-node ETL tools emerged (Informatica PowerCenter, Talend, Microsoft SSIS, IBM DataStage).
Sources
Warehouse
Critical Bottlenecks of SMP ETL:
1. Memory Boundary
Processing is constrained by a single server's RAM. Scale-up hardware swaps to disk, degrading performance exponentially.
2. Channel Saturation
SMP engines query data through a shared local storage bus. Concurrent processing threads quickly saturate physical I/O bandwidth.
3. Schema Rigidity
Traditional relational engines struggle with semi-structured JSON telemetry, nested records, or raw unstructured text logs.
The Pivot to Horizontal Scaling (Scale-Out)
To break vertical limits, modern software engineering pivoted to horizontal cluster scaling (Scale-Out) using commodity hardware.
Scale-Up (Vertical Scaling)
High resource ceiling on a single server. Capacity scaling hit physical RAM/CPU boundaries.
Scale-Out (Horizontal Scaling)
Nodes linked via local network; scaling capacity adds commodity hardware nodes linearly.
Horizontal scaling links low-cost standard servers together into a single logical cluster. When data volume doubles, new nodes are attached to the network to scale compute and storage linearly.