project · 2018-2019
PySpark ETL optimisation
Improved the reliability and throughput of data pipelines by reworking bottlenecks, parallelism, storage formats, and streaming hand-offs.
The problem
Several data-processing jobs had long runtimes and fragile hand-offs, delaying downstream machine-learning workloads.
Approach
- Profiled pipeline stages to isolate CPU, I/O, and serial-processing bottlenecks.
- Reworked the heavy transformations with PySpark and appropriate parallelism.
- Applied storage and compression choices based on access patterns, and added streaming patterns for time-sensitive messages.
Outcome
The resulting pipelines had a shorter critical path and a clearer set of performance and reliability trade-offs for future work.