project · 2019-2021
Databricks connector library
Reusable PySpark library that standardised reading, writing, configuration, and error handling across common enterprise data stores.
The problem
Data pipelines repeatedly reimplemented connection setup, schema handling, and cleanup logic for the same storage systems. The duplication made reviews slower and failures inconsistent.
Approach
- Packaged common PySpark I/O patterns behind small, well-documented interfaces.
- Centralised configuration, connection lifecycle management, and error semantics.
- Made connectors composable so pipelines could focus on their business transformations.
Outcome
The library reduced boilerplate and gave new pipelines a consistent, versioned foundation without prescribing their transformation logic.