tag
6 items · 6 projects
A multi-tenant data-quality platform: a Python SDK that runs inside each team's Databricks workspace, a FastAPI + Streamlit control plane, a fact/dim PostgreSQL store for results, threshold-based Slack alerting, and a no-code UI for non-engineers. Built on Great Expectations with a custom-expectations API for team-contributed rules.
Refactored a vehicle-telemetry processing pipeline. Transforms raw nested JSON / protobuf from millions of in-car navigation clients into clean Delta tables that power navigation-quality dashboards. Designed for query simplicity: PMs answer 'route success rate in country X this week' with a single SELECT.
Architected an enterprise data lake on Google Cloud (BigQuery, Cloud Storage, Cloud Composer / Airflow) at Oyo Vacation Homes. Designed a metadata-driven ETL framework in PySpark that cut new-pipeline development time by ~70%. Established schema-versioning and quality-validation patterns the team still uses.
A PySpark-based Python package abstracting the I/O patterns between Azure Databricks and the half-dozen sinks an enterprise pipeline typically needs. Built during the ABN AMRO Bank engagement and open-sourced internally there.
Lead engineer on an enterprise data-quality platform with a pluggable connector framework for profiling, validation, and reconciliation across heterogeneous sources. Adopted across multiple internal customer engagements.
Optimised long-running ETL pipelines for Citi Bank Singapore (TCS engagement). Cut Data Stage job execution from 4 hours to 1 hour through Python multiprocessing + PySpark parallelisation. Designed real-time message handling with Kafka, plus AVRO / Parquet conversion and HDFS compression strategies for the bank's machine-learning data pipelines.