project
Self-service data-quality platform
Designed a self-service platform for defining, running, monitoring, and investigating data-quality checks across distributed data workloads.
tag
#pyspark
9 items · 5 projects · 4 feed items
projects
project
Telemetry transformation layer
Refactored event-processing pipelines into a governed, queryable transformation layer for trusted operational analytics.
project
Enterprise data lake on GCP
Architected an enterprise data lake with a metadata-driven PySpark framework, schema versioning, and reusable data-quality patterns.
project
Databricks connector library
Reusable PySpark library that standardised reading, writing, configuration, and error handling across common enterprise data stores.
project
PySpark ETL optimisation
Improved the reliability and throughput of data pipelines by reworking bottlenecks, parallelism, storage formats, and streaming hand-offs.
feed
post
Spark Structured Streaming: watermarks, late data, and the mistakes I made
From batch to streaming, the gotchas. Watermarks that drop too aggressively, late events that get silently lost, stateful aggregations that grow without bound, and the four operational habits that keep streaming jobs healthy in production.
post
Spark partition sizing: what I learned the hard way
Why your Spark job has 200 partitions even when your data has 5 GB. How to pick a target partition size, when to repartition vs coalesce, when AQE saves you, and the diagnostic loop I run on every slow job now.
post
Delta Lake MERGE patterns I wish I'd learned a year earlier
Five MERGE patterns that solved real problems for me: idempotent upsert, soft delete, late-arriving data, deduplication on ingest, and the slowly-changing-dimension type-2 case. Plus the performance gotchas that bit me first.
post
PySpark skew detection, and the three fixes that actually work
How to know your job is skewed (the Spark UI lies more than you think), and the three fix patterns I reach for in production: salt-and-aggregate, broadcast join, and AQE-driven dynamic shuffle.