project
Self-service data-quality platform
Designed a self-service platform for defining, running, monitoring, and investigating data-quality checks across distributed data workloads.
tag
#data-engineering
20 items · 9 projects · 11 feed items
projects
project
Telemetry transformation layer
Refactored event-processing pipelines into a governed, queryable transformation layer for trusted operational analytics.
project
Schema-aware data validator
Built a Python validation library that turns sample events into editable contracts with reusable field-level quality rules.
project
ETL template for usage analytics
Refactored a fragmented set of analytics pipelines into a reusable Python template with consistent ingestion, transformation, testing, and operational behaviour.
project
Usage-analytics API layer
Built a typed API layer that turns analytical warehouse queries into reliable, self-service usage reporting for product and support teams.
project
Enterprise data lake on GCP
Architected an enterprise data lake with a metadata-driven PySpark framework, schema versioning, and reusable data-quality patterns.
project
Databricks connector library
Reusable PySpark library that standardised reading, writing, configuration, and error handling across common enterprise data stores.
project
Cloud-native data-quality platform
Led development of a configurable data-quality platform with reusable connectors for profiling, validation, and reconciliation across varied sources.
project
PySpark ETL optimisation
Improved the reliability and throughput of data pipelines by reworking bottlenecks, parallelism, storage formats, and streaming hand-offs.
feed
post
Metadata is product infrastructure, not documentation
A practical pattern for turning scattered technical knowledge into queryable metadata: stable identities, version history, clear ownership, and views people can actually use.
post
Why data contracts matter more than dashboards
Dashboards usually reveal a data problem after it has travelled through half the platform. Data contracts move the conversation upstream: clear schemas, ownership, compatibility rules, and a better way to change pipelines.
post
Streaming pipelines: the corner cases that the docs gloss over
Watermarks, exactly-once, and stream-stream joins each have a sentence in the docs and a paragraph of corner cases in production. Here's the gap between what the API says and what bites you at 3am — across Flink and Spark Structured Streaming.
post
Medallion architecture: the rules I now follow for bronze, silver, gold
After three years of building and refactoring medallion data lakes, here's the opinionated rule set that holds up: what bronze should and should not do, what makes silver actually queryable, and how to keep gold from drifting into chaos.
post
Spark Structured Streaming: watermarks, late data, and the mistakes I made
From batch to streaming, the gotchas. Watermarks that drop too aggressively, late events that get silently lost, stateful aggregations that grow without bound, and the four operational habits that keep streaming jobs healthy in production.
post
Architecture series, part 2: Kappa architecture, the reaction
Part 2 of a 2-post series. Kappa architecture is what you get when you ask 'what if we just did everything as a stream and replayed from the event log when we need to reprocess?' One pipeline, no diverged-codebase pain. Here's how it works, where it shines, and where it still bites.
post
Spark partition sizing: what I learned the hard way
Why your Spark job has 200 partitions even when your data has 5 GB. How to pick a target partition size, when to repartition vs coalesce, when AQE saves you, and the diagnostic loop I run on every slow job now.
post
Building a programmatic data-quality platform that teams actually adopt
Architectural pattern for a multi-tenant data-quality platform across many teams. Why centralising contracts beats centralising data, and how the SDK + control plane + fact/dim store pattern works regardless of which tools you reach for.
post
Delta Lake MERGE patterns I wish I'd learned a year earlier
Five MERGE patterns that solved real problems for me: idempotent upsert, soft delete, late-arriving data, deduplication on ingest, and the slowly-changing-dimension type-2 case. Plus the performance gotchas that bit me first.
post
Architecture series, part 1: Lambda architecture, where it came from and why it hurt
Part 1 of a 2-post series on streaming-data architectures. Lambda architecture solved the right problem in 2014: how to combine batch correctness with stream-low-latency. The cost was running two pipelines for the same logic. Here's why it was right then, and why most teams shouldn't pick it now.
post
PySpark skew detection, and the three fixes that actually work
How to know your job is skewed (the Spark UI lies more than you think), and the three fix patterns I reach for in production: salt-and-aggregate, broadcast join, and AQE-driven dynamic shuffle.