tag
3 items · 1 project · 2 feed items
Refactored a vehicle-telemetry processing pipeline. Transforms raw nested JSON / protobuf from millions of in-car navigation clients into clean Delta tables that power navigation-quality dashboards. Designed for query simplicity: PMs answer 'route success rate in country X this week' with a single SELECT.
After three years of building and refactoring medallion data lakes, here's the opinionated rule set that holds up: what bronze should and should not do, what makes silver actually queryable, and how to keep gold from drifting into chaos.
Five MERGE patterns that solved real problems for me: idempotent upsert, soft delete, late-arriving data, deduplication on ingest, and the slowly-changing-dimension type-2 case. Plus the performance gotchas that bit me first.