tag
6 items · 6 feed items
After three years of building and refactoring medallion data lakes, here's the opinionated rule set that holds up: what bronze should and should not do, what makes silver actually queryable, and how to keep gold from drifting into chaos.
From batch to streaming, the gotchas. Watermarks that drop too aggressively, late events that get silently lost, stateful aggregations that grow without bound, and the four operational habits that keep streaming jobs healthy in production.
Part 2 of a 2-post series. Kappa architecture is what you get when you ask 'what if we just did everything as a stream and replayed from the event log when we need to reprocess?' One pipeline, no diverged-codebase pain. Here's how it works, where it shines, and where it still bites.
Why your Spark job has 200 partitions even when your data has 5 GB. How to pick a target partition size, when to repartition vs coalesce, when AQE saves you, and the diagnostic loop I run on every slow job now.
Five MERGE patterns that solved real problems for me: idempotent upsert, soft delete, late-arriving data, deduplication on ingest, and the slowly-changing-dimension type-2 case. Plus the performance gotchas that bit me first.
Part 1 of a 2-post series on streaming-data architectures. Lambda architecture solved the right problem in 2014: how to combine batch correctness with stream-low-latency. The cost was running two pipelines for the same logic. Here's why it was right then, and why most teams shouldn't pick it now.