tag
4 items · 4 feed items
Watermarks, exactly-once, and stream-stream joins each have a sentence in the docs and a paragraph of corner cases in production. Here's the gap between what the API says and what bites you at 3am — across Flink and Spark Structured Streaming.
From batch to streaming, the gotchas. Watermarks that drop too aggressively, late events that get silently lost, stateful aggregations that grow without bound, and the four operational habits that keep streaming jobs healthy in production.
Why your Spark job has 200 partitions even when your data has 5 GB. How to pick a target partition size, when to repartition vs coalesce, when AQE saves you, and the diagnostic loop I run on every slow job now.
How to know your job is skewed (the Spark UI lies more than you think), and the three fix patterns I reach for in production: salt-and-aggregate, broadcast join, and AQE-driven dynamic shuffle.