tag
5 items · 5 feed items
Watermarks, exactly-once, and stream-stream joins each have a sentence in the docs and a paragraph of corner cases in production. Here's the gap between what the API says and what bites you at 3am — across Flink and Spark Structured Streaming.
From batch to streaming, the gotchas. Watermarks that drop too aggressively, late events that get silently lost, stateful aggregations that grow without bound, and the four operational habits that keep streaming jobs healthy in production.
Part 2 of a 2-post series. Kappa architecture is what you get when you ask 'what if we just did everything as a stream and replayed from the event log when we need to reprocess?' One pipeline, no diverged-codebase pain. Here's how it works, where it shines, and where it still bites.
Final part of the series. Generators turn an O(N) memory problem into an O(1) memory problem and let you compose pipelines that read like sentences. Five patterns: streaming files, paginated APIs, itertools chains, batching, and async generators.
Part 1 of a 2-post series on streaming-data architectures. Lambda architecture solved the right problem in 2014: how to combine batch correctness with stream-low-latency. The cost was running two pipelines for the same logic. Here's why it was right then, and why most teams shouldn't pick it now.