project
PySpark ETL optimisation
Improved the reliability and throughput of data pipelines by reworking bottlenecks, parallelism, storage formats, and streaming hand-offs.
tag
#kafka
3 items · 1 project · 2 feed items
projects
feed
post
Streaming pipelines: the corner cases that the docs gloss over
Watermarks, exactly-once, and stream-stream joins each have a sentence in the docs and a paragraph of corner cases in production. Here's the gap between what the API says and what bites you at 3am — across Flink and Spark Structured Streaming.
post
Architecture series, part 2: Kappa architecture, the reaction
Part 2 of a 2-post series. Kappa architecture is what you get when you ask 'what if we just did everything as a stream and replayed from the event log when we need to reprocess?' One pipeline, no diverged-codebase pain. Here's how it works, where it shines, and where it still bites.