tag

#kafka

2 items · 1 project · 1 feed item

projects

project 2018-2019

PySpark ETL optimisation for Citi Bank Singapore

Optimised long-running ETL pipelines for Citi Bank Singapore (TCS engagement). Cut Data Stage job execution from 4 hours to 1 hour through Python multiprocessing + PySpark parallelisation. Designed real-time message handling with Kafka, plus AVRO / Parquet conversion and HDFS compression strategies for the bank's machine-learning data pipelines.

#data-engineering #pyspark #kafka

feed

post 2025-11-15

Architecture series, part 2: Kappa architecture, the reaction

Part 2 of a 2-post series. Kappa architecture is what you get when you ask 'what if we just did everything as a stream and replayed from the event log when we need to reprocess?' One pipeline, no diverged-codebase pain. Here's how it works, where it shines, and where it still bites.

#data-engineering #architecture #kappa #streaming #kafka #learnings

← all tags