tag
3 items · 3 feed items
Why your Spark job has 200 partitions even when your data has 5 GB. How to pick a target partition size, when to repartition vs coalesce, when AQE saves you, and the diagnostic loop I run on every slow job now.
How to know your job is skewed (the Spark UI lies more than you think), and the three fix patterns I reach for in production: salt-and-aggregate, broadcast join, and AQE-driven dynamic shuffle.
The mental model behind asyncio that finally clicked, the production patterns I reach for (TaskGroup, Semaphore, run_in_executor), and the four mistakes that bite every async codebase eventually.