Inside a Spark Job: Stages, Shuffles, AQE and Memory, and How to Read Them
Apache Spark makes distributed processing look like ordinary DataFrame code, which is both its strength and the source of most production problems. A job that took ten minutes last month takes two…