All I want for Christmas is My Two Dot One

Spark 2.1-rc5 was marked as the official Spark2.1 just in time for Christmas. Streaming applications have got the Gold treatment, with Structured Streaming receiving a lot of attention in order to stabilize the API and engine. Event Time watermarks should make it much easier to deal with data arriving late. The Python API has gained a few new API features from Scala, and the API is very close now to being 1-for-1 with Scala for the majority of use cases. Performance has improved a bit in niche areas, especially ML (LinearRegression, RandomForest and K-Means). Expect much bigger performance changes in 2.2 (ETA April 2017) with Star Schema optimizations, and Cost Based Optimizations being the stars of the performance show.