Accelerator-aware Scheduler ( SPARK-24615). We have curated a list of high level changes here, grouped by major modules. You can consult JIRA for the detailed changes. To download Apache Spark 3.0.0, visit the downloads page. Here are the feature highlights in Spark 3.0: adaptive query execution dynamic partition pruning ANSI SQL compliance significant improvements in pandas APIs new UI for structured streaming up to 40x speedups for calling R user-defined functions accelerator-aware scheduler and SQL reference documentation. This release improves its functionalities and usability, including the pandas UDF API redesign with Python type hints, new pandas UDF types, and more Pythonic error handling. PySpark has more than 5 million monthly downloads on PyPI, the Python Package Index. Python is now the most widely used language on Spark. In TPC-DS 30TB benchmark, Spark 3.0 is roughly two times faster than Spark 2.4. Various related optimizations are added in this release. These enhancements benefit all the higher-level libraries, including structured streaming and MLlib, and higher level APIs, including SQL and DataFrames. 46% of the resolved tickets are for Spark SQL. Spark SQL is the top active component in this release. Nowadays, Spark is the de facto unified engine for big data processing, data science, machine learning and data analytics workloads. 
Since its initial release in 2010, Spark has grown to be one of the most active open source projects. This year is Spark’s 10-year anniversary as an open source project. With the help of tremendous contributions from the open-source community, this release resolved more than 3400 tickets as the result of contributions from over 440 contributors. Apache Spark 3.0 builds on many of the innovations from Spark 2.x, bringing new ideas as well as continuing long-term projects that have been in development. This release is based on git tag v3.0.0 which includes all commits up to June 10. The vote passed on the 10th of June, 2020.
Apache Spark 3.0.0 is the first release of the 3.x line.
0 kommentar(er)
