New Features in Apache Spark 4.0
summary
Lists concrete changes and examples: Python custom data sources let one load Hugging Face datasets with spark.read.format('huggingface').load; ANSI mode enabled by default turns silent overflows and divide‑by‑zero into exceptions; the new Variant type claims up to eight times faster JSON ops; Spark Connect decouples client and server over gRPC with Arrow record batches, enabling Rust/Go/R connectors that write Parquet/Delta; to_arrow returns PyArrow tables for small transfers; XML reader, parameterized queries that sanitize and auto‑register DataFrames, Python UDTFs, and exposed HyperLogLog sketches for reaggregation.
AI-written summary · about
chapters (17)
- 0:00Spark 4.0 new features
- 0:44Spark 4.0 key features
- 1:47Python custom data sources
- 3:31ANSI SQL safety improvements
- 6:59Variant data type for JSON
- 8:05Spark Connect architecture
- 10:12Spark Connect Rust demo
- 12:21Rust CSV to Delta conversion
- 13:26Go language connector
- 15:12R connector for Spark
- 16:19Spark Connect local setup
- 18:06DataFrames to Arrow tables
- 19:33Native XML support
- 20:36Parameterized SQL queries
- 23:50Python user-defined table functions
- 26:21HyperLogLog sketches
- 27:56Q&A
related talks
47mThe future of data: Apache Spark 3.1, Delta Lake and Koalas
29mFaster PySpark with Apache Arrow
33mArrow in Flight - New Developments in Data Connectivity
34mLeveraging Apache Arrow for ML Workflows
40mScaling with Apache Spark (or a lesson in unintended consequences)
01h 48mBig Data made easy with a Spark (3.0)
50mBig Data Processing and Data Science Environment in Python
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API