Apache Spark for a Distributed Database: Inside the Connector

on YouTube ↗45m

" summary

An overview of the Apache Spark connector developed for the Tarantool distributed database, comparing connector architectures for different databases and discussing whether Spark is suitable for multi-node distributed databases and what alternatives exist. The talk examines how to align Spark cluster configuration with the database cluster settings to achieve optimal performance, aimed at driver/connector developers and Scala programmers.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 3:11Why Spark connector
3. 10:00How Spark works
4. 15:26What is a connector
5. 19:48Task serialization issues
6. 27:09Working with connections
7. 31:10Distributed DB topologies
8. 37:19Sharding and partitions
9. 44:55Conclusion and Q&A

" related talks

• 33mTo Spark or Not to SparkRussell Alexander Spitzer · Scale By The Bay 2019
• 05mInterviewRussell Alexander Spitzer · Scale By The Bay 2019
• 46mSpark: One Tool for Everything?Iskander Fakhrutdinov · Saint HighLoad++ 2025
• 55mUse Spark from AnywhereDenny Lee, Ginger Holt · Scala Days 2023 Seattle
• 02h 18mDistributed Systems: In the Daily Life of the Data EngineerFrançois Sarradin · Devoxx France 2024
• 50mIntroduction to Real-Time Big Data with Apache SparkTaras Matyashovsky · JavaDay Lviv 2015
• 54mHarnessing the Power of Spark & Cassandra within your Spring AppSteve Pember · Spring I/O 2017
• 06mInterviewRussell Alexander Spitzer · Scale By The Bay 2018