Lightning fast cluster computing with Cassandra and Spark
summary
A Scala connector library in Scala wires Apache Spark to Apache Cassandra so Spark jobs read and write Cassandra tables with data locality by co-locate Spark executors with Cassandra nodes, avoiding bulk data movement. It implements RDDs, pushdown filters to Cassandra and automatic case-class mapping and type converters, uses the Java driver underneath, and exposes practical limits encountered: non-serializable Scala closures, reflection and implicit-resolution pitfalls. The code is open source and intended for production use.
AI-written summary · about
chapters (12)
- 0:00Cassandra and Spark integration
- 1:08What Cassandra is
- 2:55Cassandra analytics limits
- 3:40Why Spark complements Cassandra
- 5:06Cassandra-Spark connector architecture
- 7:13Cassandra vs Spark data model
- 9:22How to use the Cassandra-Spark connector
- 12:14Connector implementation details
- 13:37Scala features we love
- 17:37Scala serialization challenges
- 19:49Lessons learned and future improvements
- 20:33Project timeline and status
related talks
40mSpark Analytics on Cassandra Data
55mSolving Classical Data Analytic Task by Using Modern Distributed Databases
29mCassandra - an introduction
21mChallenges of Spark Application coexisting with NoSQL databases
01h 48mBig Data made easy with a Spark (3.0)
36mData pipelines engineering made simple with Scala
31mApache Cassandra Analytics: A Recipe to Move Petabytes of Data
50mChallenges that everyone struggles while productionizing Apache Spark workloads