How we build InsightEdge: powerful Spark distribution
summary
InsightEdge runs on top of a GigaSpaces XAP in‑memory data grid and claims 30 times faster than vanilla Spark by co‑locating compute and data and using bucket-based partitioning to map many Spark partitions onto grid buckets. It implements a custom Spark DataSource to push predicates and projections into the grid, supports Spark Streaming with transactional context, exposes a Scala-friendly API, adds geo-spatial queries with indexes, and validates clusters via Docker integration testing while noting Java/Scala UDT interoperability limits.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 6:31InsightEdge architecture
- 11:13Integration workflow
- 21:54Implementation details
- 36:54Development challenges
- 41:10Testing and code quality
- 44:00Community and future
related talks
11mHow Companies are Using Spark, and Where the Edge in Big Data Will Be
39mEdge Compute Infrastructure for Experience Business
50mDesigning a scalable data platform based on Apache Spark/Iceberg
19mBig Data
59mSpark 2.0
50mData Science in the Enterprise
44mData Science at Scale with Apache Spark
07mBuilding an Insights Factory