Large scale graph analysis using Scala and Akka
summary
Built a distributed time‑series graph engine called Athena graph using Scala, Akka and HBase with a pattern‑match query parser, actor pools and depth‑first traversal that emits streaming results to Kafka. HBase‑backed runs reach ~8,000 results/sec for three‑hop queries; after loading the graph into memory and using a consistent hashing router it reaches ~75,000 results/sec. Main limits are HBase bottleneck and at‑most‑once actor delivery; durable WAL/at‑least‑once delivery and caching are suggested fixes.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:50THE NEED GET DATA FOR ANALYTICAL AND OPERATIONAL PURPOSE
- 1:22THE NEED DET DATA FOR ANALYTICAL AND OPERATIONAL PURPOS
- 2:13THE PROBLEM
- 3:05LIKE A DATABASE BUT A BIT MORE...
- 3:56WHY A GRAPH
- 4:48PATTERN MATCHING GRAPH QUERY
- 5:52DATA MODEL IN HBASE
- 7:32MAIN COMPONENTS/CONCEPTS
- 10:35NODE ACTOR
- 13:40OPTIMIZATIONS
- 16:44CONSISTENT HASHING/IN-MEMORY GRAPH
- 19:18ACTOR NODE
- 20:15END TO END
- 25:01WHEN DOES IT END?
- 27:12WHAT NEXT
- 28:25WE ARE HIRING
- 29:19WHY NOT [X] YOU SHOULD HAVE USED
- 31:06ACTOR SYSTEM/CLUSTER
related talks
26mLarge Scale Graph Processing & Machine Learning Algorithms
55mExperience report on using Akka in production: pitfalls and rewards
19mBig Data
41mAkka in Production: Our Story
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
51mComposing high performance process workflows with Akka Streams
56mGraph Analytic Systems
48mThe three paths of Scaling with Akka