Hydra - An Open Source Processing Framework
summary
Hydra is an open-source, document-processing pipeline built around MongoDB that reads pipelines from the DB and spins independent Java workers for each stage. It supports connectors, per-document fault isolation with automatic retries, and realtime stages plus batch MapReduce jobs (for things like PageRank). Enrichment uses simple regular expressions plus NLP (language detection, sentiment, entity extraction) and scales from tiny installs to customers processing tens of millions of documents per day; source code lives on GitHub.
AI-written summary · about
chapters (17)
- 0:00Introduction to Hydra
- 1:05About our company
- 2:06Generic search architecture
- 3:52Document enrichment challenges
- 7:50Metadata extraction techniques
- 9:18File server indexing use case
- 10:02From scripts to pipelines
- 12:15Problems with PDF parsing
- 13:19Hydra design objectives
- 15:05Development simplicity
- 16:32Hydra architecture overview
- 17:57MongoDB as processing buffer
- 18:26Batch vs real-time processing
- 19:31Pipeline flexibility with MapReduce
- 21:45Future of Hydra development
- 22:48Open sourcing the framework
- 23:57Q&A session
related talks
41mHydra's Heads – Vladimir Ozerov
43mHail Hydrate! From Stream to Lake Using Open Source
14mChange streams act on events in realtime
44mAutomatically scale a serverless app with Amazon Textract & MongoDB
46mNoSQL, Data Processing, and Business Transparency with RavenDB
45mSafely Capture Institutional Knowledge Using
50mApache Hudi : The Data Lake Platform
58mSphinx 2013