Safely Capture Institutional Knowledge Using
summary
Project ingests about 5 million documents (≈6.5 TB, ~16 million PDF pages) via Apache NiFi into MarkLogic, extracts text with the document filter, enriches with Azure AI OCR/image tagging and Benchling exports, and runs CORB for batch work while serving search through Oracle APEX. Progress Semaphore classifies PII so about 1% are personal and 1% sensitive, triggers role-based lockdowns and a flagging workflow. A successful Retrieval-Augmented Generation proof-of-concept returns LLM answers from chunked documents and already recovers lost work (one case saves ~$200k).
AI-written summary · about
chapters (14)
- 0:00Introduction and agenda
- 2:43Loss of institutional knowledge
- 3:25Choosing Mark Logic solution
- 5:11Architecture overview
- 6:37Document ingestion process
- 10:09Metadata harmonization
- 11:14AI enrichment and OCR
- 14:03Security and PII handling
- 16:32Progress Semaphore for PII
- 23:56Document classification for UI
- 26:25Success stories and value
- 27:29Lessons learned
- 29:43Future plans with RAG and LLM
- 31:50Q&A session
related talks
32mDocuments Meet LLMs: Tales from the Trenches
01h 19mLearn to Unlock Document Intelligence with Open-Source AI
15mGraphRAG-Powered AI Agents: Transforming Corporate Metadata into Intelligent Knowledge
22mStructuring the Unstructured : Empowering the AI Workflows
36mTechniques to avoid common LLM pitfalls
30mBuilding a Conversational Knowledge Base
10mKeynote: Unlocking Corporate Data with Retrieval-Augmented Generation
06mInterviewed at Gov 2.0 – Paul Norcini