AI-Enhanced Big Data: Integrating Private LLMs and Vector Databases
summary
Claims that AI now automates data preparation and even writes Python analytics code, using exabytes of data (Instagram 55,000 photos/min, ~4.5M concurrent YouTube viewers) and combining Spark + Cassandra + Kafka for low‑latency pipelines with event sourcing and materialized views to hit sub‑20ms responses. Demonstrations use ChatGPT/OpenAI embeddings, LangChain, and Pinecone vector DB for RAG; shows chunking strategies (sliding windows, prepend titles), agent prototypes for sales/service, crop and sales demos that run and generate actionable outputs in production.
AI-written summary · about
chapters (11)
- 0:00Introduction to AI and Big Data
- 3:14Five V's of Big Data
- 10:03Case Studies in Big Data
- 17:34Uber’s Real-Time Data Architecture
- 22:30Data Analytics with AI Automation
- 28:33Retrieval Augmented Generation Explained
- 36:48Building AI Agents with RAG
- 45:58Product Recommendation Systems
- 50:35Fraud Detection Using Vector Databases
- 56:34Sentiment Analysis with AI
- 58:00Q&A and Summary
related talks
30mAI-Driven Operational Data Intelligence
46mEvent Streaming and its Potential for the GenAI World
50mBuilding AI ready applications
13mBuilding High-Performance AI Literature Processing
45mManaging Data & Analytics Projects Effectively
49mGenAI for Business on the Modern Data Stack
46mBusiness: AI as an enabler for unlocking Industry Data: Monetizing Enterprise Value
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)