Data Fabric

on YouTube ↗23m

" summary

IDfy runs a home‑grown data fabric that ingests per‑service instrumentation (usage, turnaround time, exceptions) in real time, handling about 1M API hits/day and ~100k video KYC profiles/day with 10x growth expectations. The stack uses Open Source pieces such as logstash and Apache Beam, plus WebRTC for video KYC, and builds a Live Support Desk product that traces drop‑offs, attributes failures to business vs technical causes, and plans to open‑source the fabric layer.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 3:01From trading exchanges to platforms
3. 11:07Defining data fabric and its necessity
4. 13:54Instrumentation methodology
5. 16:47Data fabric architecture
6. 20:38Real-time processing
7. 21:42Open source vision

" related talks

• 09mReal-time Intelligence in Microsoft FabricTessa Kloster, Wangui McKelvey · Microsoft Build 2024
• 26mReal Time analytics - Sense analyze generate insights with FabricKevin Lam, James Hutton, Tzvia Gitlin · Microsoft Build 2023
• 33mArchitecting for plasticity - Demystifying the data fabricKaranjot Jaswal · Big Data & AI Toronto 2020
• 41mReal time identity graphFakrudeen Ali Ahmed · Flink Forward Virtual 2020
• 42mReimagine Real-Time Intelligence with Microsoft FabricTessa Kloster, Alan Grogan, Yitzhak Kesselman, Cem Urfalioglu · Microsoft Build 2024
• 53mData Mesh in Practice - Assuring Data Quality at ScaleGayathri Thiyagarajan · DDD Europe 2022
• 50mMicrosoft Fabric Data Factory Q&AWee Hyong Tok, Olivia Shea, Nellie Gustafsson, Justyna Lucznik, Shawndell Elfring, Miquella de Boer, Chris Finlan, Cristin Ford, Mark Kromer, Martin Lee, Piero Morano · Microsoft Build 2023
• 19mAccelerate your data potential with FabricSwetha Mannepalli · Microsoft Build 2023