Data Classification at Scale: Taming the Hydra
summary
Describes a bytes-up approach that builds a taxonomy with semantic types and actor facets, scans assets via enumerator → scheduler → scanner, extracts feature vectors and runs heuristics, logistic regression, ML classifiers and occasional LLM postprocessing, then calibrates scores into a metas store and asset catalog. System processes trillions of samples per day and makes billions of predictions per day. Ground truth is asset-level labeled, sampling is stratified and iterative, start with high precision then raise recall, centralize taxonomies/classifiers and federate enumeration.
AI-written summary · about
chapters (7)
- 0:00Introduction and context
- 4:33Taxonomy and structure
- 6:40System overview
- 8:03Classification challenges
- 11:07Classification approaches
- 12:51Ground truth and measurement
- 15:48Lessons learned
related talks
44mContent Classification: NLP vs. AI in the Age of Data Vectorization
37mBuilding a Scientific Taxonomy at Scale
30mA study in classification
28mData Curation: Transforming Bytes into AI Gold
30mData Driven Security
30mMastering Unstructured Data with AI and ML
38mE-Commerce Product Categorization at Scale
29mData management for enterprise-grade foundation models.