Automatically categorizing news articles on nrk.no
summary
NRK builds an automated pipeline that cleans every article, trains fastText embeddings on the full NRK.no corpus, runs k-means on 400 clusters to map words to topics, then composes 100-d article vectors and trains a classifier with 57 IPTC-derived labels. It uses BigQuery, Dataflow, Dataproc/Spark, R for multinomial logistic regression, and serves predictions via Kubernetes into Elasticsearch; classifiers output per-category probabilities and run on each publish event.
AI-written summary · about
chapters (13)
- 0:00Start
- 5:04Group Content for External Purposes
- 5:53Two Duplicate Article Collections about the Same Topic
- 7:52Google Technologies
- 8:31Cloud Dataflow
- 10:25Create a Corpus
- 11:02Entity Detection
- 13:13Find a Mathematical Representation of an Article
- 13:26Word Embedding Algorithm
- 14:17Word Embeddings
- 24:37The Article Embedding
- 32:45Why Did You Do It in R
- 34:12Nrk Data and Article Classifier
related talks
36mMachine Learning in News Media: Case study 24sata.hr
21m"What's in the news?" #bbuzz
31mTaking AI/ML to the next level: Text!
34mApplying Machine Learning Techniques to Detect and Understand Fake News
26mUnsupervised learning for news summarisation
35mUnderstanding the News around the World with Web Scraping and NLP at Scale
52mMachine Learning Data Pipelines
21mFrom 0 to a crowd sourced neural network in a jiffy