How multilingual is your NLP model?
summary
Off-the-shelf language detectors like cld3 and langid frequently misclassify short, transliterated or code-switched text (examples show cld3 returning Maori or Finnish for Spanglish/Hindi input), and some tools are non-deterministic. Multilingual BERT (trained on 104 languages with a ~120k wordpiece vocab) handles long, contextual code-switch examples well but needs fine-tuning. Practical pipeline: decide target languages, augment scarce data with rule-based or MT-derived transliterations (built a simple Hindi transliterator from Wikipedia), then fine-tune mBERT and evaluate with masked‑LM top‑k and generation metrics.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 0:38About me
- 1:21Outline
- 2:04Why multilingual data
- 2:44Tasks associated with language systems
- 4:08Syntax mixing
- 6:11Transliterated text
- 7:23Language identification
- 8:45Language identification in practice
- 10:19Other examples
- 12:46Lambda ID
- 14:42Blanked
- 15:54Python
- 16:41Limitations
- 18:24Data augmentation
- 20:38Simple example
- 22:50The Transformer
- 24:09Multiheaded attention
- 26:17Stateoftheart soda
- 26:57Why is it special
- 28:13Word Piece Processing
- 30:45Statistics of Languages
- 32:55Bird Masked Language Model
- 33:42Prediction Function
- 36:31Code Switched Example
- 38:05Lyrics Example
- 39:37Task Evaluation
- 40:45Generation Evaluation
- 42:45Summary
related talks
20mPolyglot AI: The Role of Natural Language Processing (NLP)
36mScaling Data Beyond English: Multilingual Scaling and Data Mixing Laws
43mBreaking the language barrier: how do we quickly add multilanguage support in our AI application?
26mModern NLP for Proactive Harmful Content Moderation
22mHow to expand your NLP Solution to new languages
23mMultilingual Natural Language Processing
40mPanel discussion "NLP"
40mThe Future of Language Models