Transformers for extractive text summarization
summary
Builds an extractive summarizer for a corporate Data Lake using pretrained Transformers (BERT/RoBERTa variants) plus a small sentence-level Transformer, converts human abstractive summaries into labels with greedy selection, expands sequence length to ~1024 by copying the last positional embedding as a positional-embedding hack, and enforces trigram blocking to avoid redundancy. Fits a compression-ratio curve to pick summary length, chooses ELECTRA-small for production, and reports consistent ROUGE gains over the Lead-3 baseline.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:23Data Lake
- 5:46Three steps
- 7:41Rush metric
- 10:38Data set
- 11:35Greedy selection algorithm
- 14:03Mixed data
- 14:47Model
- 17:15Results
- 19:02Custom techniques
- 20:24Semantic redundancy
- 21:27Compression ratio
- 23:02Results metrics
- 24:06Questions
related talks
38mThe Art of Compression: Crafting Insightful Summaries with LLMs
57mNatural Language Processing 2020 Discussion
34mTransformers End to End Experience training...
49mBen Hall & Barbara Fusinska
52mTowards fast and coherent long-form text generation
01h 16mHow Do Transformers Work?
41mRepresentations from natural language data: successes and challenges
01h 01mKeynote: Transformers: The Rise of ChatGPT