Trillion-Token Pretraining
summary
Pretraining a payments foundation model on trillions of tokens builds a hierarchical transformer that first encodes per-transaction features then attends across sequences, running on on‑prem GPUs with Ray and multi‑node training. Key specifics: 120 million payments/day, 3,000 tps peak, BPE tokenizer + Fourier embeddings for amounts, correlated‑feature masking and time‑interval sinusoidal encodings. Results: 170% ARI uplift and 20% precision on identity resolution, 3× scripted‑attack detection, and ~3% AUC uplift on fraud when ensembled.
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 6:27From experiments to infrastructure
- 11:32Sequence modeling approach
- 19:10Key learnings and optimizations
- 24:06Infrastructure challenges and Ray setup
- 27:00Other AI team projects
- 31:10Q&A session
related talks
26mRethinking the Transformer: Toward Native Multimodal Architectures
25mTraining Large Language Models in the Open
34mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
13mScaling Generative AI Inference at Trillion-Token Scale
01h 38mTraining Large Language Models: Practices and Research Questions
11mLightning Talk: Trinity Large - Torchtitan on 2000+ B300s
16mRoad to 5 Million Tokens: Breaking Barriers in Long Context Training
34mBeyond Transformers - Intro to RWKV Architecture & The World To...