Deciphering doctor's handwriting with deep learning

on YouTube ↗51m

" summary

Project trains on ~441k labeled line images (scans anonymized, heavy image cleaning) and uses a five-layer CNN + 2 RNNs + CTC ensemble trained on Google Cloud (~9 hours for largest model). Baseline Google Vision 9.6%; model reaches ~54% exact line accuracy for death-certificates, flags high-confidence predictions to auto-handle ~30% of certificates, recovers ~20–30% of previously illegible entries. For insurance forms: codes 94% correct, dates 75%, amounts ~60% via two-step trick, grid detection 96% empty/84% line-count, stamp detection+read ~77% end-to-end.

AI-written summary · about

" chapters (45)

1. 0:00Intro
2. 0:34Business goal
3. 1:41Case 2: the Doctor's Consultation Certificate
4. 2:04Case 1: the Death Certificate
5. 3:12Data pipeline
6. 3:48Mining
7. 5:15Original image
8. 6:45Line segmentation
9. 7:32Baseline
10. 8:47Training set definition
11. 10:11Data quality
12. 12:04Data partitioning: split into text lines
13. 13:52Ensemble modelling
14. 15:00Model Training Performance
15. 15:18Model accuracy
16. 16:05Prediction confidence levels
17. 16:43Confidence plot
18. 19:47Examples target population
19. 20:01Examples false positives
20. 20:59Statistics false positives
21. 21:38Examples low confidence
22. 26:18Example: type A
23. 28:11Language detection
24. 28:37Certificate type detection
25. 29:08Preprocessing
26. 29:39Training data
27. 30:14Nomenclature code
28. 30:50Date of Consultation
29. 34:43Comparison OCR - Neural Network
30. 35:06The Grid (5)
31. 36:20Some grid examples
32. 37:23Approach
33. 39:29Examples: 2 lines
34. 40:21Training on histogram features
35. 41:40Results number of lines prediction
36. 42:14Reading the lines
37. 43:36Examples: high confidence false predictions
38. 44:22Grid summary
39. 45:00Second challenge: different patterns
40. 45:20Fourth challenge: rotations
41. 45:38Step 1: find the stamp
42. 46:29Intersection over Union
43. 47:57Summary stamp reading
44. 48:19Application Integration
45. 48:51Conclusion

" related talks

• 42mUsing Deep Learning to Understand DocumentsEitan Anzenberg · MLconf 2022 San Francisco
• 44mHello Deep LearningBert Hubert · GOTO Amsterdam 2023
• 32mDetecting Signed and Unsigned Documents with Deep Learning - Beyond Transfer...Jordan Bramble · PyData Los Angeles 2018
• 40mBuilding a Handwriting Recognition SystemShafik Quoraishee · droidcon New York 2025
• 55mDeep Learning in Medical Image DiagnosticsMahesh Balaji · ODSC India 2018
• 23mVision-Language Models for Fraud DetectionRenat Khizbullin · DSC Europe 24
• 15mBrain Tumor Detection and Classification Using Deep Learning ModelsSwaraj Sontakke · DevConf.IN 2026
• 44mBuilding Handwriting Recognition System for the NY Times CrosswordShafik Quoraishee · TestMu Conference 2025