When Computers Can't Math - Floating Point Rounding Error Explained
summary
Explains why 0.1 + 2 does not equal 3 by showing how IEEE 754 32-bit float splits into 1 sign bit, 8 exponent bits, 23 fraction bits, how 0.1 becomes a binary repeating fraction that must be rounded, and how those 23 stored bits produce a value slightly larger than 0.1 so adding yields a result fractionally above 3.0 rather than the exact integer three.
AI-written summary · about
related talks
51mFloating-Point Arithmetic on Deterministic Systems
48mFuture-Proofing Numerical Accuracy with Formal Specifications
10mNumeric Data Types in the Wild; or: Let’s Look at Last Night’s Dinner Receipt!
38mStrength in Numbers: Unums and the Quest for Reliable Arithmetic
08mYou Won’t Believe This One Weird CPU Instruction!
05mLightning Talk: Finding the Average of 2 Integers
05mBinary in JavaScript
01h 00mFloating-Point ＜charconv＞: Making Your Code 10x Faster With C++17's Final Boss