Scalable Uniques in Postgres - HyperLogLog

on YouTube ↗12m

" summary

Shows how the postgresql-hll extension implements HyperLogLog inside Postgres to count uniques, claiming a 280 bytes sketch can estimate tens of billions of distinct values with a few percent error. Describes tuning knobs (log2 m and bits-per-register), automatic switch between explicit and sparse representations, how to insert via functions, and that it was built and used in ad-reporting work with practical accuracy and small storage, though the original product ceased after acquisition.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to the problem
2. 1:32Approximate counting
3. 3:24Implementation details
4. 5:49Performance and tuning
5. 8:26Efficiency and use cases
6. 10:44Q&A and system capabilities

" related talks

• 49mDistributed COUNT(DISTINCT) with HyperLogLog on PostgreSQLBurak Yucesoy · PostgresOpen Silicon Valley 2017
• 26mBreaking Down Our Billion User Reach with HyperLogLogDaniel Avramson · Reversim Summit 2019
• 29mPostgres DemystifiedCraig Kerstiens · MountainWest RubyConf 2013
• 28mBreakout SessionGavin McQuillan · PGConf Silicon Valley 2015
• 37m​Postgres at Any Scale​Will Leinweber · RailsConf 2017
• 24mFun With UUIDsChris Ellis · POSETTE: An Event for Postgres 2025
• 25mPetabyte-scale Log management in Postgres using BaggerChris Travers · FOSSASIA PGDay 2025
• 30mLesser known features of PostgreSQLHaki Benita · Citus Con: An Event for Postgres 2022