So you'd like to import a CSV File...
summary
Code for importing EventBrite CSVs with standard C++ and Qt reads lines with std::getline and parses by character into std::string_view to avoid copies, then converts to QString only when inserting new attendees; this importer works and speeds up import. Surprises: IDs in the billions and the export order CSV not sorted so never assume ordering. Big gotcha: QString::fromUtf8 always allocates, so deferring conversion and keeping a preloaded attendee set avoids most allocations.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:06CSV import in C++
- 3:16CSV file problems
- 5:36Event backend overview
- 9:32CSV format assumptions
- 13:06Quotes and newlines challenges
- 16:32Reading CSV line by char
- 17:34Processing headers and fields
- 19:23Verifying data ordering
- 21:30Investigating unsorted data
- 24:00String conversions in C++
- 26:10Optimizing Q string allocations
- 29:01Q string conversion alternatives
- 31:48Q string decoder and UTF-8
- 33:17Final optimizations
- 35:07Order data and invoice issues
- 37:40Invoice management lessons
- 38:42Key takeaways on strings
- 40:10Large numeric IDs in CSV
- 41:55Future C++ and Qt improvements
- 43:25Q&A
related talks
42mFighting Against Chaotically Separated Values with Embulk
27mBuilding NumPy Arrays from CSV Files, Faster than Pandas
29mKEYNOTE: Big Opportunities in Small Data
33mGet your Data prod ready, Fast, with Ruby Polars!
24mHow Enables Real Time Data With Event Driven Architecture
01h 08mUsing and contributing to the data.table package for efficient big data analysis
19mStock data analysis with
21mActual trees, not b-trees: or how I found PostgreSQL through PostGIS