Fighting Against Chaotically Separated Values with Embulk
summary
Claims that Embulk auto-detects chaotic CSV dialects, runs as an embeddable Java library and a command-line tool, and supports 100+ community plugins for inputs like Salesforce, Mixpanel, S3 and outputs like PostgreSQL and BigQuery. Engine does parallel and distributed execution, offers resume-from-failure and incremental loading, and targets large batches (example: 10 GB × 720 files). Demo shows configs and filters working end-to-end and successful loads into production stores.
AI-written summary · about
chapters (13)
- 0:00Embulk introduction
- 2:10Embulk plugin architecture
- 9:48Embulk data loading purpose
- 11:35CSV to PostgreSQL demo
- 15:53PostgreSQL to MySQL demo
- 17:41Embulk architecture engine
- 23:30Failed load recovery
- 26:32Configuration versioning
- 27:37JSON column support
- 28:22Error handling future
- 31:21Dig Dag automation
- 33:54Dig Dag parameters demo
- 35:55Q&A
related talks
42mDeciphering the way data is tested
18mScalable, Code-Free ETL
29mSwiss Army Django: Small Footprint ETL
46mSparking Success: Unveiling the Journey of Apache Spark Application Development
19mSwiss Army Django: Small Footprint ETL
45mWorkshop 3: Data Engineering - A Roadmap... by Ramon Perez [Part 3]
36mData pipelines engineering made simple with Scala
29mHigh Performance Data Loss