Distributed file-systems made easy with Python's fsspec
summary
fsspec provides a single Python API for local and cloud stores, showing concrete code: pandas reads a CSV/Parquet via fsspec with S3/GCS backends, parquet engines fetch the footer and pull only needed row groups, and caching modes include simple file cache, block/bytes cache with tradeoffs. A 120-line toy filesystem is implemented to show extensibility. lakeFS-backed fsspec uses temporary branches so transactions appear atomic (semi-atomic without lakeFS), demo commits and merges work as described.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:28Why distributed storage matters
- 1:31Cloud storage API challenges
- 2:35fsspec unified interface
- 4:03fsspec benefits and features
- 5:06S3 vs fsspec code
- 6:31Partial file reading
- 6:53Pandas integration
- 7:59Local development
- 8:41Parquet access
- 9:24Caching with fsspec
- 11:32Custom file systems
- 12:35Toy file system demo
- 15:58LakeFS introduction
- 16:40What LakeFS does and how it works
- 17:44LakeFS versioning
- 18:26LakeFS + fsspec
- 19:50fsspec transactions
- 21:30Live LakeFS demo
- 27:23Summary
- 28:46Q&A
- 29:28Authentication in fsspec
- 30:34Azure Blob Storage
- 31:16Custom providers & SFTP
- 32:44MinIO with fsspec
- 34:13S3 performance
- 35:59Caching & file changes
- 36:43LakeFS vs DVC
related talks
08mAsynchronous fsspec file operations
38mScale out from the very beginning
34mLearning to Write Filesystems in Userspace with FUSE and Python
30mBlazing fast I/O of data in the cloud with Daft Dataframes
30mgitfs - building a filesystem in Python
33mThe Evolution of Python Packaging
23mDaft - The Distributed Python Dataframe for Complex Data
01h 01mParallel Python Analyzing Large Data Sets