Dask-SQL: Empowering Pythonistas for Scalable End-to-End Data Engineering
summary
Claims that Python users can run SQL over enterprise datalakes without JVM tooling and demonstrates it with three concrete demos: dask-sql executes SQL planning via Apache Calcite, registers and queries a Dask DataFrame, creates a table purely with SQL (not Python), and reads Hive metastore tables and runs on Dask clusters. Each demo compiles to Dask dataframe operations, computes into pandas for plotting, and installs via conda so the system actually runs end to end.
AI-written summary · about
chapters (5)
related talks
29mBuilding Large Scale ETL Pipelines with Dask
01h 02mData lakes beyond clusters
27mBecause SQL Is Everywhere
35mData Processing Pipelines for Small Big Data
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
36mData pipelines engineering made simple with Scala
27mUsing embedded SQL engines for plotting massive datasets on a laptop