PyData Global 2024
2024-12-03..2024-12-05
-- online --
[PyData Global 2020, PyData Global 2022, PyData Global 2023, PyData Global 2024]
| title | author | length |
|---|---|---|
| Scaling Outside the Warehouse Using DuckDB and Python | Adarsh Namala | 30m |
DuckDB integrates with Python to run in‑RAM OLAP SQL on local files, S3 and https, and scales via AWS Lambda or Coiled EC2 workers. The demo runs 30M rows… | ||
| NumHack 2024: Winners Announcements & Demo Showcase | C stosa | 42m |
Live announcements list concrete outcomes not promises: 250 registered 11 teams and 13 awards announced, with prototypes, dashboards and trained models judged… | ||
| Using AI to Spot Deforestation-related Cows on Satellite Images | Leonie Hodel | 30m |
Claims ability to detect and count cattle from 30 cm RGB satellite tiles across the Brazilian Amazon, using PyTorch to train a two-step neural network that… | ||
| Quantile Regression Workflows | Anton Antonov | 29m |
Quantile regression workflows use the Python package regression iser to specify pipelines that fit multiple quantiles, count points under each curve, and… | ||
| Understanding the end-to-end LLM training and inference pipeline | Mark Moyou, PhD | 30m |
Walkthrough of the pipeline names concrete choices and numbers: examines Llama 3.1 8B (128k vocab, 4,096 hidden size, 32 attention heads) and shows the memory… | ||
| It's in the Air Tonight. Sensor Data in RAG | Timothy J Spann | 01h 30m |
Builds a runnable Python pipeline that pulls sensor REST feeds and traffic and weather data from openAQ, enriches and vectorizes readings, and loads them into… | ||
| The Memory Efficient Indexing for Vector Databases | Akshay Ballal, Sonam Pankaj | 28m |
Claims a working vector streaming pipeline that parallelizes parsing, semantic chunking, embedding generation and upserts so embeddings never accumulate in… | ||
| Build Production Ready AI Agents with Burr | Elijah ben Izzy | 27m |
Burr is an open-source OS library that claims to make production AI agents reliable by treating apps as graphs of actions, conditions, and central state and by… | ||
| Panel: The Dashboard That Grew - A Scaling Saga | Duarte Carmo | 30m |
Claims a migration from Streamlit to Panel to handle scale and shows the concrete numbers: 100 to 2000+ users in less than 2 months. Describes running on… | ||
| Putting the data science back into LLM evaluation | Patrick Deziel | 29m |
Argues for reproducible, metric-driven LLM validation and demonstrates an open-source Django web app that compares models prompt-by-prompt, logs generation… | ||
| Making Gaussian Processes Useful | Bill Engels, Chris Fonnesbeck | 01h 30m |
Tutorial turns Gaussian processes into a usable tool: hierarchical binomial and count models on baseball batting and slugging data, implemented in PyMC, use… | ||
| Image Recognition for safety on the factory floor | Nicolò Giso | 24m |
ResNet50 backbone plus a custom U‑style classifier trains on a curated set from >30,000 images sampled every 10s and initially reaches 85% versus a 55%… | ||
| The Missing 78%: How Women in AI & Data Can Complete the Future of Innovation | Noor Aftab | 30m |
Claims specific numbers and case studies: 22% women in AI and the missing 78%; UN estimate of 131 years to close the gap; McKinsey and BCG figures like $12… | ||
| Trustworthy LLMs: Vibe checks are not all you need | Irina Vidal Migallón | 28m |
Claim: eyeballing LLM outputs is insufficient; set up an automated evaluation pipeline that measures task performance plus robustness, hallucinations, data… | ||
| Taking Data Science in industry from zero to production | Andrew Weeks | 28m |
Describes a workflow that moves projects from prototype to production by validating concrete assumptions rather than polishing features: use verifying income… | ||
| Solving Forecasting Problems in R and Python | John Mount | 29m |
Shows a concrete workflow: compare about seven R and Python packages, implement a hidden‑state decomposition that separates loyal vs transient customers, fit… | ||
| Bringing NVIDIA math libraries to Python scientific community | Sergey Maydanov | 27m |
nvmath-python packages NVIDIA math libraries into Python with JIT fusion so user-defined python device callbacks compile via Numba or Warp and run as single… | ||
| Climbing the causal ladder for fun and profit | Benjamin Vincent | 30m |
Builds a Bayesian structural causal model in PyMC that uses a censored regression to infer latent demand under stockouts, fits with the PyMC do operator, and… | ||
| From Inference to Features: Build a Core ML Platform from Scratch | Nathan Colbert | 01h 29m |
Hands-on build of five concrete components — feature store, model registry, orchestrator, inference engine and basic monitoring — using Test-Driven… | ||
| Dashboards to Aid British Government Decisions (using R) | Brookes, Horne | 28m |
Datacove builds a bespoke Shiny dashboard for DEFRA that delivers an alpha product (2023) and a beta dashboard (summer 2024) as part of a five‑year roadmap… | ||
| Dask ❤️ Xarray: Geoscience at Massive Scale | Hendrik Makait | 30m |
Demo loads a 250 terabyte National Water Model and subsets to 18 terabytes then runs an Xarray-on-Dask pipeline across a Coiled cluster of about 100 VMs… | ||
| Building Knowledge Graph-Based Agents with Structured Text Generation | Alonso Silva | 28m |
Builds a knowledge graph from plain text by forcing LLM outputs into a pydantic JSON schema and masking token probabilities with outlines then stores nodes and… | ||
| Evaluating RAGs: On the correctness and coherence of Open Source eval metrics | El Mawass, Neeman | 30m |
Evaluates open‑source RAG eval libraries (RAGAS TruLens and reuse) by running a 200‑question benchmark with 5 retrieved chunks per query using ChromaDB and… | ||
| LLMs in Regulated Industries: Challenges and Governance Solutions | Vyoma Gajjar | 24m |
Claims a typical deployment halves loan approval time (50% faster approvals), uses federated learning and synthetic data with differential privacy, and layers… | ||
| Automating SEA Retargeting for Smarter Audience Engagement and Higher Conversions | Shrikanth Singh | 28m |
An automated pipeline identifies high‑propensity site visitors via batch‑processed user journeys stored in AWS S3, engineered features (pre‑conversion events +… | ||
| Catching Bad Guys using open data and open models for graphs | Paco Nathan | 29m |
Claims that GraphRAG can be made accountable by unbundling graph construction and using open models and open data to produce explainable investigative graphs… | ||
| Python + BigQuery + DataFrames: Hands on with scalable "serverless" analysis | Hsia, Swena, Williams | 01h 29m |
Hands-on use of BigQuery DataFrames shows a pandas-like API that keeps data in the cloud (no install, couponed GCP project) and runs on a real dataset of about… | ||
| Time Series Analysis with StatsModels | Allen Downey | 01h 30m |
Shows how to build time‑series models from first principles with pandas then switch to StatsModels: compute a 12‑month moving average, extract a repeated… | ||
| Building an AI Travel Agent That Never Hallucinates | Bowne-Anderson, Nichol, Petraitytė | 01h 23m |
Hands-on tutorial shows a running AI travel agent that avoids hallucinations by separating execution from understanding: CALM and Rasa provide a flow engine… | ||
| The LEGO Approach to designing PyData Workflows | Sayantika Banik | 30m |
Claims an open‑source toolkit called DataJourney actually exists on GitHub, built over two years to stitch together interchangeable tools (ingest → monitor →… | ||
| Changing Data With Confidence using DuckDB | Hannes Mühleisen | 30m |
DuckDB provides ACID all-or-nothing semantics so updates either fully commit or automatically roll back on error or crash; a demo shows a failed insert undone… | ||
| Tips to Level-Up Your Shiny for Python Applications | Daniel Chen | 31m |
Built an adaptive filter component on Shiny for Python that prevents empty DataFrames by having filters update each other, using pandas Index.intersection… | ||
| New Features in Apache Spark 4.0 | Matthew Powers | 29m |
Lists concrete changes and examples: Python custom data sources let one load Hugging Face datasets with spark.read.format('huggingface').load; ANSI mode… | ||
| GPU development with Python 101 | Katrina Riehl, Jacob Tomlinson | 01h 29m |
Claim: Python can run real CUDA without C++; tools used are numba CUDA kernels and cuDF/pandas plugin on NVIDIA’s DLI/Jupyter environment. Concrete results: a… | ||
| PyTorch Workflow Mastery: A Guide to Track and Optimize Model Performance | Cainã Max Couto da Silva | 01h 31m |
Hands‑on pipeline that actually trains a ResNet18 on CIFAR‑10 (50,000 train images, batch 128) and logs metrics and artifacts to Weights & Biases, runs short… | ||
| PyScript - Writing a Python application in the browser | Chris Laffra | 01h 39m |
Demo shows a real to‑do web app built with PyScript and the LTK UI library running in the browser, importing Python packages like Folium and pandas in‑browser… | ||
| Discover the Julia Machine Learning Ecosystem | Daniel Molina | 28m |
Julia claims type stability, JIT compilation, and easy parallelism and shows concrete wins: a symbolic regression port to Julia produces clear runtime… | ||
| Bridging the Gap: Real-Time Predictive Analytics with Faustream | Joseph Oladokun | 18m |
Claims a working open‑source pipeline called Faustream that wires Python models into Kafka + Faust for low‑latency inference, not theory. Configuration uses a… | ||
| Navigating Cloud Expenses in Data & AI: Strategies for Scientists and Engineers | Saurabh Garg | 30m |
Shows how Metaflow and the Out‑of‑bounds managed platform attach costs to individual flows, report resource use per step, and let teams right‑size requests to… | ||
| Reproducible Python projects using Nix | Avik Basu | 29m |
Claims reproducible deterministic environments by using Nix to freeze OS and non‑Python dependencies and combine them with Python tooling. Demonstration builds… | ||
| Foundational Models for Time Series Forecasting: are we there yet? | Luca Baggi | 21m |
Transformers get repurposed for forecasting by quantize to discrete tokens (Chronos uses GPT‑2/T5 with mean scaling and binning) or by patching continuous… | ||
| Python Apps in the Browser made simple by PyCafe | Maarten Breddels | 28m |
Claims a single-click zero-cost deploy for live Python apps and demonstrates it by running Streamlit Dash Panel Solara and (soon) Gradio entirely in the… | ||
| A Deep Dive into Python-Powered Precision and Scalability | Art Anderson | 28m |
Builds a production site search that combines Gemini embeddings for 768‑dim vectors with an inverted index stored in Aerospike, using Scrapy, LlamaIndex… | ||
| Let our optima combine! | Eyal Gruss | 29m |
Shows a concrete workflow: model a 3×3×3 Hebrew perfect pangram (27 cells) with OR-Tools CP-SAT, channel integer cells to 27×22 Booleans via order encoding… | ||
| Generative AI + Python: Unlocking Efficiency, Personalization, and Insight | Tony Ojeda | 36m |
Three Streamlit apps demonstrate concrete results: a data‑analysis assistant that writes pandas and Plotly code, trains logistic/regression random forest and… | ||
| Preparing Data for LLM Applications Using Data Prep Kit | Maniyam, Nielsen | 01h 17m |
Open-source Data Prep Kit shows a concrete pipeline: PDF→chunks→dedupe→embeddings using Python tools (Colab demo, DocLing PDF extractor, sentence-transformer… | ||
| Automatic differentiation, a tale of two languages | Guillaume Dalle | 29m |
Compares Python frameworks (TensorFlow Jax PyTorch) with a dozen or so AD backends in Julia and claims choice order flips: Python picks framework first, Julia… | ||
| Open Source Component-Based Modeling with ModelingToolkit | Chris Rackauckas | 27m |
ModelingToolkit composes equation-based components, performs symbolic structural simplification and generates Julia code that plugs into fast… | ||
| The Data That Shapes Foundational LLMs | Marysia Winkels | 28m |
Explains concrete pipelines: HTML parsing, deduplication and ngram coverage filtering, and modelbased filters such as FastText to surface higherquality web… | ||
| Rapid deduplication and fuzzy matching of large datasets using Splink | Robin Linacre | 27m |
Splink is a free Python library that actually produces cluster IDs by combining unsupervised statistical scoring with fuzzy matching and dataset term… | ||
| Realtime Time Series Anomaly Detection in Production | Shreya Khurana | 30m |
Realtime pipeline that forecasts with NeuralProphet and scores residuals rather than raw values, using minute-level data with rolling forecasts and occasional… | ||
| Statically-Compiled Julia for Library Development | Jeff Bezanson | 31m |
Demonstration shows Julia producing small statically compiled C‑ABI libraries using a new trim option and a contrib compile script that finishes in about 30… | ||
| Retooling for a Smaller Data Era | Wes McKinney | 30m |
Argues that real progress is concrete: adoption of DuckDB and DataFusion as embeddable engines, a common memory format in Apache Arrow, and multi-backend APIs… | ||
| Valuable LLM lessons learnt on Kaggle's ARC AGI Challenge | Ian Ozsvald | 29m |
Six months of work produces an automatic program generator that runs Llama3 locally via llama.cpp on an RTX3090, generates Python/NumPy solutions, executes… | ||
| Graph RAG: Bringing together graph and vector search to empower retrieval | Prashanth Rao | 30m |
Claims that combining graph and vector retrieval yields more factual RAG outcomes and shows this on an Interstellar knowledge base using a hybrid pipeline… | ||
| Moving from Offline to Online Machine Learning with River | Tun Shwe | 30m |
River claims to make online ML practical in Python by learning one observation at a time and reusing scikit‑learn–style APIs for classification, regression… | ||
| Do Python and Data Science Matter in Our AI Future? | Peter Wang | 01h 03m |
Argument: Python remains central because concrete infrastructure and community practices shape AI not slogans. Evidence: an open data science stack (e.g. numpy… | ||
| Empowering New Contributors: The Evolving Role of the R Development Guide | Saranjeet Kaur Bhogal | 20m |
The R Dev Guide claims to make R contribution accessible by giving a single, rebuilt pathway: a site drafted 2021 expanded during Google Season of Docs 2022… | ||
| Empowering PySpark with Lance Format for Multi-Modal AI Data Pipelines | Lu Qiu, Allison Wang | 27m |
Claim: PySpark writes directly to Lance format using the Python Data Source API and Arrow so transformed data lands as Lance fragments without intermediate… | ||
| Fast, intuitive feature selection via regression on Shapley values | Koseoglu, Kraev | 27m |
shap-select claims feature selection needs one model fit and nothing more and implements a linear or logistic regression of the target on Shapley values… | ||
| Going Plaid: Striving for Speed of Light in CI pipelines | Michael Sarahan | 30m |
Adds telemetry to GitHub Actions CI, exporting GitHub run metadata as traces to Tempo + Grafana, using OpenTelemetry SDKs to parse job and step timings into… | ||
| Build Your Own Transformer | Borar, Liu, Shrivastava | 01h 29m |
90 minute hands-on workshop that implements self-attention from scratch in a shared Colab notebook using only NumPy. It walks a 3‑token example ("you are… | ||
| Harnessing Machine Learning to Improve Agricultural Resilience in Africa | Kristal Joi Wise | 28m |
Names specific models and platforms: TensorFlow and Google Earth Engine used with IoT ear tags, drones and phone apps; gives numbers: 36% livestock increase… | ||
| ColPali’s Vision-Powered RAG for Enterprise Documents | Zain Hasan | 36m |
Claims that ColPali embeds entire pages into multi-vector representations using VLMs and then retrieves by maximum similarity token-patch scoring… | ||
| Using and contributing to the data.table package for efficient big data analysis | Toby Dylan Hocking | 01h 08m |
An R package with C core claims dramatic practical speedups: fread is fastest on large CSVs, summarization benchmarks show up to 100× faster than base R and… | ||
| Microcontrollers + Machine Learning with MicroPython in 1-2-3 | Jon Nordby | 29m |
MicroPython runs on tiny hardware (less than 1 MB RAM) and on an ESP32S3 ($2.5 chip, ~$6 board) that boots a Python prompt in ~30 seconds; examples show a… | ||
| The Hidden Costs of Data Quality - Tackling Common Data Challenges in ML | Kalyan Prasad | 31m |
Claims a concrete ROI from fixing data: 20% missing emails found and filled, customer statuses reconciled with the CRM, and standardized formats. Uses open… | ||
| PYDATA Bloom Framework: An Approach to Data Science in University Education | Hansila Sudasinghe | 29m |
A concrete four‑year Python curriculum claims to close a skills gap by aligning Bloom’s Taxonomy with the KDD pipeline and year‑by‑year learning outcomes… | ||
| The art of wrangling your GPU Python environments | Jacob Tomlinson, Melody Wang | 27m |
Benchmarks claim pandas work that takes ~59 seconds drops to 742 milliseconds on the GPU, and the talk shows a concrete utility called RAPIDS Doctor that runs… | ||
| Boosting AI Reliability: Uncertainty Quantification with MAPIE | Cordier, Jawad, Laurent | 01h 24m |
An open‑source Python tool MAPIE scikit‑learn‑contrib wraps estimators to output calibrated prediction intervals and sets with statistical guarantees (example… | ||
| 🧠🧹 Causality - Mental Hygiene for Data Science | Eyal Kazin | 29m |
Claims that assessing whether causality applies yields value even if full causal inference is infeasible, and that using concrete tools changes outcomes: the… | ||
| Improve LLMs Alignment with Complete and Robust Preference Data | Son The Nguyen | 32m |
Claims a concrete pipeline called KURA that transforms pairwise scores with a BTL logit link, imputes missing comparisons via matrix completion (LRPR), and… | ||
| Build simple & scalable data pipelines with Polars & DeltaLake | Liam Brannigan | 29m |
Claims that Python + delta-rs (Delta Lake Rust) plus Polars projection/predicate pushdown let pipelines handle evolving messy data with real transactions. Demo… | ||
| An Evaluation of Open-Source OCR Models for Japanese Medical Documents | Bing Wang | 37m |
Open-source OCR models are evaluated head-to-head on Japanese medical PDFs using accuracy, memory use, and speed; Tesseract PaddleOCR and EasyOCR are tested on… | ||
| Hands-on Multimodal AI Development with Pixeltable | Brunelle, Kornacker | 25m |
Pixeltable claims a declarative, incremental data layer for multimodal ML that version controls tables, materializes video into frames, and runs model… | ||
| Foundational Time Series Models in Practice: The Future of Forecasting, or Just Hype? | Ahad Shoaib | 31m |
Clear claims and concrete results: patch TST uses nonoverlapping patches to get long lookbacks and reports state‑of‑the‑art leaderboard wins versus vanilla… | ||
| Measuring the User Experience and the Impact of Effort on Business Outcomes | Maggie Wolff | 29m |
Proposes an aggregate User Effort Index that weights measurable friction (errors, repeated clicks, long time on task) and uses correlations and predictive… | ||
| Holistic Evaluation of Large Language Models: From References to Human Judgment | Jhaveri, Joshi | 32m |
Evaluation compares reference-based vs reference-less methods with concrete outcomes: BLEU/ROUGE show low Spearman/Kendall correlations (~0.1–0.2) to human… | ||
| Practical Techniques for Polished Visuals with Plotnine | Nicola Rennie | 34m |
Builds an area chart of coal production since 1900 from the Carbon Majors dataset, computes the first year >100M tons and places datadriven annotations and… | ||
| Fairness Tales: How To Measure And Mitigate Unfair Bias in Machine Learning Models | John Sandall | 01h 30m |
Workshop generates a synthetic 400 CV dataset (OpenAI) and a biased recruiter simulator that produces a 41% callback baseline and a male callback rate ~70% vs… | ||
| Understanding Polars data types | Rodrigo Girão Serrão | 31m |
Polars claims 18 data types and demonstrates concrete tradeoffs: bit-packed booleans save memory, a validity mask makes is_null free at compute cost of a bit… | ||
| 3D geospatial data visualization using Python and Cesiumjs | Nompumelelo Mtsweni | 01h 30m |
Processes London TfL GeoJSON with Python + geopandas to produce colored underground-line polygons, then renders 3D buildings and lines on a Cesium globe using… | ||
| Cost-effective data annotation with Bayesian experimental design | Nguyen Hong Quan | 31m |
Claims that Bayesian experimental design picks labels far more efficiently than random sampling, shows a concrete 100-element array example where the optimal… | ||
| Let's get you started with asynchronous programming | Ryan Varley | 30m |
Claims that async is practical not mystical and shows working code using Python 3.12 (back to 3.7+), aiohttp and the standard asyncio.gather pattern with… | ||
| Python is a Joke! | Evan Wimpey | 28m |
Comedian uses concrete data jokes and code punchlines: claims a ludicrous 90% accurate stock model that cheats with a tomorrow feature, shows a toy-ML workflow… | ||
| Training Language Models to Identify Urgent Messages in Real-Time | Zhen (Tony) Zhao | 30m |
Fine-tunes Gemma-2 2B instruction model on a synthetic dataset of 13,000 synthetic messages (12k train, 1k validation) to classify urgent vs non‑urgent… | ||
| Effective GenAI Evaluations: Mitigate Hallucinations and Ship Fast | Atin Sanyal | 29m |
Claims concrete: chain pole uses chain-of-thought plus n-way binary polling and beats prior LLM-judge methods on open and closed domains; Luna is a small… | ||
| akimbo: vectorized processing of nested/ragged dataframe columns | Martin Durant | 30m |
Library akimbo brings a numpy-like API to Arrow-backed dataframe columns, letting awkward-style nested/ragged arrays be processed without Python iteration… | ||
| Embrace the Unix Command Line and Supercharge Your PyData Workflow | Jeroen Janssens | 54m |
Argues that the Unix command line measurably speeds up everyday PyData work, showing exact incantations: bat for colored file views, fx for exploring nested… | ||
| Building LLM Voice Bots with Open Source Tools | Sara Zanzottera | 32m |
Explains exactly how modern voice bots are built: a voice activity detector spots speech, Whisper (or alternatives) does ASR, a text LM handles intent and… | ||
| Off-the-shelf HuggingFace models for audio deepfake detection | Adriana Stan | 31m |
Uses wav2vec 2.0 and WavLM embeddings from HuggingFace, averages frame embeddings into a single vector, and trains a simple logistic regression classifier on… | ||
| Enabling Multi-Language Programming in Data Engineering Workflows | Daphné Grasselly | 30m |
Python code reads clinical-trial metadata and auto-generate Snakemake workflows that orchestrate SDTM→ADaM→TFL runs, launching run SAS and R inside containers… | ||
| Streamlining AI development and Deployment with KitOps | Shivay Lamba | 30m |
Claims that 80% of ML models never reach production and shows a concrete path to fix that using ModelKits and KitOps, with live CLI demos unpacking a YOLOv10… | ||
| Deep Learning in Energy Management: Non-Intrusive Load Monitoring for IoT Devices | Francesco Conti | 30m |
Claims that deep models can do NILM on low‑volume irregular IoT data using the Chain2 protocol and that training on properly simulated Chain2 samples improves… | ||
| sktime: time series anomaly detection, changepoint detection, segmentation | Kiraly, Risi, Tveten | 01h 29m |
skchange extends sktime to detect point changes, changepoints and anomalous segments using concrete components: moving window, PELT, L2 and Gaussian cost… | ||
| Understanding API Dispatching in NetworkX | Aditi Juneja | 29m |
Claim: NetworkX redirects plain-Python calls to faster implementations in separate packages by using Python entry_point registration and a backend kwarg that… | ||
| Mastering Large NDArray Handling with Blosc2 and Caterva2 | Francesc Alted | 01h 27m |
Blosc2 is a compression format and library for large NDArrays that shows concrete results: a 10,000×10,000 array (100 million elements) compresses to a few… | ||
| nothing matches | ||