Make Big Geospatial Data Accessible with Arrow
summary
Firelytics is a historical database and Shiny dashboard that maps sub-daily wildfire progression for 22,000 fires and millions of polygons, used in NOAA and NASA–funded studies. The pipeline stores geopatial data as geoparquet and serves it as an Arrow dataset in the cloud, using sfarrow and geoarrow in R, partitioning by Fire_ID and prefiltering so the app only reads one polygon snapshot at a time. Benchmarks show read-in time under half of sf and the dashboard stays responsive.
AI-written summary · about
related talks
04mAccelerating geospatial computing using Apache Arrow
22mAccelerating Analytics with Apache Arrow
33mArrow in Flight - New Developments in Data Connectivity
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
20mCloud Native Patterns for Geospatial Analytics and AI
24mLeveraging the Apache Flight Python Client and InfluxDB
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
34mApache Arrow as a full stack data engineering solution