Leveraging the Apache Flight Python Client and InfluxDB
summary
Shows using the Apache Arrow Flight Python client and the InfluxDB v3 Python client (which wraps Flight) to run SQL or InfluxQL queries and return pyarrow or pandas tables, and cites a benchmark of 329 million rows/hour (≈4 million values/sec at 160k dimensionality). Describes stack: Arrow + Parquet + DataFusion, Grafana Flight SQL plugin, Mage/Quix pipelines with MQTT/HiveMQ, Hugging Face models and an Autoencoder anomaly detection demo that streams data into InfluxDB and successfully produces alerts and dashboards.
AI-written summary · about
chapters (21)
- 0:00intro
- 0:20preamble
- 0:35about anais
- 1:26agenda
- 2:04introduction to influxdb and time series data
- 6:17influxdb 3.0
- 6:47some of our customers
- 7:50ingest benchmark
- 8:35commitment to open data architecture with the fdap stack
- 9:06data storage
- 11:34advantages of columnar data storage (sidebar)
- 13:58leveraging the arrow flight client influxdb v3 python client library
- 15:23arrow flight sql client
- 16:35influxdb v3 python client library
- 17:36projects that leverage the arrow flight
- 17:51grafana flow
- 18:24try it yourself - grafana quick starts
- 19:33mage & influxdb - anomaly detection
- 20:07quix + influxdb
- 22:10solution architecture
- 23:20join the influxdb community
related talks
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
22mExploring the InfluxDB Ecosystem Integrations, Libraries, and Plugins
33mArrow in Flight - New Developments in Data Connectivity
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
30mOLAP on Your Cassandra Data with Arrow, Flight SQL, ADBC, and DuckDB
58mComposable Data Systems, the new frontier
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API