From telescope to data centre: adventures in astronomical data pipelines
summary
Pipeline moves nightly AAT data from Siding Spring to Sydney and Canberra using a Minifi agent into an Apache NiFi flow that extracts FITS headers into MongoDB, stores files to on‑premise storage and enforces access via CAS; legacy Fortran 95 2dFDR reduction runs in containers orchestrated with Docker and Celery, processing ~120 GB in ~45 minutes on a dual‑CPU node. Notes scale: AAT ~15 TB total, AAT ~15 GB/night, SKA ~600 PB/year. Apache NiFi
AI-written summary · about
chapters (15)
- 0:00Intro
- 2:21Who am I and what is Data Central?
- 4:30Background and motivation
- 11:04Explosions, collisions and mergers
- 14:28Astronomical data formats
- 18:06Data acquisition
- 20:59Problem
- 22:36Case Study: Anglo Australian Telescope
- 25:41Our solution
- 27:19Technological solution
- 30:30MongoDB
- 32:22Central Authentication Service
- 33:11Web technologies
- 33:44Data Reduction
- 39:03Future challenges
related talks
11mProcessing TBs of Astronomy Data at Scale with the AC3 Horizon Research Project
47mBuilding Modern Data Streaming Apps with Open Source
46mApache NiFi 101: Introduction and Best Practices
30mThe Fabulous World of Data – ETL Powered by Apache NiFi and Scrapy
59mPolyglot Pipelines with Apache Nifi
39mAnalysing Public Kafka Data from NASA Satellites
06mLightning Talk: SKAO: Next-Generation Radio Astronomy-Driven Big Data
44mApache NIFi Deep Dive 300