Frictionless Data Pipelines
summary
Claims a working integration: Frictionless Data Python libraries get wired into oceanography ingest pipelines to clean and standardise real messy submissions, with a UI for data managers that produces pipeline-spec YAML and a datapackage.json package. Uses Data Package Pipelines for declarative steps, custom oceanographic processors, and an outputs-as-code workflow that reduces manual fixes and processing time. GoodTables validation and a statistics calculator are being added; adoption in research remains limited so far.
AI-written summary · about
chapters (11)
- 0:00Frictionless Data project intro
- 1:04Reproducibility crisis in research
- 5:19Open Knowledge Foundation mission
- 8:31What is frictionless data
- 12:07Frictionless Data Python libraries
- 16:04Data Package Pipelines explained
- 17:32B Co demo oceanographers case
- 21:06Streamlining B Co data workflows
- 22:55Pipeline tool UI demo
- 27:10Hands-on frictionless data tools
- 28:57Q&A
related talks
45mOpen Source for Open Science with Frictionless Data
36mPlumbing in Python: Pipelines for Data Science Applications
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
22mData pipelines != workflows: orchestrating data with Dagster
19mDemystifying Modern Data Pipeline Architecture
34mpydiverse pipedag: A library for data pipeline orchestration
23mWhat would the django of data pipelines look like? with Lisa Dusseault
49mData pipelines: building an efficient instrument to make custom workflow