How we opened up our private museum collection API
summary
ACMI opens a public collection API that serves a snapshot of its works metadata (about 42,000 works) by publishing JSON files in the same repo as the API so researchers can clone and run locally. The stack uses Flask + Elasticsearch in Alpine Docker images, deployed with Terraform on Azure and served by unicorn; nightly cron jobs pull filtered JSON from XOS (the museum OS that runs ~400 Raspberry Pis), push git changes and trigger CI. Builds take a few months to go production, requests run ~158 ms at 3 rps and ~500 ms at 18 rps, rights filtering removes many items and a full update of 45,000 records takes ~4 hours.
AI-written summary · about
chapters (50)
- 0:00Introduction
- 1:05Welcome
- 2:08Why did we build it
- 3:50Regular updates
- 4:02Accessibility
- 4:18Rich data links
- 4:33Experimentation
- 4:45Who is it for
- 4:58technologists
- 5:10internal use
- 5:47examples
- 6:14dependency map
- 7:09how we build it
- 8:33what we used
- 8:50was alpine linux
- 9:58nightly updates
- 11:20manual releases
- 11:39unicorn server
- 12:49speed test
- 12:59request speed
- 13:20time to build
- 13:42dependencies and security
- 13:58documentation
- 14:23prototypes
- 14:54Machine Dreaming
- 15:40Accessible accessibility labels
- 16:03Wikimedia fellowships
- 16:19Machine dreaming prototype
- 17:29Matrix poster
- 17:42Film stills
- 18:20The Training Set
- 18:45Challenges
- 19:47Expectations
- 20:42Automation
- 20:59Digitization
- 21:40Update time
- 22:00Get diffs
- 22:20Thank you
- 23:07Any questions
- 23:42Is there a skew
- 23:52Looking through the data
- 24:10New APIs
- 24:34Building relationships with users
- 24:47Uptime monitors
- 25:12Connecting with other organisations
- 25:35Data sovereignty issues
- 25:48Ask Seb
- 26:00Have you had issues with lawyers
- 26:19We havent yet touched wood
- 26:41Thank you Simon
related talks
24mThe Met: Art, APIs and Global Access
27mHow a major museum runs on Python
36mArkisto: an open-source, standards-based framework for digital preservation
45mManaging a search engine for over 600 million openly licensed media records
52mLibrary of the Future
28mOpenCollection - An open specification for describing API collections
14mAPI in a box
26mData with a mission: a COVID-19 API case study