How to NOT build a data integration
summary
Integrations with GitHub, PagerDuty and Jira use event webhooks, GraphQL/REST and Forge but real choices mattered: GitHub worked best via webhooks + Lambda → S3 → Snowflake + dbt because GraphQL hit a global rate limit and events include full payloads; PagerDuty forced OAuth and lacked page timestamps so use acknowledged events as a proxy; Jira’s custom fields vary per account so story points are unusable and Forge required hourly polling. Practical rules: store raw events, queue and batch writes, avoid customer tokens, and version your schema and Terraform pipelines.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:01Data integration challenges
- 5:32GitHub integration lessons
- 11:56PagerDuty integration challenges
- 16:05Jira integration pitfalls
- 20:25Key takeaways
- 21:06Lessons learned
- 23:37Improving integration practices
- 24:59Q&A
related talks
24mAvoid Common Pitfalls in API Integration: Expert Insights
42mHow Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations
39mCrafting modern data platforms: reveal your data value
52mScaling Data Teams: Three Pitfalls and How To Avoid Them
50mWorkflows of Highly Functional App & Data Engineering Teams
29mGrowing insights through hybrid data architectures
10mBuilding a Data Platform at Assembly Payments
27mBuilding modern applications with data on Heroku, Salesforce, and