Applying Tekton to ingest data in RAG model
summary
Tekton YAML pipeline pulls two sources (public docs + internal Git repo), cleans out JS/CSS, chunks text into ~500-token pieces with overlap, then runs a training step that attaches embeddings to a base language model (local training ~1 hour), and saves vectors into a Postgres/pgvector store for live consumption. Pipeline triggers via pipeline-run objects on repo changes (currently full reindex on any change), is parameterized for reuse across different RAG models, and hides training from the chat frontend. Tekton pipeline, 500-token chunks, full retrain on change
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:02Speaker intro
- 2:26Support for new tech
- 3:32RAG model and Tekton
- 4:57Tekton pipeline structure
- 7:50Training and embeddings
- 8:51Storing embeddings
- 9:41Auto pipeline trigger
- 11:07Generic RAG use
- 11:31Questions and answers
related talks
01h 32mBuild a production-ready RAG agent
14mRobust R Deployments: Building a Pipeline from RStudio to Production
35mUsing Sockeye Neural Machine Translation in a Streaming Pipeline
41mServerless LLM with AWS Lambda and LangChain: A Revolution in Application Development
35mSimplification of Data Pipelines by Containerization
32mEvents in Action With Spinnaker, Tekton and Keptn
02h 35mBringing GenAI to the Modern Enterprise. A production use-case. In Serverless Java!
44mAutomatically scale a serverless app with Amazon Textract & MongoDB