Turn your logs into actionable data at any scale with AWS
summary
A working open source log pipeline is shown: Kinesis Agent for Windows (aka kinesis tap) installs via PowerShell or MSI, tails Windows event logs and flat files, applies regex extraction and decoration into JSON, and ships to Kinesis Data Streams or Data Firehose into S3. AWS Glue auto-discovers schema, Athena runs SQL queries, and Lambda (PowerShell) turns events into metrics in seconds. Bookmarking, multiline handling, and fan‑out to multiple sinks are implemented and the live demo successfully streams and queries real data. near real-time
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:55Kinesis agent for Windows importance
- 3:17Kinesis agent for Windows
- 6:47Live demo setup
- 9:34Streaming logs to Firehose
- 13:25Installing agent
- 15:13Viewing streaming data
- 17:42Querying logs with Glue & Athena
- 22:16PowerShell script block logging
- 23:20Advanced agent options
- 29:17Sync destinations & transforms
- 33:49Filtering & fan-out
- 35:16Lambda real-time processing
- 38:03PowerShell log processing demo
- 40:54Agent features & future
- 42:16Q&A
related talks
32mBuilding and scaling a log analytics platform - a serverless approach
01h 02mCentralised Logging without the blood, sweat and tears
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
38mBuilding Data Pipelines on AWS
46mBuilding a scalable logging platform
42mTransform your data approach: Develop a modern data strategy
32mReal-Time Data Streaming with Azure Stream Analytics
46mWorkshop: Log Processing with Apache NiFi and Apache Kafka