Anonymize application logs in DevOps way
summary
Concrete claim: logs get anonymized without changing application code by routing Cloud Logging through Pub/Sub into a Dataflow (Apache Beam) ETL that batches entries and calls Cloud DLP for detection and obfuscation then writes back to storage. Implementation uses structured logs, Fluentd/Fluent Bit or agents for unstructured sources, and batching to save API cost and scale. Ownership shifts to DevOps, deployment spans regions, and the reference repo and blog show a working Google Cloud pattern.
AI-written summary · about
chapters (8)
- 0:00DevSecOps and logs anonymization
- 2:14Logs anonymization for compliance
- 6:14Compliance controls for logs
- 10:35Log anonymization challenges
- 14:56DLP services for anonymization
- 19:14Cloud-native log anonymization
- 23:12Google Cloud demo
- 28:32Resources & Q&A
related talks
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
21mAnomaly Detection for Platform Logs using Embedded Context
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
41mCentralized Logging Solution for Google Cloud Platform
31mThe Long and Winding Road to Application Logging [A]
36mGoogle Cloud Platform Solutions for DevOps Engineers
37mA Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day
33mCloud Native Logging 101 [B]