Obtain, Distribute and Use Temporary Credentials Automatically in Apache Spark and Apache Flink
summary
Apache Spark adds a Hadoop-bound delegation token framework that centralizes long‑lived credentials on the job manager, obtains short‑lived tokens, and automatically propagates and refreshes them to workers; Apache Flink implements a similar but protocol-agnostic implementation that serializes tokens and ships them to task managers. Both are production‑stable, used at thousand‑node scale, reduce KDC overload and credential exposure, and already have connectors for HDFS, S3, Hive and Kafka (and can carry TLS certs); implementations work and scale in real deployments.
AI-written summary · about
chapters (8)
- 0:00Introduction to scalable authentication
- 3:01Authentication basics and credentials
- 8:09Authentication in distributed systems
- 10:17Scaling authentication challenges
- 13:28Implementing scalable authentication plugins
- 17:03Apache Flink’s solution and flexibility
- 19:50Key takeaways and production insights
- 21:16Resources and documentation
related talks
30mFlink Security Enhancements
42mApache Flink deep-dive #bbuzz
37mScaling and Securing Spark on Kubernetes at Bloomberg
36mApache Flink in the Cloud-native Era
30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?
43mSpark Over a Mesos Secured Productive Environment
23mApache Spark meets FIPS standard
25mBridging the Gap Between Human and Workload Identity: SPIFFE Support in Keycloak