Writing a Distributed Ray Tracer with Apache Beam, Abridged

on YouTube ↗17m

" summary

Maps a Monte Carlo ray tracer onto the Apache Beam Go SDK, using Beam primitives: PCollections, DoFns, side inputs, combines and reshuffles to generate and aggregate per-pixel samples. Shows a working pipeline that runs on compatible runners like Apache Flink and Google Cloud Dataflow (demo laptop died but images were produced earlier). Identifies a scaling limit: naive sharding can produce ~3 TB of intermediate rays for a 4K image, so the practical workaround is precompute rays per pixel and keep recursion inside a single DoFn until splittable DoFns or cross-language transforms arrive.

AI-written summary · about

" chapters (4)

1. 0:00Introduction
2. 1:25Ray Tracer
3. 6:55Go SDK
4. 10:47Ray Tracing

" related talks

• 21mApache Beam (incubating)Eugene Kirpichov · Data By the Bay 2016
• 46mBuilding an Apache Beam streaming runtime in GCPSergei Sokolenko, Reuven Lax · Flink Forward Europe 2019
• 47mApache Beam - what do I gain?Łukasz Gajowy · GeeCON 2019
• 41mApache Beam: Unified Data ProcessingAlexey Diomin · Voxxed Days Minsk 2018
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 31mMaking Big Data Processing Portable. The Story of Apache Beam and gRPC - Ismaël MejíaIsmaël Mejía · KubeCon + CloudNativeCon Europe 2018
• 29mApache Beam Making Big Data Portable With gRPCIsmaël Mejía · gRPC Conf 2020
• 52mApache Beam, a unified model for Big Data processingJean-Baptiste Onofré, Ismaël Mejía · DevFest Nantes 2016