Keita Broadwater Interview
summary
Describes building a pipeline that glues sensors, messaging systems and Spark to get usable inputs and repeatedly hits roadblocks around cleaning gigabyte-scale, diverse datasets with important corner cases. Reports a concrete migration: most code and workflows currently in Python and PySpark while ongoing work rewrites core pieces into Scala to gain functional-style, Spark-efficient code, and highlights the need to formalize onboarding procedures for new clients and integration tests.
AI-written summary · about
