Event-driven task automation of Java and Spark jobs with Mesos
summary
Company builds Terra Scout, a custom Mesos framework that watches landing zones, uploads new files to HDFS, and spawns Java or Spark jobs via generated launch commands (including spark submit). Patrol/handler pattern accepts events, waits for stable inputs, and enqueues tasks in a first‑come first‑serve scheduler that groups offers per slave. Features: JVM and Spark task abstractions, per‑task UUIDs, night‑only scheduling, reconciliation across masters, REST control, metrics to InfluxDB; system replaces manual runs and proves fault tolerant in production.
AI-written summary · about
chapters (5)
- 0:00Start
- 1:59Teralytics Business Model
- 3:09Why Task Automation?
- 4:12What should Task Automation do?
- 13:35Mesos Task abstractions
related talks
59mBuilding Distributed Systems with Apache Mesos
58mDistributed Scheduling with Apache Mesos in the Cloud
42mStreaming Data Pipelines on Apache Mesos: Lessons Learned
19mKeynote Panel Discussion: SMACK in the Enterprise
38mMesomatic: The Cluster is a Library
36mIn-memory data pipeline and warehouse at scale
48mFireworks - lighting up the sky with millions of Sparks
18mFostering Chaos Engineering Culture