Dr. Elephant - Self-Serve Performance Tuning for Hadoop and Spark
summary
Dr. Elephant automatically profiles Hadoop and Spark jobs, ingesting logs and metrics and surfacing concrete diagnoses rather than vague best practices. The system runs rule-based heuristics as pluggable modules, points to specific misconfigurations (skewed tasks, wasteful memory settings, or redundant shuffles), and emits actionable tuning steps plus estimated cost or time impact. The web UI ties recommendations back to job attempts and cluster metrics so users can apply changes and observe whether throughput and resource efficiency actually improve.
AI-written summary · about
related talks
44mAutomatic parameter tuning for Spark applications
28mAI-driven query plan optimization in big data processing tools
26mFrom Spark to Elasticsearch and Back
52mDevil in the Haystack
19mBig Data
16mBeyond ETL: Revolutionizing Data Engineering Workflows
05mDBTB INT Chester Chen r
35mCreating Your Own LLM Tuning Platform with Open Source Technologies