Complex Data Transformations Made Easy
summary
Claims a cloud‑scale compute layer (Xcalar) that replaces low‑level Spark code with an IDE using SQL + Python and visual modules, demonstrated by converting 25,000 lines of Java Spark into a dozen Xcalar modules. A production streaming platform processes 2+ TB of base data and multi‑GB daily Kafka updates, cutting a 4.5 hour batch to 15 minutes (>10x), reporting ~5x dev productivity, ~20x faster time‑to‑market and ~10x lower TCO; the data mart and log parser are built and run successfully.
AI-written summary · about
chapters (14)
- 0:00Intro
- 3:27Xcalar Application Platform ccelerate development & operationalization of cloud-scale data applications with unprecedented simplicity, speed and scale.
- 7:04Xcalar Application Platform - Problem Statement
- 13:34Transparent Modular Xcalar IDE 25,000 lines of Java code - 12 visual dataflows
- 15:44Xcalar Solution replaced Spark
- 18:21Streaming Applications Platform
- 20:36Use Xcalar Saas to analyze Xcalar logs Powered by myxcalar.cloud
- 21:23Xcalar Load Load log files as is, leverage power of SQL to parse, analyze and compute
- 22:18Parserless Parsing I used SQL and Scalar Function to parse row-by-row (all CPUs work in parallel)
- 25:24Data Application Development Paradigm I used Declarative + Imperative approach to build a Data Mart
- 27:10Data Application End-to-end
- 29:40Table Function Example get_increments() Table Function
- 31:53Operationalized Data Application
- 33:21Xcalar Programming Model - Review
related talks
52mSolving Analytical Problems using Apache Spark
26mRe-architecting the Data Platform with Spark on Kubernetes
01h 03mDiscovering the value of a cloud data platform
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
06mUnleashing intelligence and data analytics at scale
36mMoving Faster: Why Intent Media Chose Cascalog
38mEnabling Big Data and Machine Learning for the Masses...