Data Pre-processing in MindSpore
summary
MindData presents a concrete system: an asynchronous data graph that pipelines preprocessing, uses parallel IO (master/worker reads six files in parallel), and feeds a device queue to GPUs and Ascend chips so training does not stall. It enforces an API → IR → runtime separation, serializes IR as JSON, and handles shared C++/Python objects and ownership. The implementation runs in MindSpore, yields measurable throughput improvements from pipelining and parallel I/O, and supports host vs Android runtime differences.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:12Deep Learning
- 3:48Training
- 8:04Accuracy
- 11:16API Complexity
- 13:16MindSpore
- 14:57Data Graph vs Training Graph
- 20:34Device Queue
- 22:15Python Example
- 25:25Pipeline
- 28:43APIIRRuntime
related talks
08mMeet MindSpore, the new open source AI framework!
24mMindSpore's Foundation Model Platform Promotes the Ecological Development of Foundat...
28mAutomatic Operator Generation for Open AI frameworks: MindSpore/AKG
01h 26mData Driven Action: a primer on Data Science
26mOne Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
11mData Science and Smart Systems: Creating the Digital Brain
52mMuvr
37mBig Data Analytics, Machine Learning, Real Time Processing