~/talk

Multi-GPU Accelerated Processing of Time-Series Data of Huge Academic

on YouTube 42m
Talkat LISA19 · on Monday, 2019-10-28· at 14:00· Salon F· [data-engineering]

speaker

name:Ruo Ando
links:none on file

resources

video:https://youtu.be/Qw3ccR0r4Ik
slides:none on file

summary

Operational deployment ties an Elastic Stack + Splunk pipeline to a DGX-1 multi‑GPU box and targets PaloAlto‑7080 session logs; processing a 729 GB test dump initially takes ~910–1600 minutes but is reduced overall about 10×. Key techniques: multi‑process invocation of Elasticsearch scroll API with 128 chunks, CUDA Thrust bitwise transform for direction discrimination replacing slow CPU bitmask loops, and pairwise MapReduce-style reduction on GPUs for histograms, shrinking histogram time from ~476 minutes to ~12 minutes and discrimination from ~704 to ~174.

AI-written summary · about

chapters (8)

  1. 0:00Intro
  2. 1:31Overview: Pipeline of Elastic stack, Multi GPU and Splunk Backgrounds and bottlenecks Strategy of parallelism for unpredictable Design philosophy Floorplan and configuration (Elastic stack and Splunk)
  3. 8:20STRATEGY of parallelism Dynamic allocation for unpredictables
  4. 12:55FLOORPLAN and cluster CONFIGURATION Elastic stack and Splunk
  5. 17:13Huge pagination without scoring Parallel scroll API invocation
  6. 20:52Ingress/egress discrimination Bulk masking by Thrust transform()
  7. 25:58Histogramming in milliseconds Map-Reduce and Tiling
  8. 35:18Conclusion and impressions

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.