Using Open Data to Predict Market Movements
summary
Claims a measurable correlation between media/article volume and vendor shifts in Gartner Magic Quadrants, demonstrated with NetApp and Hitachi Data Systems time-series. Pipeline ingests the Common Crawl S3 archive (billions of pages, monthly crawls since 2008), uses CDX index + commoncrawl-document downloader and Python to fetch pages, cleans HTML with DKPro-c4, runs Spark/Scala on EMR/Zeppelin for n‑gram and frequency analysis, and outputs d3.js visualizations; the automated workflow scales to hundreds of terabytes and produced actionable signals that tracked MQ movements.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:41Outline
- 1:46Background overview
- 5:10Data Exploration & Findings
- 6:59Data Analysis Process
- 7:28Objectives
- 9:44Data Extraction : Data Sources Identified
- 10:30Data Extraction : Downloading the index data
- 12:50Common Crawl Index Server
- 14:36Data Pre-Processing : HTML Removal
- 15:13Data Pre-Processing : Before HTML Removal
- 15:48Data Processing & Analysis
- 17:46Data visualization
- 19:45Important Links
- 19:59The Team
related talks
14mHarnessing Common Crawl for AI and ML Applications
01h 35mDeep Dive into the Big Data Landscape
31mFinding Needles in a Growing Haystack
19mBig Data
30mHitachi applying AI in Industry 4.0 digital transformation projects
49mA Real-World Example of Using Network Behaviour to Detect Cyber-Compromise
01h 26mData Driven Action: a primer on Data Science
24mUncovering Community and Project Insights Through Data Driven Methods