~/talk

Dirt Simple Datamining

on YouTube 31m
Talkat Ruby|Web Conference 2010 · 2010-09-09..09-10· [data-visualization, web-platform]

speaker

name:Matthew Thorley
links:none on file

resources

video:https://youtu.be/6b4O5iuFuNA
slides:none on file

summary

Logs into Bebo by POSTing to sign_in JSP, handles the extra hidden parameter (signInBack), and uses mechanize to persist cookies and follow forms; discovers AJAX steps with Tamper Data and bypasses JS by posting the right fields. Crawler extracts member IDs, downloads profile and friends pages, and parses them with a mix of nokogiri and regex parsing, writing pages to disk for debugging. Code lives on GitHub; throttling, proxies, robots.txt and legal limits determine how far the crawler runs.

AI-written summary · about

chapters (14)

  1. 0:00Intro
  2. 4:10Simple Example
  3. 6:37Source
  4. 8:13Websites
  5. 9:16Tamper
  6. 10:52Profile Page
  7. 12:03Get Member ID
  8. 13:31Class Overview
  9. 15:35Profile Pages
  10. 18:38Parser
  11. 20:11Profile
  12. 21:34Parse
  13. 22:15Code
  14. 23:50Caveats

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.