![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
[[Socio-metrics]] Case: Open Office Project *For Bug-Tracking System [#u8522a57] **fetching script [#x3d81b16] usage: crawl_oof.sh <start> <end> #!/bin/bash START=$1 END=$2 mkdir -p data mkdir -p history WGET_CMD="wget \ --tries=5 \ --connect-timeout=5 \ --wait=10 \ --random-wait \ --no-verbose \ --no-check-certificate" \ for (( i=$START; i<=$END ; i++ )) do $WGET_CMD --output-document=data/bug_$i.html http://openoffice.org/bugzilla/show_bug.cgi? id=$i sleep 5 $WGET_CMD --output-document=history/history_$i.html http://openoffice.org/bugzilla/show_activity.cgi?id=$i sleep 5 done This script fetch both bug page itself and and its history page. ~ The execution is (if saving in crawl_oof.sh) ./crawl_oof.sh <starting bug_id> <ending bug_id> Both contents will be saved in ./data and ./history folder - note that if an error existed (most comes from proxy error at the server side) please execute ~ du -a | grep ^0 for probing the error files, that needs to be fetched manually again when it's available. *For the Mailing-list Archive [#aff8bdbc] The Open Office's mailing archive is only in html format.~ It needs an html-parsing approach to extract the structure ~ I recommend Jericho html parser over Java or nokogiri over Ruby. For this task I used Jericho Html Parser in java and extract the structure in as following example In dev archive 1997-1 2 2000-5 1 2000-10 29 left column is year-month and the total email in each year-month are in the right column My example code is here [] After we have all the lists, we use it as a fetching source file. ** fetching script [#a408a3ac] usage ./mailscript.sh <Source file's name> #!/bin/bash fname=$1 dir=${fname}_dir if [ ! -d $dir ]; then mkdir $dir fi exec<$fname url=http://openoffice.org/projects/www/lists/${fname}/archive while read line do str=`echo $line | cut -d" " -f1` tot=`echo $line | cut -d" " -f2` mkdir $dir/$str for ((j=0 ; j<$tot; j++)) do #echo $url/${str}/massage/${j} wget --output-document=$dir/$str/$j.html $url/${str}/message/${j} sleep 5 done done the input is only the source file name. ~ it will create a directory corresponded to the source file, and the read each line and looping fetch from each email id from year-month to year-month. it will create a directory corresponded to the source file, and the read each line ~ and looping fetch from each email id from year-month to year-month.
