Extracting HTML data from an OSS project の履歴(No.1)

• 履歴一覧
• 差分 を表示
• 現在との差分 を表示
• ソース を表示
• Extracting HTML data from an OSS project へ行く。
  • 1 (2011-10-05 (水) 06:05:32)

Socio-metrics

Case: Open Office Project

For Bug-Tracking System†

fetching script†

usage: crawl_oof.sh <start> <end>

#!/bin/bash

START=$1
END=$2

mkdir -p data
mkdir -p history

WGET_CMD="wget \
 --tries=5 \
 --connect-timeout=5 \
 --wait=10 \
 --random-wait \
 --no-verbose \
 --no-check-certificate" \
 
for (( i=$START; i<=$END ; i++ ))
do
	$WGET_CMD --output-document=data/bug_$i.html http://openoffice.org/bugzilla/show_bug.cgi? id=$i
	sleep 5
	$WGET_CMD --output-document=history/history_$i.html http://openoffice.org/bugzilla/show_activity.cgi?id=$i
	sleep 5
done

This script fetch both bug page itself and and its history page.
The execution is (if saving in crawl_oof.sh) ./crawl_oof.sh <starting bug_id> <ending bug_id>

Both contents will be saved in ./data and ./history folder

• note that if an error existed (most comes from proxy error at the server side) please execute
  

  du -a | grep ^0

  for probing the error files, that needs to be fetched manually again when it's available.

For the Mailing-list Archive†

The Open Office's mailing archive is only in html format.
It needs an html-parsing approach to extract the structure
I recommend Jericho html parser over Java or nokogiri over Ruby.

For this task I used Jericho Html Parser in java and extract the structure in as following example

In dev archive
 1997-1 2
 2000-5 1
 2000-10 29

left column is year-month and the total email in each year-month are in the right column My example code is here []

After we have all the lists, we use it as a fetching source file.

fetching script†

usage ./mailscript.sh <Source file's name>

 #!/bin/bash

fname=$1
dir=${fname}_dir

if [ ! -d $dir ]; then
  mkdir $dir
fi

exec<$fname

url=http://openoffice.org/projects/www/lists/${fname}/archive

while read line 
do
 str=`echo $line | cut -d" " -f1`
 tot=`echo $line | cut -d" " -f2`

 mkdir $dir/$str 

 for ((j=0 ; j<$tot; j++))
 do
    #echo $url/${str}/massage/${j}
    wget --output-document=$dir/$str/$j.html $url/${str}/message/${j}
    sleep 5	
 done
done

the input is only the source file name. ~ it will create a directory corresponded to the source file, and the read each line and looping fetch from each email id from year-month to year-month.