Extracting HTML data from an OSS project
をテンプレートにして作成
[
トップ
] [
新規
|
一覧
|
検索
|
最終更新
|
ヘルプ
]
開始行:
[[Socio-metrics]]
Case: Open Office Project
*For Bug-Tracking System [#u8522a57]
**fetching script [#x3d81b16]
usage: crawl_oof.sh <start> <end>
#!/bin/bash
START=$1
END=$2
mkdir -p data
mkdir -p history
WGET_CMD="wget \
--tries=5 \
--connect-timeout=5 \
--wait=10 \
--random-wait \
--no-verbose \
--no-check-certificate" \
for (( i=$START; i<=$END ; i++ ))
do
$WGET_CMD --output-document=data/bug_$i.html http://open...
sleep 5
$WGET_CMD --output-document=history/history_$i.html http...
sleep 5
done
This script fetch both bug page itself and and its histor...
The execution is (if saving in crawl_oof.sh) ./crawl_oof....
Both contents will be saved in ./data and ./history folder
- note that if an error existed (most comes from proxy er...
du -a | grep ^0
for probing the error files, that needs to be fetched man...
*For the Mailing-list Archive [#aff8bdbc]
The Open Office's mailing archive is only in html format.~
It needs an html-parsing approach to extract the structur...
I recommend Jericho html parser over Java or nokogiri ove...
For this task I used Jericho Html Parser in java and extr...
In dev archive
1997-1 2
2000-5 1
2000-10 29
left column is year-month and the total email in each yea...
My example code is here []
After we have all the lists, we use it as a fetching sour...
** fetching script [#a408a3ac]
usage ./mailscript.sh <Source file's name>
#!/bin/bash
fname=$1
dir=${fname}_dir
if [ ! -d $dir ]; then
mkdir $dir
fi
exec<$fname
url=http://openoffice.org/projects/www/lists/${fname}/ar...
while read line
do
str=`echo $line | cut -d" " -f1`
tot=`echo $line | cut -d" " -f2`
mkdir $dir/$str
for ((j=0 ; j<$tot; j++))
do
#echo $url/${str}/massage/${j}
wget --output-document=$dir/$str/$j.html $url/${str}...
sleep 5
done
done
the input is only the source file name. ~
it will create a directory corresponded to the source fil...
and looping fetch from each email id from year-month to y...
終了行:
[[Socio-metrics]]
Case: Open Office Project
*For Bug-Tracking System [#u8522a57]
**fetching script [#x3d81b16]
usage: crawl_oof.sh <start> <end>
#!/bin/bash
START=$1
END=$2
mkdir -p data
mkdir -p history
WGET_CMD="wget \
--tries=5 \
--connect-timeout=5 \
--wait=10 \
--random-wait \
--no-verbose \
--no-check-certificate" \
for (( i=$START; i<=$END ; i++ ))
do
$WGET_CMD --output-document=data/bug_$i.html http://open...
sleep 5
$WGET_CMD --output-document=history/history_$i.html http...
sleep 5
done
This script fetch both bug page itself and and its histor...
The execution is (if saving in crawl_oof.sh) ./crawl_oof....
Both contents will be saved in ./data and ./history folder
- note that if an error existed (most comes from proxy er...
du -a | grep ^0
for probing the error files, that needs to be fetched man...
*For the Mailing-list Archive [#aff8bdbc]
The Open Office's mailing archive is only in html format.~
It needs an html-parsing approach to extract the structur...
I recommend Jericho html parser over Java or nokogiri ove...
For this task I used Jericho Html Parser in java and extr...
In dev archive
1997-1 2
2000-5 1
2000-10 29
left column is year-month and the total email in each yea...
My example code is here []
After we have all the lists, we use it as a fetching sour...
** fetching script [#a408a3ac]
usage ./mailscript.sh <Source file's name>
#!/bin/bash
fname=$1
dir=${fname}_dir
if [ ! -d $dir ]; then
mkdir $dir
fi
exec<$fname
url=http://openoffice.org/projects/www/lists/${fname}/ar...
while read line
do
str=`echo $line | cut -d" " -f1`
tot=`echo $line | cut -d" " -f2`
mkdir $dir/$str
for ((j=0 ; j<$tot; j++))
do
#echo $url/${str}/massage/${j}
wget --output-document=$dir/$str/$j.html $url/${str}...
sleep 5
done
done
the input is only the source file name. ~
it will create a directory corresponded to the source fil...
and looping fetch from each email id from year-month to y...
ページ名:
![[PukiWiki]](image/wadanyan.png "[PukiWiki]")
