Wikipedia̓c[Wik-IE

1. Wik-IEƂ
Wik-IEHadoopvbgtH[œ삷Wikipediaf[^̓c[łB
Wik-IE𗘗p邱ƂWikipediaŔzzĂf[^t@C𒼐ډ͂A
ėpeՂȃ^u؂`̃eLXgt@CƂďo͂܂B
܂ɔˑłׂĂ̌łɗp\łB


2. UłƃX^hA
Wik-IEɂ͂łHadoopĂ̕UłƁA
HadoopȂ̃X^hAł܂B
X^hAłɂHadoop0.16.4gݍ܂Ă܂B
Uł̊JɎgĂ̂Hadoop0.16.4łB
ȊÕo[WHadoopœ삷ۏ؂͂܂B


3. gpł@\
node y[W̃^CgƎނ\f[^
edge LEJeSE_CNgԂ̊֌W\f[^
variation y[WԃÑAJ[eLXgƃNGg̃^Cg擾\Lf[^
interWiki ԃN\f[^
isbn LɎQlƂċLqĂoŕISBNR[h
history ߋ̕ҏWpxTPʂŏWvf[^


4. f[^t@C
Wik-IEWikipediaŔzzĂf[^t@C͂܂B

{ł̏ꍇAhistoryȊŐ@\Ŏgpt@C
http://download.wikimedia.org/jawiki/latest/
ŌJĂ
jawiki-latest-pages-meta-current.xml.bz2
Ƃt@CWJĂłXMLt@CłB
͓{WikipediȃOy[W̃f[^łB

history@\gꍇ
jawiki-latest-stub-meta-history.xml.gz
Ƃt@CWJĂłXMLt@CłB
͓{Wikipedia̕ҏŴҏWҏW҂Ȃǂ̃f[^łB

̃t@CWJƐGB鋐ȃt@Cł̂ŒӂĂB

{ňȊÕf[^gꍇꂼΉt@CA
płȂ
http://download.wikimedia.org/enwiki/latest/
hCcłȂ
http://download.wikimedia.org/dewiki/latest/
Ȃǌ̕URL_E[hĂB


5. s@
EU
hadoopsł郆[UɂȂA
hadoop jar wikie_distribute.jar -function input.xml output

EX^hA
java -jar wikie_standalone.jar -function input.xml output

function3ŋ@\(node, edgeȂ)łB
input.xml4Őf[^t@C̃pXłB
output/͌ʂo͂fBNg̃pXłBsɂ݂͑ȂfBNgw肷Kv܂B

ႦΎۂɂ͎̂悤Ɏs܂B
hadoop jar wikie_distribute.jar -node jawiki-latest-pages-meta-current.xml node_result

6. o̓t@C
o̓t@Cׂ͂ă^u؂t@CłB

Enode
id    title    kind
id: GgID
title: Gg̃^Cg
kind: Gg̎ʁBleafiӂ̋Ljnode(WikipediaCategoryjredirect3ނ̂ꂩłB

Eedge
id(current)    id(target)    relation
id(current): edgě̃GgID
id(target): edge̐̃GgID
relation: nodeԂ̊֌WBhypernymiJeS֌Wjtargeti_CNg֌Wj2ނ̂ꂩłB

EinterWiki
id    title1    title2    title3...
id: GgID
title: ԃNBp̃vtBbNX^Cg̑Oɕt^܂B1sɏA˂Ă܂B

Evariation
title    variation
title: Gg̃^Cg
title: \Lꕶ

Eisbn
id    isbn
id: GgID
isbn: ISBNR[hBISBNR[hɂ10̋Ki13̐VKi܂A10̃R[h͎I13ɕϊ܂B

Ehistory
title    date    frequency
title: Gg̃^Cg
date: ̏T̏߂̓tB2008-7-7̂悤Ȍ`B
frequency: ҏW񐔁B1TԂ̂̂ɓꃆ[UҏWĂĂJEg1̂݁B


7. CZX
Wik-IEApache License 2.0ŔzzĂ܂B


8. 
X
https://sourceforge.jp/projects/wik-ie