维基百科语料库

原文：

Tagged and Cleaned Wikipedia (TC Wikipedia) and its Ngram

Wikipedia is a relatively big and consistent resource for NLP researchers to work with. However, it is not straightforward even to extract meaningful sentences and portions which are useful for the research. In order to avoid the duplication of the laborious efforts, we will make our "Tagged and Cleand Wikipedia (TC Wikipedia) available for the community.

5 Data (See some samples below)

Static Wikipedia data : Wikipedia as of 18:12, June 8, 2008 version. The static data was downloaded in November 2008.

List of headwords : List of headword with ID number (used in the other data), Filename, Category and Redirection list

articles.lst.gz (134MB)

Infobox data : Infobox data. html format of infobox data (html) and text format seperated by newlines and tabs (txt).infoboxes.html.gz (551MB)

infoboxes.txt.gz (207MB)

Sentences tagged by NLP tools : All sentences which are thought to be contents are tagged by NLP tools (now it has tokenization, POS, NE and link URL information).wikipedia-tagged2_1.txt.gz (12GB, Get through FTP)

This new version includes two POS, one chunk and two NE information.

Old version (1.1) includes only one POS (Stanford), one chunk and one NE (Stanford) information. (SEE BELOW)

TWO BUGS in version 1.1

There are several empty or junk lines outside of #s-doc and #e-doc tags. Use the tag to extract meaningful text contents, rather than just excluding the lines starting with "#[se]-(label)".

Some NE labels have "0 (zero)" rather than "O (oh)". Please replace them to "O".

These bugs will be fixed in the next distribution (which includes more annotations). Sorry for the inconvenience.

Ngram : ngram data of the corpus (no frequency threshold, no filtering of any types of tokens, no normalization of numbers)README

1gram (31MB)

2gram (447MB)

3gram (1.9GB)

4gram (4.3GB)

5gram (7.1GB)

6gram (10GB)

7gram (13GB)

描述： 包含4400000篇文章 及19亿单词，可用来做语言建模

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。