- c - 在位数组中找到第一个零
- linux - Unix 显示有关匹配两种模式之一的文件的信息
- 正则表达式替换多个文件
- linux - 隐藏来自 xtrace 的命令
对于所有其他 NLTK 语料库,调用 corpus.raw()
会从文件中生成原始文本。例如:
>>> from nltk.corpus import webtext
>>> webtext.raw()[:10]
'Cookie Man'
但是,当调用 brown.raw()
时,您会得到标记文本。
>>> from nltk.corpus import brown
>>> brown.raw()[:10]
'\n\n\tThe/at '
我已经阅读了我能找到的所有文档,但似乎找不到明显的解释或获取未标记版本的方法。这个语料库被标记而其他语料库没有标记的原因是什么?
最佳答案
import nltk
nltk.download('brown')
nltk.download('nonbreaking_prefixes')
nltk.download('perluniprops')
from nltk.corpus import brown
from nltk.tokenize.moses import MosesDetokenizer
mdetok = MosesDetokenizer()
brown_natural = [mdetok.detokenize(' '.join(sent).replace('``', '"').replace("''", '"').replace('`', "'").split(), return_str=True) for sent in brown.sents()]
for sent in brown_natural:
print(sent)
这是因为 Brown 语料库的“原始”版本已被标记化和标记,即语料库带有标记,这是语料库的原始形式 =)
您可以查看nltk_data
目录中的各个文件:
$ head -n10 nltk_data/corpora/brown/ca01
The/at Fulton/np-tl County/nn-tl Grand/jj-tl Jury/nn-tl said/vbd Friday/nr an/at investigation/nn of/in Atlanta's/np$ recent/jj primary/nn election/nn produced/vbd ``/`` no/at evidence/nn ''/'' that/cs any/dti irregularities/nns took/vbd place/nn ./.
The/at jury/nn further/rbr said/vbd in/in term-end/nn presentments/nns that/cs the/at City/nn-tl Executive/jj-tl Committee/nn-tl ,/, which/wdt had/hvd over-all/jj charge/nn of/in the/at election/nn ,/, ``/`` deserves/vbz the/at praise/nn and/cc thanks/nns of/in the/at City/nn-tl of/in-tl Atlanta/np-tl ''/'' for/in the/at manner/nn in/in which/wdt the/at election/nn was/bedz conducted/vbn ./.
The/at September-October/np term/nn jury/nn had/hvd been/ben charged/vbn by/in Fulton/np-tl Superior/jj-tl Court/nn-tl Judge/nn-tl Durwood/np Pye/np to/to investigate/vb reports/nns of/in possible/jj ``/`` irregularities/nns ''/'' in/in the/at hard-fought/jj primary/nn which/wdt was/bedz won/vbn by/in Mayor-nominate/nn-tl Ivan/np Allen/np Jr./np ./.
如果你想要语料库中的单词,你可以使用 brown.words()
,例如
>>> from nltk.corpus import brown
>>> brown.words()
[u'The', u'Fulton', u'County', u'Grand', u'Jury', ...]
>>> ' '.join(brown.words()[:30])
u"The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place . The jury further said in"
如果你想从一个特定的文件中获取单词:
>>> brown.fileids()[:10] # The first 10 fileids from brown.
[u'ca01', u'ca02', u'ca03', u'ca04', u'ca05', u'ca06', u'ca07', u'ca08', u'ca09', u'ca10']
>>> ' '.join(brown.words('ca01')[:30]) # First 30 words from the 'ca01' file.
u"The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place . The jury further said in"
以及来自特定文件的句子:
>>> brown.sents('ca01')
[[u'The', u'Fulton', u'County', u'Grand', u'Jury', u'said', u'Friday', u'an', u'investigation', u'of', u"Atlanta's", u'recent', u'primary', u'election', u'produced', u'``', u'no', u'evidence', u"''", u'that', u'any', u'irregularities', u'took', u'place', u'.'], [u'The', u'jury', u'further', u'said', u'in', u'term-end', u'presentments', u'that', u'the', u'City', u'Executive', u'Committee', u',', u'which', u'had', u'over-all', u'charge', u'of', u'the', u'election', u',', u'``', u'deserves', u'the', u'praise', u'and', u'thanks', u'of', u'the', u'City', u'of', u'Atlanta', u"''", u'for', u'the', u'manner', u'in', u'which', u'the', u'election', u'was', u'conducted', u'.'], ...]
要打印出单个句子:
>>> for sent in brown.sents('ca01')[:5]: # First 5 sentences.
... print(' '.join(sent))
...
The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place .
The jury further said in term-end presentments that the City Executive Committee , which had over-all charge of the election , `` deserves the praise and thanks of the City of Atlanta '' for the manner in which the election was conducted .
The September-October term jury had been charged by Fulton Superior Court Judge Durwood Pye to investigate reports of possible `` irregularities '' in the hard-fought primary which was won by Mayor-nominate Ivan Allen Jr. .
`` Only a relative handful of such reports was received '' , the jury said , `` considering the widespread interest in the election , the number of voters and the size of this city '' .
The jury said it did find that many of Georgia's registration and election laws `` are outmoded or inadequate and often ambiguous '' .
尝试去标记化语料库相当困惑,可能会或可能不会,但您可以尝试 MosesDetokenizer
:
首先下载MosesDetokenizer需要的数据:
>>> import nltk
>>> nltk.download('perluniprops')
[nltk_data] Downloading package perluniprops to
[nltk_data] /home/ltan/nltk_data...
[nltk_data] Unzipping misc/perluniprops.zip.
True
>>> nltk.download('nonbreaking_prefixes')
[nltk_data] Downloading package nonbreaking_prefixes to
[nltk_data] /home/ltan/nltk_data...
[nltk_data] Package nonbreaking_prefixes is already up-to-date!
True
然后初始化MosesDetokenizer
:
>>> from nltk.tokenize.moses import MosesDetokenizer
>>> mdetok = MosesDetokenizer()
并使用MosesDetokenizer.detokenize()
:
>>> for sent in brown.sents('ca01')[:5]: # First 5 sentences.
... # Join the words in sentences and convert the `` -> "
... # also convert '' -> " and ` -> '
... munged_sentence = ' '.join(sent).replace('``', '"').replace("''", '"').replace('`', "'")
... print(mdetok.detokenize(munged_sentence.split(), return_str=True)) # MosesDetokenizer expects a list of strings as input.
...
The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced "no evidence" that any irregularities took place.
The jury further said in term-end presentments that the City Executive Committee, which had over-all charge of the election, "deserves the praise and thanks of the City of Atlanta" for the manner in which the election was conducted.
The September-October term jury had been charged by Fulton Superior Court Judge Durwood Pye to investigate reports of possible "irregularities" in the hard-fought primary which was won by Mayor-nominate Ivan Allen Jr..
"Only a relative handful of such reports was received", the jury said, "considering the widespread interest in the election, the number of voters and the size of this city".
The jury said it did find that many of Georgia's registration and election laws "are outmoded or inadequate and often ambiguous".
将brown
中的每个句子转换为自然阅读文本:
from nltk.tokenize.moses import MosesDetokenizer
mdetok = MosesDetokenizer()
brown_natural = [mdetok.detokenize(' '.join(sent).replace('``', '"').replace("''", '"').replace('`', "'").split(), return_str=True) for sent in brown.sents()]
[输出]:
>>> for sent in brown_natural:
... print(sent)
... break
...
The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced "no evidence" that any irregularities took place.
关于python - 如何从 Brown 语料库访问原始文档?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/47301140/
关闭。这个问题是opinion-based 。目前不接受答案。 想要改进这个问题吗?更新问题,以便 editing this post 可以用事实和引文来回答它。 . 已关闭 4 年前。 Improv
PowerShell Web Access 允许您通过 Web 浏览器运行 PowerShell cmdlet。它显示了一个基于 Web 的控制台窗口。 有没有办法运行 cmdlet 而无需在控制台窗
我尝试在无需用户登录的情况下访问 Sharepoint 文件。 我可以通过以下任一方式获取访问 token 方法一: var client = new RestClient("https://logi
我目前正在尝试通过 Chrome 扩展程序访问 Google 服务。我的理解是,对于 JS 应用程序,Google 首选的身份验证机制是 OAuth。我的应用目前已成功通过 OAuth 向服务进行身份
假设我有纯抽象类 IHandler 和派生自它的类: class IHandler { public: virtual int process_input(char input) = 0; };
我有一个带有 ThymeLeaf 和 Dojo 的 Spring 应用程序,这给我带来了问题。当我从我的 HTML 文件中引用 CSS 文件时,它们在 Firebug 中显示为中止。但是,当我通过在地
这个问题已经有答案了: JavaScript property access: dot notation vs. brackets? (17 个回答) 已关闭 6 年前。 为什么这不起作用? func
我想将所有流量重定向到 https,只有 robot.txt 应该可以通过 http 访问。 是否可以为 robot.txt 文件创建异常(exception)? 我的 .htaccess 文件: R
我遇到了 LinkedIn OAuth2: "Unable to verify access token" 中描述的相同问题;但是,那里描述的解决方案并不能解决我的问题。 我能够成功请求访问 toke
问题 我有一个暴露给 *:8080 的 Docker 服务容器. 我无法通过 localhost:8080 访问容器. Chrome /curl无限期挂断。 但是如果我使用任何其他本地IP,我就可以访
我正在使用 Google 的 Oauth 2.0 来获取用户的 access_token,但我不知道如何将它与 imaplib 一起使用来访问收件箱。 最佳答案 下面是带有 oauth 2.0 的 I
我正在做 docker 入门指南:https://docs.docker.com/get-started/part3/#recap-and-cheat-sheet-optional docker-co
我正在尝试使用静态 IP 在 AKS 上创建一个 Web 应用程序,自然找到了一个带有 Nginx ingress controller in Azure's documentation 的解决方案。
这是我在名为 foo.js 的文件中的代码。 console.log('module.exports:', module.exports) console.log('module.id:', modu
我试图理解访问键。我读过https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-se
我正在使用 MGTwitterEngine"将 twitter 集成到我的应用程序中。它在 iOS 4.2 上运行良好。当我尝试从任何 iOS 5 设备访问 twitter 时,我遇到了身份验证 to
我试图理解访问键。我读过https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-se
我正在使用以下 API 列出我的 Facebook 好友。 https://graph.facebook.com/me/friends?access_token= ??? 我想知道访问 token 过
401 Unauthorized - Show headers - { "error": { "errors": [ { "domain": "global", "reas
我已经将我的 django 应用程序部署到 heroku 并使用 Amazon s3 存储桶存储静态文件,我发现从 s3 存储桶到 heroku 获取数据没有问题。但是,当我测试查看内容存储位置时,除
我是一名优秀的程序员,十分优秀!