gpt4 book ai didi

java - OpenNLP 模型构建器插件无法继续

转载 作者:行者123 更新时间:2023-11-30 09:19:00 27 4
gpt4 key购买 nike

我正在使用 OpenNLP模型构建器插件来创建更好的 NER 模型。据此post ,我使用了markg发布的代码:

public class ModelBuilderAddonUse {

private static List<String> getSentencesFromSomewhere() throws Exception
{
List<String> list = new ArrayList<String>();
BufferedReader reader = new BufferedReader(new FileReader("D:\\Work\\workspaces\\default\\UpdateModel\\documentrequirements.docx"));
String line;
while ((line = reader.readLine()) != null)
{
list.add(line);
}
reader.close();
return list;

}

public static void main(String[] args) throws Exception {
/**
* establish a file to put sentences in
*/
File sentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\sentences.text");

/**
* establish a file to put your NER hits in (the ones you want to keep based
* on prob)
*/
File knownEntities = new File("D:\\Work\\workspaces\\default\\UpdateModel\\knownentities.txt");

/**
* establish a BLACKLIST file to put your bad NER hits in (also can be based
* on prob)
*/
File blacklistedentities = new File("D:\\Work\\workspaces\\default\\UpdateModel\\blentities.txt");

/**
* establish a file to write your annotated sentences to
*/
File annotatedSentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\annotatedSentences.txt");

/**
* establish a file to write your model to
*/
File theModel = new File("D:\\Work\\workspaces\\default\\UpdateModel\\nl-ner-person.bin");


//------------create a bunch of file writers to write your results and sentences to a file

FileWriter sentenceWriter = new FileWriter(sentences, true);
FileWriter blacklistWriter = new FileWriter(blacklistedentities, true);
FileWriter knownEntityWriter = new FileWriter(knownEntities, true);

//set some thresholds to decide where to write hits, you don't have to use these at all...
double keeperThresh = .95;
double blacklistThresh = .7;


/**
* Load your model as normal
*/
TokenNameFinderModel personModel = new TokenNameFinderModel(new File("D:\\Work\\workspaces\\default\\UpdateModel\\nl-ner-person.bin"));
NameFinderME personFinder = new NameFinderME(personModel);
/**
* do your normal NER on the sentences you have
*/
for (String s : getSentencesFromSomewhere()) {
sentenceWriter.write(s.trim() + "\n");
sentenceWriter.flush();

String[] tokens = s.split(" ");//better to use a tokenizer really
Span[] find = personFinder.find(tokens);
double[] probs = personFinder.probs();
String[] names = Span.spansToStrings(find, tokens);
for (int i = 0; i < names.length; i++) {
//YOU PROBABLY HAVE BETTER HEURISTICS THAN THIS TO MAKE SURE YOU GET GOOD HITS OUT OF THE DEFAULT MODEL
if (probs[i] > keeperThresh) {
knownEntityWriter.write(names[i].trim() + "\n");
}
if (probs[i] < blacklistThresh) {
blacklistWriter.write(names[i].trim() + "\n");
}
}
personFinder.clearAdaptiveData();
blacklistWriter.flush();
knownEntityWriter.flush();
}
//flush and close all the writers
knownEntityWriter.flush();
knownEntityWriter.close();
sentenceWriter.flush();
sentenceWriter.close();
blacklistWriter.flush();
blacklistWriter.close();

/**
* THIS IS WHERE THE ADDON IS GOING TO USE THE FILES (AS IS) TO CREATE A NEW MODEL. YOU SHOULD NOT HAVE TO RUN THE FIRST PART AGAIN AFTER THIS RUNS, JUST NOW PLAY WITH THE
* KNOWN ENTITIES AND BLACKLIST FILES AND RUN THE METHOD BELOW AGAIN UNTIL YOU GET SOME DECENT RESULTS (A DECENT MODEL OUT OF IT).
*/
DefaultModelBuilderUtil.generateModel(sentences, knownEntities, blacklistedentities, theModel, annotatedSentences, "person", 3);


}
}

它也运行,但我的输出在以下位置退出:

    annotated sentences: 1862
knowns: 58
Building Model using 1862 annotations
reading training data...

但是在 post 中的示例中它应该像这样进一步发展:

Indexing events using cutoff of 5

Computing event counts... done. 561755 events
Indexing... done.
Sorting and merging events... done. Reduced 561755 events to 127362.
Done indexing.
Incorporating indexed data for training...
done.
Number of Event Tokens: 127362
Number of Outcomes: 3
Number of Predicates: 106490
...done.

任何人都可以帮我解决这个问题,以便我可以生成模型吗?我确实进行了很多搜索,但找不到任何关于它的好的文档。真的很感激,谢谢。

最佳答案

更正训练数据文件的路径,如下所示:

File sentences = new File("D:/Work/workspaces/default/UpdateModel/sentences.text");

而不是

File sentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\sentences.text");

更新

这是通过将文件添加到项目文件夹中来使用的。尝试这样 -

File sentences = new File("src/training/resources/CreateModel/sentences.txt");

Check my respository for reference on Github

这应该有帮助。

关于java - OpenNLP 模型构建器插件无法继续,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/46950036/

27 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com