machine-learning - 如何使用 Java 中的 Weka 将文本转换为 TF-IDF 格式？

Question

假设，我有以下具有两个属性的示例 ARFF 文件:

(1) 情绪:积极 [1] 或消极 [-1]

(2) 推文:文本

@relation sentiment_analysis

@attribute sentiment {1, -1}
@attribute tweet string

@data
-1,'is upset that he can\'t update his Facebook by texting it... and might cry as a result  School today also. Blah!'
-1,'@Kenichan I dived many times for the ball. Managed to save 50\%  The rest go out of bounds'
-1,'my whole body feels itchy and like its on fire '
-1,'@nationwideclass no, it\'s not behaving at all. i\'m mad. why am i here? because I can\'t see you all over there. '
-1,'@Kwesidei not the whole crew '
-1,'Need a hug '
1,'@Cliff_Forster Yeah, that does work better than just waiting for it  In the end I just wonder if I have time to keep up a good blog.'
1,'Just woke up. Having no school is the best feeling ever '
1,'TheWDB.com - Very cool to hear old Walt interviews!  ? http://blip.fm/~8bmta'
1,'Are you ready for your MoJo Makeover? Ask me for details '
1,'Happy 38th Birthday to my boo of alll time!!! Tupac Amaru Shakur '
1,'happy #charitytuesday @theNSPCC @SparksCharity @SpeakingUpH4H '

我想将第二个属性的值转换为等效的 TF-IDF 值。

顺便说一句，我尝试了以下代码，但其输出 ARFF 文件不包含相应实例的正(1)值的第一个属性。

// Set the tokenizer
NGramTokenizer tokenizer = new NGramTokenizer();
tokenizer.setNGramMinSize(1);
tokenizer.setNGramMaxSize(1);
tokenizer.setDelimiters("\\W");

// Set the filter
StringToWordVector filter = new StringToWordVector();
filter.setAttributeIndicesArray(new int[]{1});
filter.setOutputWordCounts(true);
filter.setTokenizer(tokenizer);
filter.setInputFormat(inputInstances);
filter.setWordsToKeep(1000000);
filter.setDoNotOperateOnPerClassBasis(true);
filter.setLowerCaseTokens(true);
filter.setTFTransform(true);
filter.setIDFTransform(true);

// Filter the input instances into the output ones
outputInstances = Filter.useFilter(inputInstances, filter);

示例输出 ARFF 文件:

@data
{0 -1,320 1,367 1,374 1,397 1,482 1,537 1,553 1,681 1,831 1,1002 1,1033 1,1112 1,1119 1,1291 1,1582 1,1618 1,1787 1,1810 1,1816 1,1855 1,1939 1,1941 1}
{0 -1,72 1,194 1,436 1,502 1,740 1,891 1,935 1,1075 1,1256 1,1260 1,1388 1,1415 1,1579 1,1611 1,1818 2,1849 1,1853 1}
{0 -1,374 1,491 1,854 1,873 1,1120 1,1121 1,1197 1,1337 1,1399 1,2019 1}
{0 -1,240 1,359 2,369 1,407 1,447 1,454 1,553 1,1019 1,1075 3,1119 1,1240 1,1244 1,1373 1,1379 1,1417 1,1599 1,1628 1,1787 1,1824 1,2021 1,2075 1}
{0 -1,198 1,677 1,1379 1,1818 1,2019 1}
{0 -1,320 1,1070 1,1353 1}
{0 -1,210 1,320 2,477 2,867 1,1020 1,1067 1,1075 1,1212 1,1213 1,1240 1,1373 1,1404 1,1542 1,1599 1,1628 1,1815 1,1847 1,2067 1,2075 1}
{179 1,1815 1}
{298 1,504 1,662 1,713 1,752 1,1163 1,1275 1,1488 1,1787 1,2011 1,2075 1}
{144 1,785 1,1274 1}
{19 1,256 1,390 1,808 1,1314 1,1350 1,1442 1,1464 1,1532 1,1786 1,1823 1,1864 1,1908 1,1924 1}
{84 1,186 1,320 1,459 1,564 1,636 1,673 1,810 1,811 1,966 1,997 1,1094 1,1163 1,1207 1,1592 1,1593 1,1714 1,1836 1,1853 1,1964 1,1984 1,1997 2,2058 1}
{9 1,1173 1,1768 1,1818 1}
{86 1,935 1,1112 1,1337 1,1348 1,1482 1,1549 1,1783 1,1853 1}

正如您所看到的，前几个实例没问题(因为它们包含 -1 类以及其他功能)，但最后剩余的实例不包含正类属性 (1)。

我的意思是，输出 ARFF 文件的最后一个实例中应该有 {0 1,...} 作为第一个属性，但它丢失了。

Answer 1

您必须在 java 程序中明确指定哪个是您的类属性，因为当您应用 StringToWordVector 过滤器时，您的输入将被划分为指定的 n 元语法。因此，一旦 StringToWordVector 对输入进行矢量化，类属性位置就会发生变化。您可以使用重新排序文件管理器，它最终会将类属性放置在最后一个位置，Weka 将选择最后一个属性作为类属性。

有关 Weka 中重新排序的更多信息，请访问 http://weka.sourceforge.net/doc.stable-3-8/weka/filters/unsupervised/attribute/Reorder.html 。示例 5 位于 http://www.programcreek.com/java-api-examples/index.php?api=weka.filters.unsupervised.attribute.Reorder可以帮助您重新排序。

希望有帮助。