- c - 在位数组中找到第一个零
- linux - Unix 显示有关匹配两种模式之一的文件的信息
- 正则表达式替换多个文件
- linux - 隐藏来自 xtrace 的命令
我发现gensim有BM25排序功能。但是,我找不到教程如何使用它。
就我而言,我有一个疑问。从搜索引擎中检索到的一些文档。如何使用 gensim BM 25 排序来比较查询和文档以找到最相似的?
我是 gensim 的新手。谢谢。
查询:
"experimental studies of creep buckling ."
文档 1:
" the 7 x 7 in . hypersonic wind tunnel at rae farnborough, part 1, design, instrumentation and flow visualization techniques . this is the first of three parts of the calibration report on the r.a.e. some details of the design and lay-out of the plant are given, together with the calculated performance figures, and the major components of the facility are briefly described . the instrumentation provided for the wind-tunnel is described in some detail, including the optical and other methods of flow visualization used in the tunnel . later parts will describe the calibration of the flow in the working-section, including temperature measurements . a discussion of the heater performance will also be included as well as the results of tests to determine starting and running pressure ratios, blockage effects, model starting loads, and humidity of the air flow ."
文档 2:
" the 7 in. x 7 in. hypersonic wind tunnel at r.a.e. farnborough part ii. heater performance . tests on the storage heater, which is cylindrical in form and mounted horizontally, show that its performance is adequate for operation at m=6.8 and probably adequate for flows at m=8.2 with the existing nozzles . in its present state, the maximum design temperature of 680 degrees centigrade for operation at m=9 cannot be realised in the tunnel because of heat loss to the outlet attachments of the heater and quick-acting valve which form, in effect, a large heat sink . because of this heat loss there is rather poor response of stagnation temperature in the working section at the start of a run . it is hoped to cure this by preheating the heater outlet cone and the quick-acting valve . at pressures greater than about 100 p.s.i.g. free convection through the fibrous thermal insulation surrounding the heated core causes the top of the heater shell to become somewhat hotter than the bottom, which results in /hogging/ distortion of the shell . this free convection cools the heater core and a vertical temperature gradient is set up across it after only a few minutes at high pressure . modifications to be incorporated in the heater to improve its performance are described ."
文档 3:
" supersonic flow at the surface of a circular cone at angle of attack . formulas for the inviscid flow properties on the surface of a cone at angle of attack are derived for use in conjunction with the m.i.t. cone tables . these formulas are based upon an entropy distribution on the cone surface which is uniform and equal to that of the shocked fluid in the windward meridian plane . they predict values for the flow variables which may differ significantly from the corresponding values obtained directly from the cone tables . the differences in the magnitudes of the flow variables computed by the two methods tend to increase with increasing free-stream mach number, cone angle and angle of attack ."
文档 4:
" theory of aircraft structural models subjected to aerodynamic heating and external loads . the problem of investigating the simultaneous effects of transient aerodynamic heating and external loads on aircraft structures for the purpose of determining the ability of the structure to withstand flight to supersonic speeds is studied . by dimensional analyses it is shown that .. constructed of the same materials as the aircraft will be thermally similar to the aircraft with respect to the flow of heat through the structure will be similar to those of the aircraft when the structural model is constructed at the same temperature as the aircraft . external loads will be similar to those of the aircraft . subjected to heating and cooling that correctly simulate the aerodynamic heating of the aircraft, except with respect to angular velocities and angular accelerations, without requiring determination of the heat flux at each point on the surface and its variation with time . acting on the aerodynamically heated structural model to those acting on the aircraft is determined for the case of zero angular velocity and zero angular acceleration, so that the structural model may be subjected to the external loads required for simultaneous simulation of stresses and deformations due to external loads ."
最佳答案
完全公开我没有任何使用 BM25 排名的经验,但是我对 gensim 的 TF-IDF 和 LSI 分布式模型以及 gensim 的相似性指数有相当多的经验。
作者在保持可读代码库方面做得非常好,所以如果您再次遇到类似问题,我建议直接跳转到源代码。
查看源代码:https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/summarization/bm25.py
所以我用上面粘贴的文档初始化了一个 BM25()
对象。
看起来我们的老 friend Radim 没有为我们计算 average_idf
的函数,这没什么大不了的,我们可以为了我们的原因而抄袭第 65 行:
average_idf = sum(map(lambda k: float(bm25.idf[k]), bm25.idf.keys()))/len(bm25.idf.keys())
那么,如果我正确理解 get_scores
的初衷,你应该得到每个 BM25 分数相对于你的原始查询,只需做
scores = bm_25_object.get_scores(query_doc, average_idf)
返回每个文档的所有分数,然后,如果我根据我在维基百科页面上阅读的内容理解 BM25 排名:https://en.wikipedia.org/wiki/Okapi_BM25
您应该能够只选择得分最高的文档,如下所示:
best_result = docs[scores.index(max(scores))]
所以第一个文档应该与您的查询最相关?无论如何,我希望这就是您所期待的,我希望这在某种程度上有所帮助。祝你好运!
关于python - 如何在python中使用gensim BM25排名,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/40966014/
比方说, word2vec.model 是我训练好的 word2vec 模型。当出现词汇外单词( oov_word )时,我计算向量 vec 使用 compute_vec(oov_word) 方法。现
我有一个现有的 gensim Doc2Vec 模型,我正在尝试对训练集以及模型进行迭代更新。 我拿新文件,照常进行预处理: stoplist = nltk.corpus.stopwords.words
使用 gensim.models.LdaMallet 有什么区别和 gensim.models.LdaModel ?我注意到参数并不完全相同,想知道什么时候应该使用一个而不是另一个? 最佳答案 TL;
我训练了一个 gensim.models.doc2vec.Doc2Vec 模型 d2v_model = Doc2Vec(sentences, size=100, window=8, min_count
我在 gensim 中有一个 word2vec 模型,训练了 98892 个文档。对于句子数组中不存在的任何给定句子(即我训练模型的集合),我需要用该句子更新模型,以便下次查询时给出一些结果。我这样做
我对 Gensim 很陌生,我正在尝试使用 word2vec 模型训练我的第一个模型。我看到所有参数都非常简单易懂,但是我不知道如何跟踪模型的损失以查看进度。此外,我希望能够在每个 epoch 之后获
请帮助我理解如何 TaggedDocument 之间的区别和 LabeledSentence的 gensim作品。我的最终目标是使用 Doc2Vec 进行文本分类模型和任何分类器。我正在关注这个 bl
尝试使用以下代码行在 gensim 中加载文件: model = gensim.models.KeyedVectors.load_word2vec_format(r"C:/Users/dan/txt_
我有一组用神经网络训练的嵌入,与 gensim 的 word2vec 无关。 我想使用这些嵌入作为 gensim.Word2vec 中的初始权重。 现在我看到的是,我可以model.load(SOME
我尝试使用 gensim 导入 import gensim 但出现以下错误 ImportError Traceback (most rece
我正在关注 https://radimrehurek.com/gensim/wiki.html#latent-dirichlet-allocation 上的“英语维基百科”gensim 教程 它解释了
我正在使用 24 核虚拟 CPU 和 100G 内存来训练 Doc2Vec 与 Gensim,但无论修改核数,CPU 的使用率始终在 200% 左右。 top htop 上面两张图显示了cpu使用率,
在将文本文档列表转换为语料库字典,然后使用以下方法将其转换为词袋模型之后: dictionary = gensim.corpora.Dictionary(docs) # docs is a list
我已经使用 Gensim 3.8.0 训练了一个 Word2Vec 模型。后来我尝试在 GCP 上使用使用 Gensim 4.0.o 的预训练模型。我使用了以下代码: model = KeyedVec
我正在构建一个多标签文本分类程序,我正在尝试使用 OneVsRestClassifier+XGBClassifier 对文本进行分类。最初,我使用 Sklearn 的 Tf-Idf 矢量化来矢量化文本
我发现关于 word2vec.similarity() 的警告如下: >d:\python\lib\site-packages\gensim\matutils.py:737: FutureWarnin
我正在尝试使用版本为 3.6 的 Python 的 Gensim 库运行程序。 每当我运行该程序时,我都会遇到这些语句: C:\Python36\lib\site-packages\gensim-2.
我有一个通过 Java 中的 Mallet 训练的 LDA 模型。 Mallet LDA 模型生成了三个文件,这使我能够从文件运行模型并推断新文本的主题分布。 现在我想实现一个 Python 工具,它
我正在使用gensim doc2vec。我想知道是否有任何有效的方法来了解doc2vec的词汇量。一种粗略的方法是计算单词总数,但是如果数据量很大(1GB或更多),那么这将不是一种有效的方法。 最佳答
documentation有点不清楚如何将 fasttext 模型保存到磁盘 - 如何在参数中指定路径,我尝试这样做,但失败并出现错误 文档中的示例 >>> from gensim.test.util
我是一名优秀的程序员,十分优秀!