python-2.7 - 如何比较使用 scikit-learn 库 load_svmlight_file 存储的 2 个稀疏矩阵？

Question

我正在尝试比较测试和训练数据集中存在的特征向量。这些特征向量使用 scikitlearn 库 load_svmlight_file 以稀疏格式存储。两个数据集的特征向量的维度相同。但是，我收到此错误:“具有多个元素的数组的真值不明确。使用 a.any() 或 a.all()。”

为什么会出现此错误？我该如何解决？

提前致谢!

from sklearn.datasets import load_svmlight_file
pathToTrainData="../train.txt"
pathToTestData="../test.txt"
X_train,Y_train= load_svmlight_file(pathToTrainData);
X_test,Y_test= load_svmlight_file(pathToTestData);

for ele1 in X_train:
    for ele2 in X_test:
        if(ele1==ele2):
           print "same vector"


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-c1f145f984a6> in <module>()
      7 for ele1 in X_train:
      8     for ele2 in X_test:
----> 9         if(ele1==ele2):
     10            print "same vector"

/Users/rkasat/anaconda/lib/python2.7/site-packages/scipy/sparse/base.pyc in __bool__(self)
    181             return True if self.nnz == 1 else False
    182         else:
--> 183             raise ValueError("The truth value of an array with more than one "
    184                              "element is ambiguous. Use a.any() or a.all().")
    185     __nonzero__ = __bool__

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

Answer 1

您可以使用此条件来检查两个稀疏数组是否完全相等，而无需对它们进行加密:

if (ele1 - ele2).nnz == 0:
    # Matched, do something ...

nnz 属性给出了稀疏数组中非零元素的数量。

一些简单的测试运行来显示差异:

import numpy as np
from scipy import sparse

A = sparse.rand(10, 1000000).tocsr()

def benchmark1(A):
    for s1 in A:
        for s2 in A:
            if (s1 - s2).nnz == 0:
                pass

def benchmark2(A):
    for s1 in A:
        for s2 in A:
            if (s1.toarray() == s2).all() == 0:
                pass

%timeit benchmark1(A)
%timeit benchmark2(A)

一些结果:

# Computer 1
10 loops, best of 3: 36.9 ms per loop # with nnz
1 loops, best of 3: 734 ms per loop # with toarray

# Computer 2
10 loops, best of 3: 28 ms per loop
1 loops, best of 3: 312 ms per loop