python - numpy.save() 内的 Pickle TypeError

Question

我有一个函数来计算特征，然后将特征保存到 pickle 中。

test_knn_feats = NNF.predict(X_test) 
np.save('data/knn_feats_%s_test.npy' % metric , test_knn_feats)

在函数中，如果n_jobs大于1，则将执行下面的代码。

fest_feats =[]
pool = Pool(processes = self.n_jobs) 
for i in range(X.shape[0]):
    test_feats.append(pool.apply_async(self.get_features_for_one(X[i:i+1])))
pool.close()
pool.join()

return np.vstack(test_feats)

但是，出现错误:

TypeError                                 Traceback (most recent call last)
<ipython-input-96-4f707b7cd533> in <module>()
     12     print(test_knn_feats)
     13     # Dump the features to disk
---> 14     np.save('data/knn_feats_%s_test.npy' % metric , test_knn_feats)

/opt/conda/lib/python3.6/site-packages/numpy/lib/npyio.py in save(file, arr, allow_pickle, fix_imports)
    507         arr = np.asanyarray(arr)
    508         format.write_array(fid, arr, allow_pickle=allow_pickle,
--> 509                            pickle_kwargs=pickle_kwargs)
    510     finally:
    511         if own_fid:

/opt/conda/lib/python3.6/site-packages/numpy/lib/format.py in write_array(fp, array, version, allow_pickle, pickle_kwargs)
    574         if pickle_kwargs is None:
    575             pickle_kwargs = {}
--> 576         pickle.dump(array, fp, protocol=2, **pickle_kwargs)
    577     elif array.flags.f_contiguous and not array.flags.c_contiguous:
    578         if isfileobj(fp):

函数get_features_for_one将返回一个列表，如下所示。

...
knn_feats = np.hstack(return_list)
assert knn_feats.shape == (239,) or knn_feats.shape == (239, 1)
return knn_feats

*更新:

test_feats =[]      
pool = Pool(processes = self.n_jobs) 
for i in range(X.shape[0]):
    test_feats.append(pool.apply_async(self.get_features_for_one, (X[i:i+1],)))
test_feats= [res.get() for res in test_feats]        
pool.close()
pool.join()
return np.vstack(test_feats)

Answer 1

这里有两个主要错误:

test_feats =[] # you called it fest_feats, I assume a typo
pool = Pool(processes = self.n_jobs) 
for i in range(X.shape[0]):
    test_feats.append(pool.apply_async(self.get_features_for_one(X[i:i+1])))
    pool.close()
    pool.join()

return np.vstack(test_feats)

1. 首先，您创建一个池。然后，对于每个i，您提交一项作业，然后关闭并加入池中。您应该只在最后、在循环之外关闭并加入池一次。

2. test_feats 最终成为“ future ”列表，而不是实际数据。所以 vstack() 对它们来说是没有意义的。您需要对每个 future 调用 get() 来获取 get_features_for_one() 的结果，然后将该列表传递给 vstack()。例如np.vstack([res.get() for res in test_feats])。

简而言之，你的问题与你最终从 numpy.save() 收到的 TypeError 无关——你的问题是你的逻辑完全被破坏了，你的数据不是你想要的。认为是这样。