python - 如何像 `squareform` pd.DataFrame 一样索引压缩的 `Pandas` 矩阵？ (Python 3)

Question

我使用pandas对于我所有的平方距离/相似性/相异性矩阵，直到最近它都工作得很好。我一直在处理越来越大的数据集，创建/存储这些具有 100k 属性的成对数据点更是一项艰巨的任务(1e5**2 = 10,000,000,000 个单元格在我的矩阵中)。我意识到这效率低下且计算简单，但在处理具有 1000 个属性的数据集时它从来都不是问题。

我的问题是，如何使用 pandas 的索引功能存储效率为scipy.spatial.distance.squareform凝聚矩阵？

我可以编写一个可以执行此操作的工具，但如果 pandas 中已存在该工具我更喜欢使用这种方法。如果pandas中不存在这个并且应该是功能请求，我可以在 GitHub 上这样做。

from sklearn.datasets import load_iris
import pandas as pd
from scipy.spatial import distance

# Load data
X = pd.DataFrame(load_iris().data, columns = load_iris().feature_names, index = map(lambda x:"iris_%d"%x, range(150)))

# Get distance matrix (labeled)
DF_dism = 1 - X.T.corr()
DF_dism.shape
(150, 150)

# Index the matrix to get a pairwise distance between 2 labeled objects
print(DF_dism.loc["iris_5","iris_140"])
# 0.410805649878

# Condense into squareform
distance.squareform(DF_dism)

# Now it's not labeled and I can't index it
# array([  4.00133876e-03,   2.60889537e-05,   1.83154822e-03, ...,
#          4.29187441e-03,   5.53987884e-03,   8.41229000e-05])

版本:

Python= 3.6.0 |Anaconda 4.3.0 (x86_64)| (default, Dec 23 2016, 13:19:00) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Pandas= 0.19.2
SciPy= 0.18.1

Answer 1

更新:从成对距离数组生成多索引系列

In [236]: from itertools import combinations

In [237]: s = pd.Series(distance.pdist(X, 'correlation'),
                        index=pd.MultiIndex.from_tuples(tuple(combinations(X.index, 2))))

In [238]: s
Out[238]:
iris_0    iris_1      0.004001
          iris_2      0.000026
          iris_3      0.001832
          iris_4      0.000653
          iris_5      0.000414
          iris_6      0.001189
          iris_7      0.000462
          iris_8      0.001923
          iris_9      0.003448
          iris_10     0.000015
                        ...
iris_145  iris_146    0.001535
          iris_147    0.002642
          iris_148    0.013283
          iris_149    0.015462
iris_146  iris_147    0.003431
          iris_148    0.011531
          iris_149    0.013519
iris_147  iris_148    0.004292
          iris_149    0.005540
iris_148  iris_149    0.000084
dtype: float64

“标记”访问示例:

In [239]: s.loc[("iris_5","iris_140")]
Out[239]: 0.41080564987753798

结果系列的形状:

In [240]: s.shape
Out[240]: (11175,)

<小时/>

旧答案:

试试这个:

In [205]: r = pd.DataFrame(distance.squareform(distance.pdist(X, 'correlation')),
     ...:                  columns=X.index,
     ...:                  index=X.index)
     ...:

In [206]: r.loc["iris_5","iris_140"]
Out[206]: 0.41080564987753798

生成的 DF:

In [207]: r
Out[207]:
            iris_0    iris_1    iris_2    iris_3    iris_4    ...     iris_145  iris_146  iris_147  iris_148  iris_149
iris_0    0.000000  0.004001  0.000026  0.001832  0.000653    ...     0.353135  0.394002  0.346527  0.366083  0.366842
iris_1    0.004001  0.000000  0.003393  0.002603  0.007767    ...     0.294121  0.332886  0.291017  0.313743  0.315165
iris_2    0.000026  0.003393  0.000000  0.001667  0.000939    ...     0.348695  0.389447  0.342444  0.362369  0.363194
iris_3    0.001832  0.002603  0.001667  0.000000  0.003281    ...     0.313620  0.352149  0.305462  0.322263  0.322775
iris_4    0.000653  0.007767  0.000939  0.003281  0.000000    ...     0.374509  0.415817  0.365971  0.383464  0.383862
iris_5    0.000414  0.006408  0.000623  0.002167  0.000117    ...     0.362864  0.403644  0.354281  0.371513  0.371912
iris_6    0.001189  0.009279  0.001562  0.003861  0.000086    ...     0.380048  0.421352  0.370647  0.387134  0.387373
iris_7    0.000462  0.002882  0.000395  0.000454  0.001497    ...     0.333187  0.372898  0.325761  0.343900  0.344528
iris_8    0.001923  0.001454  0.001644  0.000167  0.003969    ...     0.307973  0.346558  0.301079  0.319367  0.320108
iris_9    0.003448  0.000967  0.003014  0.000693  0.006239    ...     0.292241  0.330160  0.285997  0.304751  0.305618
...            ...       ...       ...       ...       ...    ...          ...       ...       ...       ...       ...
iris_140  0.402175  0.342250  0.397769  0.358920  0.423142    ...     0.003630  0.000877  0.002683  0.006913  0.008383
iris_141  0.314419  0.257357  0.310069  0.277623  0.335490    ...     0.001735  0.006496  0.005755  0.019704  0.022182
iris_142  0.425351  0.367426  0.421202  0.379547  0.444834    ...     0.012273  0.006979  0.006358  0.003563  0.004110
iris_143  0.415332  0.357244  0.411146  0.370246  0.435053    ...     0.009473  0.005029  0.004562  0.003381  0.004123
iris_144  0.396952  0.338613  0.392700  0.353271  0.417104    ...     0.005162  0.002333  0.002158  0.004033  0.005143
iris_145  0.353135  0.294121  0.348695  0.313620  0.374509    ...     0.000000  0.001535  0.002642  0.013283  0.015462
iris_146  0.394002  0.332886  0.389447  0.352149  0.415817    ...     0.001535  0.000000  0.003431  0.011531  0.013519
iris_147  0.346527  0.291017  0.342444  0.305462  0.365971    ...     0.002642  0.003431  0.000000  0.004292  0.005540
iris_148  0.366083  0.313743  0.362369  0.322263  0.383464    ...     0.013283  0.011531  0.004292  0.000000  0.000084
iris_149  0.366842  0.315165  0.363194  0.322775  0.383862    ...     0.015462  0.013519  0.005540  0.000084  0.000000

[150 rows x 150 columns]