python - 初始化高维稀疏矩阵

Question

我想使用 sklearn 初始化 300,000 x 300,0000 稀疏矩阵，但它需要内存，就好像它不是稀疏矩阵一样:

>>> from scipy import sparse
>>> sparse.rand(300000,300000,.1)   

它给出了错误:

MemoryError: Unable to allocate 671. GiB for an array with shape (300000, 300000) and data type float64

这与我使用 numpy 初始化的错误相同:

np.random.normal(size=[300000, 300000])

即使我的密度非常低，它也会重现错误:

>>> from scipy import sparse
>>> from scipy import sparse
>>> sparse.rand(300000,300000,.000000000001)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../python3.8/site-packages/scipy/sparse/construct.py", line 842, in rand
    return random(m, n, density, format, dtype, random_state)
  File ".../lib/python3.8/site-packages/scipy/sparse/construct.py", line 788, in random
    ind = random_state.choice(mn, size=k, replace=False)
  File "mtrand.pyx", line 980, in numpy.random.mtrand.RandomState.choice
  File "mtrand.pyx", line 4528, in numpy.random.mtrand.RandomState.permutation
MemoryError: Unable to allocate 671. GiB for an array with shape (90000000000,) and data type int64

是否有更节省内存的方法来创建这样的稀疏矩阵？

Answer 1

只生成你需要的。

from scipy import sparse
import numpy as np

n, m = 300000, 300000
density = 0.00000001
size = int(n * m * density)

rows = np.random.randint(0, n, size=size)
cols = np.random.randint(0, m, size=size)
data = np.random.rand(size)

arr = sparse.csr_matrix((data, (rows, cols)), shape=(n, m))

如果它们足够稀疏以适合内存，这使您可以构建怪物稀疏数组。

>>> arr
<300000x300000 sparse matrix of type '<class 'numpy.float64'>'
    with 900 stored elements in Compressed Sparse Row format>

无论如何，这可能就是 sparse.rand 构造函数的工作方式。如果任何行、col 对发生冲突，它会将数据值加在一起，这可能适用于我能想到的所有应用程序。