python - SKLearn 感知器对于稀疏和密集的行为不同

Question

感知器在给出密集格式的矩阵时，与给出稀疏格式的相同矩阵相比，会给出不同的结果。我认为这可能是一个洗牌问题，所以我使用 cross_validate 运行交叉验证来自sklearn.model_selection但没有运气。

讨论了类似的问题here 。但给出了一些理由。这里面有什么道理吗？

仅供引用，我使用感知器的参数是: penalty='l2', alpha=0.0001, fit_intercept=True, max_iter=10000, tol=1e-8, shuffle=True, verbose=0, eta0=1.0, n_jobs=1, random_state=0, class_weight=None, warm_start=False, n_iter=None

我正在使用sparse.csr_matrix将密集矩阵转换为稀疏矩阵作为可接受的答案 here

Answer 1

这里有一个基本原理。

感知器 shares大部分代码带有 SGDClassifier

Perceptron and SGDClassifier share the same underlying implementation. In fact, Perceptron() is equivalent to SGDClassifier(loss=”perceptron”, eta0=1, learning_rate=”constant”, penalty=None).

和SGDClassifier是better documented :

Note: The sparse implementation produces slightly different results than the dense implementation due to a shrunk learning rate for the intercept.

我们有更多详细信息latter :

In the case of sparse feature vectors, the intercept is updated with a smaller learning rate (multiplied by 0.01) to account for the fact that it is updated more frequently.

请注意，此实现细节来自Leon Bottou :

The learning rate for the bias is multiplied by 0.01 because this frequently improves the condition number.

为了完整起见，在scikit-learn code中:

SPARSE_INTERCEPT_DECAY = 0.01
# For sparse data intercept updates are scaled by this decay factor to avoid
# intercept oscillation.

奖金示例:

import numpy as np
import scipy.sparse as sp
from sklearn.linear_model import Perceptron

np.random.seed(42)
n_samples, n_features = 1000, 10
X_dense = np.random.randn(n_samples, n_features)
X_csr = sp.csr_matrix(X_dense)
y = np.random.randint(2, size=n_samples)

for X in [X_dense, X_csr]:
    model = Perceptron(penalty='l2', alpha=0.0001, fit_intercept=True,
                       max_iter=10000, tol=1e-8, shuffle=True, verbose=0,
                       eta0=1.0, n_jobs=1, random_state=0, class_weight=None,
                       warm_start=False, n_iter=None)
    model.fit(X, y)
    print(model.coef_)

您可以检查系数是否不同。将 fit_intercept 更改为 False 会使系数相等，但拟合可能会较差。