gpt4 book ai didi

python - SGD - 损失在一些迭代后开始增加

转载 作者:行者123 更新时间:2023-11-28 17:31:29 25 4
gpt4 key购买 nike

我正在尝试实现具有两个约束的随机梯度下降,因此不能使用 scikit-learn。不幸的是,我已经在没有这两个限制的情况下与常规 SGD 作斗争。训练集上的损失(平方损失)在一些迭代中下降,但在一段时间后开始增加,如图所示。这些是我使用的功能:

def loss_prime_simple(w,node,feature,data):
x = data[3]
y = data[2]
x_f = x[node][feature]
y_node = y[node]
ret = (y_node - w[feature] * x_f) * (-x_f)
return ret

def update_weights(w,data,predecs,children,node, learning_rate):
len_features = len(data[3][0])
w_new = np.zeros(len_features)
for feature_ in range(len_features):
w_new[feature_] = loss_prime_simple(w,node,feature_,data)
return w - learning_rate * w_new

def loss_simple(w,data):
y_p = data[2]
x = data[3]
return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

这显示了具有两个不同学习率 (0.001, 0.0001) 的训练集的损失 http://postimg.org/image/43nbmh8x5/

任何人都可以找到错误或有如何调试的建议吗?谢谢

编辑:

正如 lejlot 指出的那样,拥有数据会很好。这是我用于 x 的数据(单个样本):http://textuploader.com/5x0f1

y=2

这会导致损失:http://postimg.org/image/o9d97kt9v/

更新后的代码:

def loss_prime_simple(w,node,feature,data):
x = data[3]
y = data[2]
x_f = x[node][feature]
y_node = y[node]
return -(y_node - w[feature] * x_f) * x_f

def update_weights(w,data,predecs,children,node, learning_rate):
len_features = len(data[3][0])
w_new = np.zeros(len_features)
for feature_ in range(len_features):
w_new[feature_] = loss_prime_simple(w,node,feature_,data)
return w - learning_rate * w_new

def loss_simple2(w,data):
y_p = data[2]
x = data[3]
return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

import numpy as np
X = [#put array from http://textuploader.com/5x0f1 here]
y = [2]

data = None, None, y, X

w = np.random.rand(4096)

a = [ loss_simple2(w, data) ]

for _ in range(200):
for j in range(X.shape[0]):
w = update_weights(w,data,None,None,j, 0.0001)
a.append( loss_simple2(w, data) )

from matplotlib import pyplot as plt
plt.figure()
plt.plot(a)
plt.show()

最佳答案

可以注意到的主要错误是您reshape 而不是transpose,比较:

>>> import numpy as np
>>> X = np.array(range(10)).reshape(2,-1)
>>> X
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
>>> X.reshape(-1, 2)
array([[0, 1],
[2, 3],
[4, 5],
[6, 7],
[8, 9]])
>>> X.T
array([[0, 5],
[1, 6],
[2, 7],
[3, 8],
[4, 9]])
>>> X.reshape(-1, 2) == X.T
array([[ True, False],
[False, False],
[False, False],
[False, False],
[False, True]], dtype=bool)

接下来看起来不好的事情是调用 sum( array ),你应该调用 array.sum()

>>> import numpy as np
>>> x = np.array(range(10)).reshape(2, 5)
>>> x
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
>>> sum(x)
array([ 5, 7, 9, 11, 13])
>>> x.sum()
45

在这之后,它工作得很好

def loss_prime_simple(w,node,feature,data):
x = data[3]
y = data[2]
x_f = x[node][feature]
y_node = y[node]
ret = w[feature]
return -(y_node - w[feature] * x_f) * x_f

def update_weights(w,data,predecs,children,node, learning_rate):
len_features = len(data[3][0])
w_new = np.zeros(len_features)
for feature_ in range(len_features):
w_new[feature_] = loss_prime_simple(w,node,feature_,data)
return w - learning_rate * w_new

def loss_simple(w,data):
y_p = data[2]
x = data[3]
return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

import numpy as np

X = np.random.randn(1000, 3)
y = np.random.randn(1000)

data = None, None, y, X

w = np.array([1,3,3])

loss = [loss_simple(w, data)]

for _ in range(20):
for j in range(X.shape[0]):
w = update_weights(w, data, None, None, j, 0.001)
loss.append(loss_simple(w, data))

from matplotlib import pyplot as plt
plt.figure()
plt.plot(loss)
plt.show()

enter image description here

关于python - SGD - 损失在一些迭代后开始增加,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34004088/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com