I have two data arrays, xdata of shape (40,) and ydata of shape (40, 721, 1440) (time, lat, lon). My final goal is to compute the regression slope between these two datasets and obtain the errors from the slope to show the error distribution, where x is average sea surface temperatures for 40 years and y is an atmospheric variable. I have done this using one method where I calculated the covariance and then took the square root of this array, which if I understand correct, leaves me with the standard error. To verify this approach, I found the scipy.stat.linregress function and wanted to use this as it returns the standard error while calculating the slope. Although, I am running into errors when coding this.
I有两个数据数组,形状为(40,)扩展数据和形状为(40,721,1440)(时间,经度)的ydata。我的最终目标是计算这两个数据集之间的回归斜率,并从斜率中获得误差以显示误差分布,其中x是40年来的平均海洋表面温度,y是大气变量。我使用的是一种方法,我计算协方差,然后取这个数组的平方根,如果我理解正确,就会得到标准误差。为了验证这种方法,我找到了scipy.stat.linregress函数,并希望使用它,因为它在计算斜率时返回标准误差。不过,我在编写代码时遇到了错误。
To get my arrays the same size:
让我的数组大小相同:
Y = ydata.stack(allpoints = ['lat','lon'])
X = xdata.values[:, None] * np.ones(Y.shape)
Here, I have stacked my ydata into a 2D array rather than a 3D array, leaving with me with the dimensions of (40, 1038240). Then I create a temporary array to of the xdata, with an extra dimensions filled with ones to get the two arrays of the same shape. After this I pass it through the function:
在这里,我将ydata堆叠到一个2D数组中,而不是3D数组中,剩下的维度是(40,1038240)。然后,我为扩展数据创建了一个临时数组To,用一个额外的维度填充1,以获得形状相同的两个数组。在此之后,我将其传递给函数:
test = stats.linregress(X,Y)
And I am left with a value error saying:
我留下了一个值错误,写道:
ValueError: too many values to unpack (expected 4)
UPDATE:
最新情况:
I was able to get the package I was interested in using working:
我能够获得我感兴趣的使用Working的包:
from scipy import stats
ny = 721
nx = 1440
n = nx * ny
cape_2d = cape_ds.values.reshape(40,n)
reg_results = np.empty((5,n))
for i in range(n):
reg_results[:,i] = stats.linregress(b,cape_2d[:,i])
slope, intercept, r_val, p_val, std_err = reg_results.reshape((5,ny,nx))
err_shape = std_err.reshape(721*1440)
plt.plot(err_shape)
At this point, I think I may have the correct answer, but Im worried that my standard errors are too high...Im not sure what to for it to look like, I was just hoping to get a normal distribution.
在这一点上,我想我可能有正确的答案,但我担心我的标准误差太高了…我不确定它看起来是什么样子,我只是希望得到一个正态分布。
更多回答
Won't this regression produce the same value for every Y point at the same time-step? Seems like you could simplify the problem by taking the mean of Y across all latitude and longitudes, giving you a Y array of shape (40,).
这种回归不是会在相同的时间步长为每个Y点产生相同的值吗?似乎你可以通过取所有纬度和经度的Y的平均值来简化问题,得到一个形状为(40,)的Y数组。
Im not sure? I dont think so...I have successfully completed the regression part of this problem using another method, but I am now just trying to see my error distribution and my initial method did not calculate this directly so I tried to take the sqrt(cov) and to check this, I wanted to use this package/function for the standard error.
我不确定?我不这么认为……我已经用另一种方法成功地完成了这个问题的回归部分,但我现在只是试图查看我的错误分布,而我最初的方法没有直接计算它,所以我尝试使用SQRT(Cov)来检查这一点,我想使用这个包/函数来处理标准错误。
If you have an already-working way of doing this regression, you could run your linear regression, get Y_pred, and calculate Y - Y_pred
. You could then flatten that, and plot it using a histogram. That would show you the magnitude of the residuals, as well as what distribution it has.
如果您已经有了执行此回归的有效方法,则可以运行您的线性回归,获得Y_pred,并计算Y-Y_pred。然后,您可以将其展平,并使用直方图绘制它。这将显示残差的大小,以及它的分布情况。
I would get Y_pred by y_pred = slope*X + intercept, right? And then just subtract from the Y data array I input into the regression function? I have done this but the histogram does not seem like what I would expect it to be...
我会通过y_pred=斜率*X+截距得到Y_pred,对吗?然后从我输入到回归函数的Y数据数组中减去?我已经这样做了,但柱状图似乎不像我期望的那样……
Most linear regression packages give some .fit() or .predict() method for getting predictions from X values. Are you using a linear regression package, or did you write your own code for fitting it?
大多数线性回归包都提供了一些.fit()或.recast()方法,用于从X值获得预测。您使用的是线性回归包,还是您自己编写的代码来拟合它?
我是一名优秀的程序员,十分优秀!