python - 线性回归预测预测输入数据(测试数据)，而不是测试结果

Question

我正在尝试预测以下内容:

• list( [ 收盘价(当日) - 开盘价(当日) ] )

使用以下内容作为输入:

• list( [ 开盘价(当天) - 收盘价(昨天) ] )

<小时/>

但是，我的 test_prediction 结果是对错误事物的预测。

来自 sklearn 和 statsmodels 线性回归模型的预测显示输入数据 (test_data) 和 预测之间有大约 100% 的相关性results，而预测结果应与test_result相关。

我做错了什么(或遗漏了什么)，如何解决？ 代码将生成 4 个图，显示不同列表之间的相关性。

<小时/>

###### Working usable example and code below ######

import numpy as np
from plotly.offline import plot
import plotly.graph_objs as go

from sklearn import linear_model
import statsmodels.api as sm

def xy_corr( x, y, fname ):
    trace1 = go.Scatter( x          = x, 
                         y          = y, 
                         mode       = 'markers',
                         marker     = dict( size  = 6,
                                            color = 'black',
                                            ),
                         showlegend = False
                         )
    layout = go.Layout( title  = fname )

    fig    = go.Figure( data   = [trace1],
                        layout = layout
                        )
    plot( fig, filename = fname + '.html' )

open_p  = [23215, 23659, 23770, 23659, 23659, 23993, 23987, 23935, 24380, 24271, 24314, 24018, 23928, 23240, 24193, 23708, 23525, 23640, 23494, 23333, 23451, 23395, 23395, 23925, 23936, 24036, 24008, 24248, 24249, 24599, 24683, 24708, 24510, 24483, 24570, 24946, 25008, 24880, 24478, 24421, 24630, 24540, 24823, 25090, 24610, 24866, 24578, 24686, 24465, 24225, 24526, 24645, 24780, 24538, 24895, 24921, 24743, 25163, 25163, 25316, 25320, 25158, 25375, 25430, 25466, 25231, 25103, 25138, 25138, 25496, 25502, 25610, 25625, 25810, 25789, 25533, 25785, 25698, 25373, 25558, 25594, 25026, 24630, 24509, 24535, 24205, 24465, 23847, 24165, 23840, 24216, 24355, 24158, 23203, 23285, 23423, 23786, 23729, 23944, 23637]
close_p = [23656, 23758, 23663, 23659, 23989, 23978, 24142, 24152, 24279, 24271, 24393, 23942, 23640, 24102, 23710, 23708, 23705, 23693, 23561, 23441, 23395, 23395, 23990, 23900, 24158, 24188, 24241, 24248, 24699, 24678, 24715, 24523, 24486, 24483, 24947, 24904, 24923, 24478, 24434, 24421, 24409, 24705, 25047, 24642, 24875, 24866, 24698, 24463, 24262, 24396, 24633, 24645, 24528, 24895, 24895, 24839, 25178, 25163, 25315, 25323, 25149, 25387, 25375, 25469, 25231, 25073, 25138, 25138, 25448, 25611, 25705, 25623, 25813, 25798, 25560, 25518, 25743, 25305, 25654, 25579, 25315, 24783, 24508, 24532, 24208, 24176, 24047, 24148, 24165, 24159, 24286, 24249, 23635, 23128, 23438, 23869, 23420, 23756, 23705, 24018]

open_prev_close_diff    = np.array( [ open_p[i] - close_p[i-1] for i in range( 1, len( open_p ) )] )[np.newaxis].T
open_current_close_diff = np.array( [close_p[i] -  open_p[i]   for i in range( 1, len( open_p ) )] )

train_data = open_prev_close_diff[  :80]
test_data  = open_prev_close_diff[80:]

train_result = open_current_close_diff[  :80]
test_result  = open_current_close_diff[80:]

regressor    = linear_model.LinearRegression()
regressor.fit( train_data, train_result )

test_prediction = np.array( [int(i) for i in regressor.predict( test_data )] )

xy_corr( [int(i) for i in test_result], test_prediction, 'known_result_and_prediction_result_sklearn')
xy_corr( [int(i) for i in test_data],   test_prediction, 'input_data_and_prediction_result_sklearn'  )

olsmod = sm.OLS( train_result, train_data )
olsres = olsmod.fit()

test_prediction = np.array( [int(i) for i in olsres.predict( test_data )] )

xy_corr( [int(i) for i in test_result], test_prediction, 'known_result_and_prediction_result_smOLS')
xy_corr( [int(i) for i in test_data],   test_prediction, 'input_data_and_prediction_result_smOLS'  )

Answer 1

_{With a hope no one would consider this impolite and/or harmfull,}

let me cite a lovely point of view on the underlying assumption from “Correlation does not imply Causation“: that many contemporary Quantitative Finance modellers neglect or abstract from:

对于任意两个相关事件，A 和 B，
以下关系是可能的:
A 导致 B； (直接因果关系)
B 导致 A； (反向因果关系)
A 和 B 是共同原因的结果，但互不导致；
A 导致 B 且 B 导致 A(双向或循环因果关系)；< br/>A 导致 C，而 C 又导致 B(间接因果关系)；
A 和 B 之间没有任何联系；相关性是 coincidence .

因此，仅从因果关系无法得出关于因果关系存在或方向的结论事实上A和B是相关的。

<小时/>

更新:

LinearRegressor() 除了一条线之外没有产生任何其他东西，这应该没什么奇怪的——也就是说，它的每个预测确实都在非常符合模特的线条。因此，对于这种预测器来说，一半的图显然是必须做的。
Q.E.D.

另一半没有显示任何其他内容，只是线性模型所采用的过度简化程度尽管是可观察到的现实。

当然，所经历的真实行为不是线性的(但不要责怪预测器“无法拟合”，它有责任制定一个 MSE - 最小化器驱动的线性模型，它无法在 DataSET 的训练部分找到任何更好的线性拟合，并且如果它是通过 进行训练的y = x^2 合成DataSET _{(对抛物线形状有先验知识)} ，再次它只会产生一个线性模型，在的训练部分分数上具有最小MSE DataSET 和我们都提前很确定，因此任何行都会产生完全有缺陷的预测OoS，但不是由于它无法很好地工作，但由于外部灌输的尝试在(故意二次)上下文中使用线性模型预测器的主要废话，它不遵循(已知的)现实)。

<小时/>

数据集:

作为一种基本的定量观点，比严格的Kolmogorov-Smirnov测试简单得多，对于未表达的假设，
检查 日内缺口中负差异的百分比(开盘[i] - 收盘[i-1]) [ 75% ]仅 100 个样本的浅 DataSET
相对于
当天蜡烛主体的负差(收盘价[i] - 开盘价[i]) 仅包含 100 个样本的相当浅的 DataSET 中的 [ 55% ]

无论如何，即使在设计得更好的预测模型上，训练少至 80 天的样本也很难很好地泛化，并且不仅应该关注更好的泛化能力，还应该关注避免季节性偏差等。

To have some idea where the ML goes into this field, my best preforming AI/ML-models have about 0k3 features ( many of 'em highly non-linear synthetic features ) and get intensively trained accross 30k+ DataSETs with carefully carving out their risk to overfitting and searching vast space of the learner-engines' hyperparameter StateSPACE

|
|>>> QuantFX.get_LDF_GDF_fromGivenRANGE( [ open_PRICE[i] - close_PRICE[i-1] for i in range( 1, len( close_PRICE ) ) ], nBINs_ = 31, aPrefixTEXT_ = "" )
     0:  ~ -432.00 LDF =      1 |____ 1.0 % _||____ 1 %
     1:  ~ -408.10 LDF =      1 |____ 1.0 % _||____ 2 %
     2:  ~ -384.19 LDF =      1 |____ 1.0 % _||____ 3 %
     3:  ~ -360.29 LDF =      0 |____ 0.0 % _||____ 3 %
     4:  ~ -336.39 LDF =      1 |____ 1.0 % _||____ 4 %
     5:  ~ -312.48 LDF =      1 |____ 1.0 % _||____ 5 %
     6:  ~ -288.58 LDF =      1 |____ 1.0 % _||____ 6 %
     7:  ~ -264.68 LDF =      0 |____ 0.0 % _||____ 6 %
     8:  ~ -240.77 LDF =      1 |____ 1.0 % _||____ 7 %
     9:  ~ -216.87 LDF =      3 |____ 3.0 % _||___ 10 %
    10:  ~ -192.97 LDF =      2 |____ 2.0 % _||___ 12 %
    11:  ~ -169.06 LDF =      1 |____ 1.0 % _||___ 13 %
    12:  ~ -145.16 LDF =      1 |____ 1.0 % _||___ 14 %
    13:  ~ -121.26 LDF =      2 |____ 2.0 % _||___ 16 %
    14:  ~  -97.35 LDF =      5 |____ 5.1 % _||___ 21 %
    15:  ~  -73.45 LDF =      3 |____ 3.0 % _||___ 24 %
    16:  ~  -49.55 LDF =      5 |____ 5.1 % _||___ 29 %
    17:  ~  -25.65 LDF =     18 |___ 18.2 % _||___ 47 %
    18:  ~   -1.74 LDF =     28 |___ 28.3 % _||___ 75 %
    19:  ~   22.16 LDF =      5 |____ 5.1 % _||___ 80 %
    20:  ~   46.06 LDF =      5 |____ 5.1 % _||___ 85 %
    21:  ~   69.97 LDF =      2 |____ 2.0 % _||___ 87 %
    22:  ~   93.87 LDF =      1 |____ 1.0 % _||___ 88 %
    23:  ~  117.77 LDF =      4 |____ 4.0 % _||___ 92 %
    24:  ~  141.68 LDF =      1 |____ 1.0 % _||___ 93 %
    25:  ~  165.58 LDF =      1 |____ 1.0 % _||___ 94 %
    26:  ~  189.48 LDF =      1 |____ 1.0 % _||___ 95 %
    27:  ~  213.39 LDF =      1 |____ 1.0 % _||___ 96 %
    28:  ~  237.29 LDF =      0 |____ 0.0 % _||___ 96 %
    29:  ~  261.19 LDF =      1 |____ 1.0 % _||___ 97 %
    30:  ~  285.10 LDF =      2 |____ 2.0 % _||__ 100 %
+0:00:06.234000
|
|
|>>> QuantFX.get_LDF_GDF_fromGivenRANGE( [ close_PRICE[i] - open_PRICE[i] for i in range( 1, len( close_PRICE ) ) ], nBINs_ = 31, aPrefixTEXT_ = "" )
     0:  ~ -523.00 LDF =      2 |____ 2.0 % _||____ 2 %
     1:  ~ -478.32 LDF =      1 |____ 1.0 % _||____ 3 %
     2:  ~ -433.65 LDF =      3 |____ 3.0 % _||____ 6 %
     3:  ~ -388.97 LDF =      1 |____ 1.0 % _||____ 7 %
     4:  ~ -344.29 LDF =      1 |____ 1.0 % _||____ 8 %
     5:  ~ -299.61 LDF =      2 |____ 2.0 % _||___ 10 %
     6:  ~ -254.94 LDF =      7 |____ 7.1 % _||___ 17 %
     7:  ~ -210.26 LDF =      3 |____ 3.0 % _||___ 20 %
     8:  ~ -165.58 LDF =      2 |____ 2.0 % _||___ 22 %
     9:  ~ -120.90 LDF =      5 |____ 5.1 % _||___ 27 %
    10:  ~  -76.23 LDF =      6 |____ 6.1 % _||___ 33 %
    11:  ~  -31.55 LDF =     22 |___ 22.2 % _||___ 55 %
    12:  ~   13.13 LDF =      7 |____ 7.1 % _||___ 62 %
    13:  ~   57.81 LDF =      5 |____ 5.1 % _||___ 67 %
    14:  ~  102.48 LDF =      4 |____ 4.0 % _||___ 71 %
    15:  ~  147.16 LDF =      8 |____ 8.1 % _||___ 79 %
    16:  ~  191.84 LDF =      6 |____ 6.1 % _||___ 85 %
    17:  ~  236.52 LDF =      2 |____ 2.0 % _||___ 87 %
    18:  ~  281.19 LDF =      3 |____ 3.0 % _||___ 90 %
    19:  ~  325.87 LDF =      2 |____ 2.0 % _||___ 92 %
    20:  ~  370.55 LDF =      2 |____ 2.0 % _||___ 94 %
    21:  ~  415.23 LDF =      3 |____ 3.0 % _||___ 97 %
    22:  ~  459.90 LDF =      0 |____ 0.0 % _||___ 97 %
    23:  ~  504.58 LDF =      0 |____ 0.0 % _||___ 97 %
    24:  ~  549.26 LDF =      0 |____ 0.0 % _||___ 97 %
    25:  ~  593.94 LDF =      1 |____ 1.0 % _||___ 98 %
    26:  ~  638.61 LDF =      0 |____ 0.0 % _||___ 98 %
    27:  ~  683.29 LDF =      0 |____ 0.0 % _||___ 98 %
    28:  ~  727.97 LDF =      0 |____ 0.0 % _||___ 98 %
    29:  ~  772.65 LDF =      0 |____ 0.0 % _||___ 98 %
    30:  ~  817.32 LDF =      1 |____ 1.0 % _||__ 100 %
+0:01:13.172000