python - 为什么我不能在千层面回归模型的最后一层使用 dropout？

Question

玩具回归示例。使用 dropout=0.0 这很好用并且成本降低了。使用 dropout=0.5 我得到错误:

ValueError: Got num_leading_axes=1 for a 1-dimensional input, leaving no trailing axes for the dot product.

有什么想法吗？

import theano
import theano.tensor as T
import lasagne
import numpy as np

num_features=10
N=1000

# Set up the network                                                                                                                                                                                                                       
x=T.fmatrix('x')
y=T.fvector('y')

dropout=0.5
network = lasagne.layers.InputLayer(shape=(None, num_features), input_var=x)
if dropout > 0.0:
    network = lasagne.layers.dropout(network, p=dropout),
network = lasagne.layers.DenseLayer( network, num_units=1, nonlinearity=None )

pred = lasagne.layers.get_output(network)

cost = lasagne.objectives.squared_error(pred, y).mean()

# Compile training function                                                                                                                                                                                                                
params = lasagne.layers.get_all_params(network, trainable=True)
train_fn = theano.function([x, y], cost, updates=lasagne.updates.adamax(cost, params) )

# Generate some synthetic data                                                                                                                                                                                                             
X=np.random.randn( N,num_features ).astype( theano.config.floatX )
b=np.random.randn( num_features ).astype( theano.config.floatX )
Y=np.dot( X, b ) + np.random.randn( N ).astype(theano.config.floatX ) * 0.1

# Train for 100 iterations                                                                                                                                                                                                                 
for i in range(100):
    print train_fn(X,Y)

Answer 1

删除 dropout 层后面的逗号。该代码将在 InputLayer 或 DenseLayer 之后立即处理 dropout。逗号创建一个元组，其中包含导致错误的网络变量 (network, )。

network = lasagne.layers.InputLayer(shape=(None, num_features), input_var=x)
if dropout > 0.0:
    network = lasagne.layers.dropout(network, p=dropout),
network = lasagne.layers.DenseLayer( network, num_units=1, nonlinearity=None )