- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我试图建立一个序列到序列模型,以根据前几个输入预测传感器信号随时间的变化(见下图)
该模型工作正常,但我想“增加趣味”并尝试在两个 LSTM 层之间添加一个注意力层。
型号代码:
def train_model(x_train, y_train, n_units=32, n_steps=20, epochs=200,
n_steps_out=1):
filters = 250
kernel_size = 3
logdir = os.path.join(logs_base_dir, datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = TensorBoard(log_dir=logdir, update_freq=1)
# get number of features from input data
n_features = x_train.shape[2]
# setup network
# (feel free to use other combination of layers and parameters here)
model = keras.models.Sequential()
model.add(keras.layers.LSTM(n_units, activation='relu',
return_sequences=True,
input_shape=(n_steps, n_features)))
model.add(keras.layers.LSTM(n_units, activation='relu'))
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse', metrics=['mse'])
# train network
history = model.fit(x_train, y_train, epochs=epochs,
validation_split=0.1, verbose=1, callbacks=[tensorboard_callback])
return model, history
fit_generator
拟合我的数据时 layers = [35, 35] # Number of hidden neuros in each layer of the encoder and decoder
learning_rate = 0.01
decay = 0 # Learning rate decay
optimiser = keras.optimizers.Adam(lr=learning_rate, decay=decay) # Other possible optimiser "sgd" (Stochastic Gradient Descent)
num_input_features = train_x.shape[2] # The dimensionality of the input at each time step. In this case a 1D signal.
num_output_features = 1 # The dimensionality of the output at each time step. In this case a 1D signal.
# There is no reason for the input sequence to be of same dimension as the ouput sequence.
# For instance, using 3 input signals: consumer confidence, inflation and house prices to predict the future house prices.
loss = "mse" # Other loss functions are possible, see Keras documentation.
# Regularisation isn't really needed for this application
lambda_regulariser = 0.000001 # Will not be used if regulariser is None
regulariser = None # Possible regulariser: keras.regularizers.l2(lambda_regulariser)
batch_size = 128
steps_per_epoch = 200 # batch_size * steps_per_epoch = total number of training examples
epochs = 100
input_sequence_length = n_steps # Length of the sequence used by the encoder
target_sequence_length = 31 - n_steps # Length of the sequence predicted by the decoder
num_steps_to_predict = 20 # Length to use when testing the model
# Define an input sequence.
encoder_inputs = keras.layers.Input(shape=(None, num_input_features), name='encoder_input')
# Create a list of RNN Cells, these are then concatenated into a single layer
# with the RNN layer.
encoder_cells = []
for hidden_neurons in layers:
encoder_cells.append(keras.layers.GRUCell(hidden_neurons,
kernel_regularizer=regulariser,
recurrent_regularizer=regulariser,
bias_regularizer=regulariser))
encoder = keras.layers.RNN(encoder_cells, return_state=True, name='encoder_layer')
encoder_outputs_and_states = encoder(encoder_inputs)
# Discard encoder outputs and only keep the states.
# The outputs are of no interest to us, the encoder's
# job is to create a state describing the input sequence.
encoder_states = encoder_outputs_and_states[1:]
# The decoder input will be set to zero (see random_sine function of the utils module).
# Do not worry about the input size being 1, I will explain that in the next cell.
decoder_inputs = keras.layers.Input(shape=(None, 20), name='decoder_input')
decoder_cells = []
for hidden_neurons in layers:
decoder_cells.append(keras.layers.GRUCell(hidden_neurons,
kernel_regularizer=regulariser,
recurrent_regularizer=regulariser,
bias_regularizer=regulariser))
decoder = keras.layers.RNN(decoder_cells, return_sequences=True, return_state=True, name='decoder_layer')
# Set the initial state of the decoder to be the ouput state of the encoder.
# This is the fundamental part of the encoder-decoder.
decoder_outputs_and_states = decoder(decoder_inputs, initial_state=encoder_states)
# Only select the output of the decoder (not the states)
decoder_outputs = decoder_outputs_and_states[0]
# Apply a dense layer with linear activation to set output to correct dimension
# and scale (tanh is default activation for GRU in Keras, our output sine function can be larger then 1)
decoder_dense = keras.layers.Dense(num_output_features,
activation='linear',
kernel_regularizer=regulariser,
bias_regularizer=regulariser)
decoder_outputs = decoder_dense(decoder_outputs)
model = keras.models.Model(inputs=[encoder_inputs, decoder_inputs],
outputs=decoder_outputs)
model.compile(optimizer=optimiser, loss=loss)
model.summary()
Layer (type) Output Shape Param # Connected to
==================================================================================================
encoder_input (InputLayer) (None, None, 20) 0
__________________________________________________________________________________________________
decoder_input (InputLayer) (None, None, 20) 0
__________________________________________________________________________________________________
encoder_layer (RNN) [(None, 35), (None, 13335 encoder_input[0][0]
__________________________________________________________________________________________________
decoder_layer (RNN) [(None, None, 35), ( 13335 decoder_input[0][0]
encoder_layer[0][1]
encoder_layer[0][2]
__________________________________________________________________________________________________
dense_5 (Dense) (None, None, 1) 36 decoder_layer[0][0]
==================================================================================================
Total params: 26,706
Trainable params: 26,706
Non-trainable params: 0
__________________________________________________________________________________________________
history = model.fit([train_x, decoder_inputs],train_y, epochs=epochs,
validation_split=0.3, verbose=1)
When feeding symbolic tensors to a model, we expect the tensors to have a static batch size. Got tensor with shape: (None, None, 20)
最佳答案
Keras 中的注意力层不是可训练层(除非我们使用 scale 参数)。它只计算矩阵运算。在我看来,如果直接应用于时间序列,这一层可能会导致一些错误,但让我们继续按顺序进行……
在我们的时间序列问题上复制注意力机制的最自然选择是采用提出的解决方案 here并再次解释here .这是注意力在 NLP 中 enc-dec 结构中的经典应用
在 TF 实现之后,对于我们的注意力层,我们需要 3d 格式的查询、值、键张量。我们直接从循环层获得这些值。更具体地说,我们利用序列输出和隐藏状态。这些就是我们构建注意力机制所需的全部内容。
查询是输出序列[batch_dim, time_step, features]
值是隐藏状态 [batch_dim, features],其中我们为矩阵操作添加了时间维度 [batch_dim, 1, features]
作为键,我们像以前一样使用隐藏状态所以键=值
在上面的定义和实现中我发现了两个问题:
def attention_keras(query_value):
query, value = query_value # key == value
score = tf.matmul(query, value, transpose_b=True) # (batch, timestamp, 1)
score = tf.nn.softmax(score) # softmax on -1 axis ==> score always = 1 !!!
print((score.numpy()!=1).any()) # False ==> score always = 1 !!!
score = tf.matmul(score, value) # (batch, timestamp, feat)
return score
np.random.seed(33)
time_steps = 20
features = 50
sample = 5
X = np.random.uniform(0,5, (sample,time_steps,features))
state = np.random.uniform(0,5, (sample,features))
attention_keras([X,tf.expand_dims(state,1)]) # ==> the same as Attention(dtype='float64')([X,tf.expand_dims(state,1)])
def attention_seq(query_value, scale):
query, value = query_value
score = tf.matmul(query, value, transpose_b=True) # (batch, timestamp, 1)
score = scale*score # scale with a fixed number (it can be finetuned or learned during train)
score = tf.nn.softmax(score, axis=1) # softmax on timestamp axis
score = score*query # (batch, timestamp, feat)
return score
np.random.seed(33)
time_steps = 20
features = 50
sample = 5
X = np.random.uniform(0,5, (sample,time_steps,features))
state = np.random.uniform(0,5, (sample,features))
attention_seq([X,tf.expand_dims(state,1)], scale=0.05)
######### KERAS #########
inp = Input((time_steps,features))
seq, state = GRU(32, return_state=True, return_sequences=True)(inp)
att = Attention()([seq, tf.expand_dims(state,1)])
######### CUSTOM #########
inp = Input((time_steps,features))
seq, state = GRU(32, return_state=True, return_sequences=True)(inp)
att = Lambda(attention_seq, arguments={'scale': 0.05})([seq, tf.expand_dims(state,1)])
######### KERAS #########
inp = Input((n_features, n_steps))
seq, state = GRU(n_units, activation='relu',
return_state=True, return_sequences=True)(inp)
att = Attention()([seq, tf.expand_dims(state,1)])
x = GRU(n_units, activation='relu')(att)
x = Dense(64, activation='relu')(x)
x = Dropout(0.5)(x)
out = Dense(n_steps_out)(x)
model = Model(inp, out)
model.compile(optimizer='adam', loss='mse', metrics=['mse'])
model.summary()
######### CUSTOM #########
inp = Input((n_features, n_steps))
seq, state = GRU(n_units, activation='relu',
return_state=True, return_sequences=True)(inp)
att = Lambda(attention_seq, arguments={'scale': 0.05})([seq, tf.expand_dims(state,1)])
x = GRU(n_units, activation='relu')(att)
x = Dense(64, activation='relu')(x)
x = Dropout(0.5)(x)
out = Dense(n_steps_out)(x)
model = Model(inp, out)
model.compile(optimizer='adam', loss='mse', metrics=['mse'])
model.summary()
关于tensorflow - 序列到序列 - 用于时间序列预测,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/61757475/
我想将模型及其各自训练的权重从 tensorflow.js 转换为标准 tensorflow,但无法弄清楚如何做到这一点,tensorflow.js 的文档对此没有任何说明 我有一个 manifest
我有一个运行良好的 TF 模型,它是用 Python 和 TFlearn 构建的。有没有办法在另一个系统上运行这个模型而不安装 Tensorflow?它已经经过预训练,所以我只需要通过它运行数据。 我
当执行 tensorflow_model_server 二进制文件时,它需要一个模型名称命令行参数,model_name。 如何在训练期间指定模型名称,以便在运行 tensorflow_model_s
我一直在 R 中使用标准包进行生存分析。我知道如何在 TensorFlow 中处理分类问题,例如逻辑回归,但我很难将其映射到生存分析问题。在某种程度上,您有两个输出向量而不是一个输出向量(time_t
Torch7 has a library for generating Gaussian Kernels在一个固定的支持。 Tensorflow 中有什么可比的吗?我看到 these distribu
在Keras中我们可以简单的添加回调,如下所示: self.model.fit(X_train,y_train,callbacks=[Custom_callback]) 回调在doc中定义,但我找不到
我正在寻找一种在 tensorflow 中有条件打印节点的方法,使用下面的示例代码行,其中每 10 个循环计数,它应该在控制台中打印一些东西。但这对我不起作用。谁能建议? 谢谢,哈米德雷萨, epsi
我想使用 tensorflow object detection API 创建我自己的 .tfrecord 文件,并将它们用于训练。该记录将是原始数据集的子集,因此模型将仅检测特定类别。我不明白也无法
我在 TensorFlow 中训练了一个聊天机器人,想保存模型以便使用 TensorFlow.js 将其部署到 Web。我有以下内容 checkpoint = "./chatbot_weights.c
我最近开始学习 Tensorflow,特别是我想使用卷积神经网络进行图像分类。我一直在看官方仓库中的android demo,特别是这个例子:https://github.com/tensorflow
我目前正在研究单图像超分辨率,并且我设法卡住了现有的检查点文件并将其转换为 tensorflow lite。但是,使用 .tflite 文件执行推理时,对一张图像进行上采样所需的时间至少是使用 .ck
我注意到 tensorflow 的 api 中已经有批量标准化函数。我不明白的一件事是如何更改训练和测试之间的程序? 批量归一化在测试和训练期间的作用不同。具体来说,在训练期间使用固定的均值和方差。
我创建了一个模型,该模型将 Mobilenet V2 应用于 Google colab 中的卷积基础层。然后我使用这个命令转换它: path_to_h5 = working_dir + '/Tenso
代码取自:- http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ import tensorflow as tf fr
好了,所以我准备在Tensorflow中运行 tf.nn.softmax_cross_entropy_with_logits() 函数。 据我了解,“logit”应该是概率的张量,每个对应于某个像素的
tensorflow 服务构建依赖于大型 tensorflow ;但我已经成功构建了 tensorflow。所以我想用它。我做这些事情:我更改了 tensorflow 服务 WORKSPACE(org
Tensoflow 嵌入层 ( https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding ) 易于使用, 并且有大量的文
我正在尝试使用非常大的数据集(比我的内存大得多)训练 Tensorflow 模型。 为了充分利用所有可用的训练数据,我正在考虑将它们分成几个小的“分片”,并一次在一个分片上进行训练。 经过一番研究,我
根据 Sutton 的书 - Reinforcement Learning: An Introduction,网络权重的更新方程为: 其中 et 是资格轨迹。 这类似于带有额外 et 的梯度下降更新。
如何根据条件选择执行图表的一部分? 我的网络有一部分只有在 feed_dict 中提供占位符值时才会执行.如果未提供该值,则采用备用路径。我该如何使用 tensorflow 来实现它? 以下是我的代码
我是一名优秀的程序员,十分优秀!