python - 如何跟踪使用 CPU 与 GPU 进行深度学习的时间？

Question

我想知道我的脚本运行时有多少时间花费在 CPU 和 GPU 上 - 有没有办法跟踪这个？

寻找通用答案，但如果对于这个玩具解决方案(来自 keras 的 multi_gpu_model 示例)来说太抽象了，那就太好了。

import tensorflow as tf
from keras.applications import Xception
from keras.utils import multi_gpu_model
import numpy as np
num_samples = 1000
height = 224
width = 224
num_classes = 1000
# Instantiate the base model (or "template" model).
# We recommend doing this with under a CPU device scope,
# so that the model's weights are hosted on CPU memory.
# Otherwise they may end up hosted on a GPU, which would
# complicate weight sharing.
with tf.device('/cpu:0'):
    model = Xception(weights=None,
                     input_shape=(height, width, 3),
                     classes=num_classes)
# Replicates the model on 8 GPUs.
# This assumes that your machine has 8 available GPUs.
parallel_model = multi_gpu_model(model, gpus=8)
parallel_model.compile(loss='categorical_crossentropy',
                       optimizer='rmsprop')
# Generate dummy data.
x = np.random.random((num_samples, height, width, 3))
y = np.random.random((num_samples, num_classes))
# This `fit` call will be distributed on 8 GPUs.
# Since the batch size is 256, each GPU will process 32 samples.
parallel_model.fit(x, y, epochs=20, batch_size=256)
# Save model via the template model (which shares the same weights):
model.save('my_model.h5')

Answer 1

您只需添加基于 Chrome 的 timeline 分析 CPU/GPU 从 Tensorflow API 到您的 Keras 模型!

这是 Tensorflow 问题跟踪器中提供的示例:

https://github.com/tensorflow/tensorflow/issues/9868#issuecomment-306188267

这是 Keras 问题跟踪器中一个更复杂的示例:

https://github.com/keras-team/keras/issues/6606#issuecomment-380196635

最后，这个分析的输出是这样的:

https://towardsdatascience.com/howto-profile-tensorflow-1a49fb18073d