tensorflow - 深度学习中的融合核(或融合层)是什么？

Question

我正在阅读 Apex AMP documentation :

A Python-only build omits:

• Fused kernels required to use apex.optimizers.FusedAdam.
• Fused kernels required to use apex.normalization.FusedLayerNorm.
• Fused kernels that improve the performance and numerical stability of apex.parallel.SyncBatchNorm.
• Fused kernels that improve the performance of apex.parallel.DistributedDataParallel and apex.amp. DistributedDataParallel, amp, and SyncBatchNorm will still be usable, but they may be slower.

似乎还有一个 "FusedAdam" optimizer :

The Adam optimizer in Pytorch (like all Pytorch optimizers) carries out optimizer.step() by looping over parameters, and launching a series of kernels for each parameter. This can require hundreds of small launches that are mostly bound by CPU-side Python looping and kernel launch overhead, resulting in poor device utilization. Currently, the FusedAdam implementation in Apex flattens the parameters for the optimization step, then carries out the optimization step itself via a fused kernel that combines all the Adam operations. In this way, the loop over parameters as well as the internal series of Adam operations for each parameter are fused such that optimizer.step() requires only a few kernel launches.

The current implementation (in Apex master) is brittle and only works with Amp opt_level O2. I’ve got a WIP branch to make it work for any opt_level (https://github.com/NVIDIA/apex/pull/351). I recommend waiting until this is merged then trying it.

这部分地解释了它。我还有更多问题:

内核是什么意思？层还是优化器？

融合层的想法与融合优化器相同吗？

Answer 1

这里的“内核”用于计算内核:https://en.wikipedia.org/wiki/Compute_kernel
像卷积这样的操作通常使用计算内核来实现，以提高效率。计算内核可以使用 C、CUDA、OpenCL 甚至汇编来编写，以实现最高效率。因此，“仅 Python 构建”不支持...并不令人惊讶...

“融合”意味着计算步骤的通用化。基本上，这是通过在单个硬件(GPU、CPU 或 TPU)操作中组合类似操作来更有效地运行代码的实现技巧。因此，“融合层”是操作受益于“融合”实现的层。