c# - C# 中的神经网络 - NaN 和无穷大

Question

也许这里有人可以帮助我。我有点卡住了。现在，我正在尝试用 C# 编写自己的神经网络。我让它工作了一些(它与 XOR 一起工作)。它是一个简单的神经网络，具有输入、隐藏和输出，我使用 ReLU 作为我的激活函数。我的问题是，当我将隐藏层的数量增加到大于 ~16 时，我往往会得到一些 NaN 或无穷大，这很快就会把一切搞乱。我尝试降低学习率，但这没有帮助。我认为问题出在我的 SGD 函数中，但我无法真正找到它，特别是因为它适用于较少的层。

这是函数:

private void SGD(double learningRate, double[] weightedSumHidden, double[] errors_output)
    {
        /*---------------------------------------------------------------
        * -- Calculate Delta of the weight between hidden and output --
        ---------------------------------------------------------------*/
        var HiddenTransposed = Hidden.Transpose();
        var deltaWeightOutput = HiddenTransposed.Dot(errors_output);
        double[,] deltaWeightOutput2D = Matrix.Create(deltaWeightOutput); //Convert to Matrix
        WeightsHiddenOutput = WeightsHiddenOutput.Add(deltaWeightOutput2D.Multiply(learningRate));

        /*---------------------------------------------------------------
         * -- Calculate Delta of the weight between input and hidden --
         ---------------------------------------------------------------*/
        //First we have to calculate the Error in the hidden nodes ...
        //Transposed because we are going Backwards through the Network
        var WHOTransposed = WeightsHiddenOutput.Transpose();
        //Moves the Error to the output layer
        var errors_hidden = WHOTransposed.Dot(errors_output);
        //Element Wise multiplication (schur product)
        weightedSumHidden = ApplyDerivativeReLU(weightedSumHidden);
        //Moves the Error backthrough the Neuron
        errors_hidden = errors_hidden.Multiply(weightedSumHidden);

        //... then we can Calculate the Delta
        var InputTransposed = Inputs.Transpose();
        var deltaWeightHidden = InputTransposed.Dot(errors_hidden);
        double[,] deltaWeightHidden2D = Matrix.Create(deltaWeightHidden); //Convert to Matrix
        deltaWeightHidden2D = Inputs.Transpose().Dot(deltaWeightHidden2D);

        /*---------------------------------------------------------------
         * --        Adjust Weights and Biases using the delta         --
         ---------------------------------------------------------------*/
        //The Biases just get adjusted by adding the Errors multiplied by the learning rate
        BiasOutput = BiasOutput.Add(errors_output.Multiply(learningRate)); //Output Bias
        BiasHidden = BiasHidden.Add(errors_hidden.Multiply(learningRate)); //Hidden Bias

        WeightsInputHidden = WeightsInputHidden.Add(deltaWeightHidden2D.Multiply(learningRate));           
    }

如果有人能在这方面帮助我，我将非常感激，我已经在这方面坚持了好几天了。我使用本指南(http://neuralnetworksanddeeplearning.com/chap2.html)作为我的代码的基础。另外，我使用 Accord.Math 进行矩阵数学计算。

谢谢!

Answer 1

您可以使用这些断点来检查错误开始的位置:

if (double.IsNan(value))
if (double.IsInfinity(value))
if (float.IsNan(value))
if (float.IsInfinity(value))

我遇到了同样的问题(NaN)，异常帮助我找到了问题:

if (double.IsNan(value) || double.IsIninity(value)) throw new Exception();

Visual Studio 的调试工具很有帮助 - 您可以使用断点来检查对象中的值。