python - 改善简单的1层神经网络

Question

我创建了自己的非常简单的1层神经网络，专门研究二进制分类问题。输入数据点乘以权重并加上一个偏差。将整个内容相加（加权和）并通过激活函数（例如relu或sigmoid）进行输入。那将是预测输出。不涉及其他任何层（即隐藏层）。

仅出于我自己对数学方面的理解，我不想使用现有的库/程序包（例如Keras，PyTorch，Scikit-learn ..etc），而只是想使用简单的python代码创建神经网络。该模型是在方法（simple_1_layer_classification_NN）中创建的，该方法采用必要的参数进行预测。但是，我遇到了一些问题，因此在下面列出了一些问题以及我的代码。

P.s.对于包含这么多代码，我真的很抱歉，但是我不知道如何在不参考相关代码的情况下提出问题。

问题：

1-当我通过一些训练数据集来训练网络时，我发现最终平均准确度会随着不同数量的时期而完全不同，而对于某种最佳数量的时期绝对没有明确的规律。我将其他参数保持不变：learning rate = 0.5，activation = sigmoid（因为它是1层-既是输入层又是输出层。不涉及任何隐藏层。我读过sigmoid比relu更适合输出层>），cost function = squared error。以下是不同时期的结果：

纪元= 100,000。
平均准确度：50.10541638874056

时期= 500,000。
平均准确度：50.08965597645948

纪元= 1,000,000。
平均准确度：97.56879179064482

时代= 7,500,000。
平均准确度：49.994692515332524

时代750,000。
平均准确度：77.0028368954157

纪元= 100。
平均准确度：48.96967591507596

纪元= 500。
平均准确度：48.20721972881673

纪元= 10,000。
平均准确度：71.58066454336122

纪元= 50,000
平均准确度：62.52998222597177

纪元= 100,000。
平均准确度：49.813675726563424

纪元= 1,000,000。
平均准确度：49.993141329926374

如您所见，似乎没有明确的模式。我尝试了100万个时代，并获得了97.6％的准确性。然后，我尝试了750万个时代，获得了50％的准确性。五百万个纪元也获得了50％的准确性。 100个纪元导致49％的准确性。然后是真正奇怪的一个，再次尝试了100万个时代，并获得了50％。

因此，我在下面共享我的代码，因为我不相信网络在做任何学习。似乎只是随机猜测。我应用了反向传播和偏导数的概念来优化权重和偏差。所以我不确定我的代码在哪里出问题。

2- simple_1_layer_classification_NN参数是我包含在input_dimension方法的参数列表中的参数之一。起初，我认为需要锻炼输入层所需的权数。然后我意识到，只要将dataset_input_matrix（特征矩阵）参数传递给该方法，我就可以访问矩阵的随机索引以访问来自矩阵（input_observation_vector = dataset_input_matrix[ri]）的随机观察向量。然后遍历观察以访问每个功能。观察向量的循环数（或长度）将准确地告诉我需要多少个权重（因为每个特征将需要一个权重（作为其系数）。因此(len(input_observation_vector))会告诉我输入中所需的权重数）层，因此我不需要让用户将input_dimension参数传递给该方法。
所以我的问题很简单，当可以简单地通过评估输入矩阵中观察向量的长度来计算出input_dimension参数时，是否有任何必要/理由？

3-当我尝试绘制costs值的数组时，什么都没有显示-plt.plot(y_costs)。 cost值（从每个纪元产生）仅每50个纪元附加到costs数组。这是为了避免在纪元数确实很高的情况下在数组中添加了太多的cost元素。在行：

if i % 50 == 0:
          costs.append(cost)

当我进行一些调试时，发现方法返回后， costs数组为空。我不确定为什么会这样，何时应该每第50个时期追加一个 cost值。可能我忽略了一个看不见的非常愚蠢的东西。

在此先感谢许多人，并再次为冗长的代码道歉。

from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
import sys
# import os

class NN_classification:

    def __init__(self):
        self.bias = float()
        self.weights = []
        self.chosen_activation_func = None
        self.chosen_cost_func = None
        self.train_average_accuracy = int()
        self.test_average_accuracy = int()

    # -- Activation functions --: 
    def sigmoid(x):
        return 1/(1 + np.exp(-x))

    def relu(x):
        return np.maximum(0.0, x)

    # -- Derivative of activation functions --:
    def sigmoid_derivation(x): 
        return NN_classification.sigmoid(x) * (1-NN_classification.sigmoid(x))

    def relu_derivation(x):
        if x <= 0:
            return 0
        else:
            return 1

    # -- Squared-error cost function --:
    def squared_error(pred, target):
        return np.square(pred - target)

    # -- Derivative of squared-error cost function --:
    def squared_error_derivation(pred, target):
        return 2 * (pred - target)

     # --- neural network structure diagram --- 

    #    O  output prediction
    #   / \   w1, w2, b
    #  O   O  datapoint 1, datapoint 2

    def simple_1_layer_classification_NN(self, dataset_input_matrix, output_data_labels, input_dimension, epochs, activation_func='sigmoid', learning_rate=0.2, cost_func='squared_error'):
        weights = []
        bias = int()
        cost = float()
        costs = []
        dCost_dWeights = []
        chosen_activation_func_derivation = None
        chosen_cost_func = None
        chosen_cost_func_derivation = None
        correct_pred = int()
        incorrect_pred = int()

        # store the chosen activation function to use to it later on in the activation calculation section and in the 'predict' method
        # Also the same goes for the derivation section.        
        if activation_func == 'sigmoid':
            self.chosen_activation_func = NN_classification.sigmoid
            chosen_activation_func_derivation = NN_classification.sigmoid_derivation
        elif activation_func == 'relu':
            self.chosen_activation_func = NN_classification.relu
            chosen_activation_func_derivation = NN_classification.relu_derivation
        else:
            print("Exception error - no activation function utilised, in training method", file=sys.stderr)
            return   

        # store the chosen cost function to use to it later on in the cost calculation section.
        # Also the same goes for the cost derivation section.    
        if cost_func == 'squared_error':
            chosen_cost_func = NN_classification.squared_error
            chosen_cost_func_derivation = NN_classification.squared_error_derivation
        else:
           print("Exception error - no cost function utilised, in training method", file=sys.stderr)
           return

        # Set initial network parameters (weights & bias):
        # Will initialise the weights to a uniform distribution and ensure the numbers are small close to 0.
        # We need to loop through all the weights to set them to a random value initially.
        for i in range(input_dimension):
            # create random numbers for our initial weights (connections) to begin with. 'rand' method creates small random numbers. 
            w = np.random.rand()
            weights.append(w)

        # create a random number for our initial bias to begin with.
        bias = np.random.rand()

        # We perform the training based on the number of epochs specified
        for i in range(epochs):
            # create random index
            ri = np.random.randint(len(dataset_input_matrix))
            # Pick random observation vector: pick a random observation vector of independent variables (x) from the dataset matrix
            input_observation_vector = dataset_input_matrix[ri]

            # reset weighted sum value at the beginning of every epoch to avoid incrementing the previous observations weighted-sums on top. 
            weighted_sum = 0

            # Loop through all the independent variables (x) in the observation
            for i in range(len(input_observation_vector)):
                # Weighted_sum: we take each independent variable in the entire observation, add weight to it then add it to the subtotal of weighted sum
                weighted_sum += input_observation_vector[i] * weights[i]

            # Add Bias: add bias to weighted sum
            weighted_sum += bias

            # Activation: process weighted_sum through activation function
            activation_func_output = self.chosen_activation_func(weighted_sum)    

            # Prediction: Because this is a single layer neural network, so the activation output will be the same as the prediction
            pred = activation_func_output

            # Cost: the cost function to calculate the prediction error margin
            cost = chosen_cost_func(pred, output_data_labels[ri])
            # Also calculate the derivative of the cost function with respect to prediction
            dCost_dPred = chosen_cost_func_derivation(pred, output_data_labels[ri])

            # Derivative: bringing derivative from prediction output with respect to the activation function used for the weighted sum.
            dPred_dWeightSum = chosen_activation_func_derivation(weighted_sum)

            # Bias is just a number on its own added to the weighted sum, so its derivative is just 1
            dWeightSum_dB = 1

            # The derivative of the Weighted Sum with respect to each weight is the input data point / independant variable it's multiplied by. 
            # Therefore I simply assigned the input data array to another variable I called 'dWeightedSum_dWeights'
            # to represent the array of the derivative of all the weights involved. I could've used the 'input_sample'
            # array variable itself, but for the sake of readibility, I created a separate variable to represent the derivative of each of the weights.
            dWeightedSum_dWeights = input_observation_vector

            # Derivative chaining rule: chaining all the derivative functions together (chaining rule)
            # Loop through all the weights to workout the derivative of the cost with respect to each weight:
            for dWeightedSum_dWeight in dWeightedSum_dWeights:
                dCost_dWeight = dCost_dPred * dPred_dWeightSum * dWeightedSum_dWeight
                dCost_dWeights.append(dCost_dWeight)

            dCost_dB = dCost_dPred * dPred_dWeightSum * dWeightSum_dB

            # Backpropagation: update the weights and bias according to the derivatives calculated above.
            # In other word we update the parameters of the neural network to correct parameters and therefore 
            # optimise the neural network prediction to be as accurate to the real output as possible
            # We loop through each weight and update it with its derivative with respect to the cost error function value. 
            for i in range(len(weights)):
                weights[i] = weights[i] - learning_rate * dCost_dWeights[i]

            bias = bias - learning_rate * dCost_dB

            # for each 50th loop we're going to get a summary of the
            # prediction compared to the actual ouput
            # to see if the prediction is as expected.
            # Anything in prediction above 0.5 should match value 
            # 1 of the actual ouptut. Any prediction below 0.5 should
            # match value of 0 for actual output 
            if i % 50 == 0:
                costs.append(cost)

            # Compare prediction to target
            error_margin = np.sqrt(np.square(pred - output_data_labels[ri]))
            accuracy = (1 - error_margin) * 100
            self.train_average_accuracy += accuracy

            # Evaluate whether guessed correctly or not based on classification binary problem 0 or 1 outcome. So if prediction is above 0.5 it guessed 1 and below 0.5 it guessed incorrectly. If it's dead on 0.5 it is incorrect for either guesses. Because it's no exactly a good guess for either 0 or 1. We need to set a good standard for the neural net model.
            if (error_margin < 0.5) and (error_margin >= 0):
                correct_pred += 1 
            elif (error_margin >= 0.5) and (error_margin <= 1):
                incorrect_pred += 1
            else:
                print("Exception error - 'margin error' for 'predict' method is out of range. Must be between 0 and 1, in training method", file=sys.stderr)
                return
        # store the final optimised weights to the weights instance variable so it can be used in the predict method.
        self.weights = weights

        # store the final optimised bias to the weights instance variable so it can be used in the predict method.
        self.bias = bias

        # Calculate average accuracy from the predictions of all obervations in the training dataset
        self.train_average_accuracy /= epochs

        # Print out results 
        print('Average Accuracy: {}'.format(self.train_average_accuracy))
        print('Correct predictions: {}, Incorrect Predictions: {}'.format(correct_pred, incorrect_pred))
        print('costs = {}'.format(costs))
        y_costs = np.array(costs)
        plt.plot(y_costs)
        plt.show()

from numpy import array
#define array of dataset
# each observation vector has 3 datapoints or 3 columns: length, width, and outcome label (0, 1 to represent blue flower and red flower respectively).  
data = array([[3,   1.5, 1],
        [2,   1,   0],
        [4,   1.5, 1],
        [3,   1,   0],
        [3.5, 0.5, 1],
        [2,   0.5, 0],
        [5.5, 1,   1],
        [1,   1,   0]])

# separate data: split input, output, train and test data.
X_train, y_train, X_test, y_test = data[:6, :-1], data[:6, -1], data[6:, :-1], data[6:, -1]

nn_model = NN_classification()

nn_model.simple_1_layer_classification_NN(X_train, y_train, 2, 1000000, learning_rate=0.5)

Answer 1

您是否尝试过降低学习率？您的网络过高可能会跳过本地最小值。

这是一篇关于学习率的文章：https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10

永远不会增加成本的原因是因为您在嵌套的for循环中使用了相同的变量“ i”。

# We perform the training based on the number of epochs specified
    for i in range(epochs):
        # create random index
        ri = np.random.randint(len(dataset_input_matrix))
        # Pick random observation vector: pick a random observation vector of independent variables (x) from the dataset matrix
        input_observation_vector = dataset_input_matrix[ri]

        # reset weighted sum value at the beginning of every epoch to avoid incrementing the previous observations weighted-sums on top.
        weighted_sum = 0

        # Loop through all the independent variables (x) in the observation
        for i in range(len(input_observation_vector)):
            # Weighted_sum: we take each independent variable in the entire observation, add weight to it then add it to the subtotal of weighted sum
            weighted_sum += input_observation_vector[i] * weights[i]

        # Add Bias: add bias to weighted sum
        weighted_sum += bias

        # Activation: process weighted_sum through activation function
        activation_func_output = self.chosen_activation_func(weighted_sum)

        # Prediction: Because this is a single layer neural network, so the activation output will be the same as the prediction
        pred = activation_func_output

        # Cost: the cost function to calculate the prediction error margin
        cost = chosen_cost_func(pred, output_data_labels[ri])
        # Also calculate the derivative of the cost function with respect to prediction
        dCost_dPred = chosen_cost_func_derivation(pred, output_data_labels[ri])

        # Derivative: bringing derivative from prediction output with respect to the activation function used for the weighted sum.
        dPred_dWeightSum = chosen_activation_func_derivation(weighted_sum)

        # Bias is just a number on its own added to the weighted sum, so its derivative is just 1
        dWeightSum_dB = 1

        # The derivative of the Weighted Sum with respect to each weight is the input data point / independant variable it's multiplied by.
        # Therefore I simply assigned the input data array to another variable I called 'dWeightedSum_dWeights'
        # to represent the array of the derivative of all the weights involved. I could've used the 'input_sample'
        # array variable itself, but for the sake of readibility, I created a separate variable to represent the derivative of each of the weights.
        dWeightedSum_dWeights = input_observation_vector

        # Derivative chaining rule: chaining all the derivative functions together (chaining rule)
        # Loop through all the weights to workout the derivative of the cost with respect to each weight:
        for dWeightedSum_dWeight in dWeightedSum_dWeights:
            dCost_dWeight = dCost_dPred * dPred_dWeightSum * dWeightedSum_dWeight
            dCost_dWeights.append(dCost_dWeight)

        dCost_dB = dCost_dPred * dPred_dWeightSum * dWeightSum_dB

        # Backpropagation: update the weights and bias according to the derivatives calculated above.
        # In other word we update the parameters of the neural network to correct parameters and therefore
        # optimise the neural network prediction to be as accurate to the real output as possible
        # We loop through each weight and update it with its derivative with respect to the cost error function value.
        for i in range(len(weights)):
            weights[i] = weights[i] - learning_rate * dCost_dWeights[i]

        bias = bias - learning_rate * dCost_dB

        # for each 50th loop we're going to get a summary of the
        # prediction compared to the actual ouput
        # to see if the prediction is as expected.
        # Anything in prediction above 0.5 should match value
        # 1 of the actual ouptut. Any prediction below 0.5 should
        # match value of 0 for actual output

到if语句时，这导致“ i”始终为1

        if i % 50 == 0:
            costs.append(cost)

        # Compare prediction to target
        error_margin = np.sqrt(np.square(pred - output_data_labels[ri]))
        accuracy = (1 - error_margin) * 100
        self.train_average_accuracy += accuracy

编辑

因此，我尝试对模型进行1000次训练，随机学习率介于0和1之间，初始学习率似乎没有任何区别。其中有0.3％的精度达到0.60以上，没有一个达到70％以上。
然后，我以自适应学习率运行了相同的测试：

# Modify the learning rate based on the cost
# Placed just before the bias is calculated
learning_rate = 0.999 * learning_rate + 0.1 * cost

这导致大约10-12％的模型的精度高于60％，其中大约2.5％的模型的精度高于70％