fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[3], 100, subplot=ax1, loss=True, title="Reference: eta 0.01")
nn_plot(hist[4], 100, subplot=ax2, loss=True, title="eta 0.025")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[3], 100, subplot=ax1, acc=True, title="Reference: eta 0.01")
nn_plot(hist[4], 100, subplot=ax2, acc=True, title="eta 0.025")

The learning rate $\eta = 0.025$ looks to be a good value for our analysis. It converges faster and better than the value $\eta = 0.01$. It can be seen that with 60 epochs, the model converges.

We will apply $\eta = 0.025$ to the entire training set now. We will start by applying batch size of 64, oposed to previous 10, due to the large size of the training set. Later we will investigate the batch size.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation=tf.nn.sigmoid, input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[-1])