fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[7], 200, subplot=ax1, loss=True, title="Reference: activation sigmoidal")
nn_plot(hist[14], 200, subplot=ax2, loss=True, title="activation relu")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[7], 200, subplot=ax1, acc=True, title="Reference: activation sigmoidal")
nn_plot(hist[14], 200, subplot=ax2, acc=True, title="activation relu")

By analyzing the metrics, it shows that activation function ReLU converges slightly faster than Sigmoidal function. It also shows a slighty improvement in the model. However after 100 epochs, it appears our model becomes overfitted. Before moving on, we will calculate the metrics for the entire training set first, using function ReLU.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])