fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[16], 200, subplot=ax1, loss=True, title="Reference: 1 hidden layer: 30 nodes")
nn_plot(hist[19], 200, subplot=ax2, loss=True, title="2 hidden layers: 100 nodes each")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[16], 200, subplot=ax1, acc=True, title="Reference: 1 hidden layer: 30 nodes")
nn_plot(hist[19], 200, subplot=ax2, acc=True, title="2 hidden layers: 100 nodes each")

Analyzing the graphics, we see that the model with 100 nodes on two layers has similar outcome, but slightly better. When applying several layers, one should be careful with the vanishing gradients and loss of information. We will proceed now to analyze with the full training set.

We compare the results applying one layer with 100 nodes and two layers with 100 nodes each.

Full training set: 1 hidden layer: 100 nodes

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])