fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[16], 200, subplot=ax1, loss=True, title="Reference: 1 hidden layer: 30 nodes")
nn_plot(hist[18], 200, subplot=ax2, loss=True, title="2 hidden layers: 30 nodes each")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[16], 200, subplot=ax1, acc=True, title="Reference: 1 hidden layer: 30 nodes")
nn_plot(hist[18], 200, subplot=ax2, acc=True, title="2 hidden layers: 30 nodes each")

We can see that the performance of our model does not go better with 2 layers. One of the possible reasons is too few nodes in each layer.

2 hidden layers: 100 nodes each

We will run 2 layers again, but with 100 nodes in each layer now.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(100, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', name='hidden_2_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])

model[-1].summary()

Model: "sequential_20"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 hidden_1_layer (Dense)      (None, 100)               78500     
                                                                 
 hidden_2_layer (Dense)      (None, 100)               10100     
                                                                 
 output_layer (Dense)        (None, 10)                1010      
                                                                 
=================================================================
Total params: 89,610
Trainable params: 89,610
Non-trainable params: 0
_________________________________________________________________

tf.keras.utils.plot_model(model[-1], show_shapes=True, show_layer_names=True)