fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[15], 100, subplot=ax1, loss=True, title="Reference: full training set / 1 hidden layer: 30 nodes")
nn_plot(hist[21], 100, subplot=ax2, loss=True, title="full training set / 2 hidden layers: 100 nodes each")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[15], 100, subplot=ax1, acc=True, title="Reference: full training set / 1 hidden layer: 30 nodes")
nn_plot(hist[21], 100, subplot=ax2, acc=True, title="full training set / 2 hidden layers: 100 nodes each")

# Evaluate the model on the test data using 'evaluate'
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test, batch_size=64))
print("test loss, test acc:", results[3])

Evaluate on test data
157/157 [==============================] - 1s 4ms/step - loss: 0.2827 - sparse_categorical_accuracy: 0.9661
test loss, test acc: [0.28272709250450134, 0.9660999774932861]

When applying two hidden layers, for the entire training set, the model achieved a slightly improvement. The model with 1 layer, 100 nodes, showed an accuracy of 96.28% while with two layers we have 96.61%. For the small training set we will keep with 1 layer with 100 nodes, but for the full training set we will use 2 layers with 100 nodes each.

We still have some more hyperparameters to analyze, in order to further improve our final model.

Regularization

During all our analyzes, we used the regularization factor $\lambda = 0.01$, the default value for L2 Regularization in keras. Now we will try other values for $\lambda$ as well as other regularization methods.

Regularization is used in order to add a fraction of the layers weights to the original loss function. The factor $\lambda$ controls how much these weights will influence the function. This process helps the model to find better weight values, and also minimizes outliers impacts.

L2 Regularization: $\lambda = 0.001$

We will start with $\lambda = 0.001$, ten times smaller than the default value.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, kernel_regularizer=tf.keras.regularizers.l2(0.001),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])