print("Execution time to fit 50 epochs: %s minutes" % ((end_time[2] - start_time[2])/60))

Execution time to fit 50 epochs: 29.403780702749888 minutes

nn_plot(hist[2], 50, loss=True, merge=False)

nn_plot(hist[2], 50, acc=True)

# Evaluate the model on the test data using 'evaluate'
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test_label, batch_size=64))
print("test loss, test acc:", results[0])

Evaluate on test data
157/157 [==============================] - 1s 7ms/step - loss: 0.0929 - categorical_accuracy: 0.9874
test loss, test acc: [0.09292740374803543, 0.9873999953269958]

With this simple model, even with aparent overfitting, we already reached an accuracy of 98.74% over the test set, better than feedforward topologies we previously analyzed. This shows how powerful the convolutional neural network architecture is. There is no one specific best architecture, and there are many different hyperparameters, which can be changed.

Hyperparameters optimization

Since the MNIST digits dataset has only 28x28 squares of grayscale pixels, the digits classification problem is too simple for convolutional neural networks. Thus we hardly can see differences between hyperparameters tunning. We will highlight some key hyperparameters.

Applying earlier stopping

One important hyperparameter of neural networks, which is not related to the convolutional topology but to all of them, is the number of epochs for training. Since we don't want to reach overfitting, we set the earlier stopping method. This helps the model to achieve the highest accuracy, before overtraining and reaching overfitting.

We will first compare the previous model with earlier stopping.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(16, (3, 3), input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Conv2D(32, (3, 3)),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Conv2D(32, (3, 3)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
]))

nn_compile(model[-1], params[-1])