print("Execution time with earlier stopping: %s minutes" % ((end_time[7] - start_time[7])/60))

Execution time with earlier stopping: 338.4456728657087 minutes

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[4], 23, subplot=ax1, loss=True, title="Reference: structure 1")
nn_plot(hist[7], 23, subplot=ax2, loss=True, title="structure 4")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[4], 23, subplot=ax1, acc=True, title="Reference: structure 1")
nn_plot(hist[6], 23, subplot=ax2, acc=True, title="structure 4")

# Evaluate the model on the test data using 'evaluate'
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test_label, batch_size=64))
print("test loss, test acc:", results[5])

Evaluate on test data
157/157 [==============================] - 17s 109ms/step - loss: 0.0162 - categorical_accuracy: 0.9954
test loss, test acc: [0.016218937933444977, 0.9954000115394592]

We chose to replicate this model to showcase the performance of a much more complex model, with almost 1 million parameters. In our training step, we took 5h 30min to fit it and achieved an accuracy of 99.54% over the test set. It is important to notice that libraries like tensorflow and pytorch are designed to run models with the GPU, taking much less time through their parallel computing. However this computer did not have GPU, thus used the CPU for processing.

While we achieved an accuracy of 99.54%, the author was capable of achieving an accuracy of 99.67% with data augmentation and by applying different optimizer and learning rate annealer, which will not be studied here.

Final Model Prediction

One important topic to notice is that Convolutional Neural Networks have their parameters randomly initialized. Therefore to evaluate the real performance, we should train multiple times and take the average value for every model. However our foccus here is just illustrate different architectures and analyze the possibilities of employing convolutional neural networks, so only running and training once is enough.

In our studied models, the best accuracy was 99.54% with the last model. We will use it to predict the test set and show some of the errors.

predictions = model[7].predict(x_test)
classes = np.argmax(predictions, axis=1)

pred_right = np.where(classes == y_test)[0]
pred_wrong = np.where(classes != y_test)[0]

# Showing 16 wrong predictions
rows = 4
cols = 4
axes = []
fig = plt.figure()
ind = random.sample(range(len(pred_wrong)), 16)

for a in range(rows*cols):
    i = ind[a]
    axes.append( fig.add_subplot(rows, cols, a+1))
    axes[-1].set_title("Correct: {}\nPredicted: {}".format(y_test[pred_wrong[i]], classes[pred_wrong[i]]))  
    img = plt.imshow(x_test[pred_wrong[i]].reshape((28, 28)), cmap=plt.cm.gray_r)
    img.axes.get_xaxis().set_visible(False)
    img.axes.get_yaxis().set_visible(False)
plt.tight_layout()
plt.show()

By analyzing the wrong predictions it becomes clear that some of these digits are misleading and hard to predict even for humans. Due to the human ability to write outside of any standard, it is relatively impossible to obtain 100% accuracy.

The studies performed here were intended to demonstrate the variety of convolutional neural networks, with countless possibilities of structures, use of layers and hyperparameters. Thus, the complexity of models can vary, and their ability to solve a problem, the number of parameters and time to train as well.