fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[24], 100, subplot=ax1, loss=True, title="Reference: full training set / without dropout")
nn_plot(hist[26], 100, subplot=ax2, loss=True, title="full training set / with dropout")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[24], 100, subplot=ax1, acc=True, title="Reference: full training set / without dropout")
nn_plot(hist[26], 100, subplot=ax2, acc=True, title="full training set / with dropout")

# Evaluate the model on the test data using `evaluate`
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test, batch_size=64))
print("test loss, test acc:", results[5])

Evaluate on test data
157/157 [==============================] - 0s 3ms/step - loss: 0.2722 - sparse_categorical_accuracy: 0.9615
test loss, test acc: [0.27221280336380005, 0.9614999890327454]

We can verify again the dropout affecting the training results. However the validation and test results did not improve from our previous model. This indicates that for this specific problem, dropout may not be the best technique to be applied, since the predictors may have intrinsec correlation (pixels near other pixels) that helps for the correct classification and the dropout tries to separate these dependencies.

Final Model Prediction

Our final model which achieved accuracy of 96.86% over the test set consists of: 2 layers, with 100 nodes each, activation function ReLU, L2 Regularization $\lambda = 0.005$, Stochastic Gradient Descent $\eta = 0.025$, mini-batches of 64 samples, 100 epochs training.

We can use it to predict the test set and show randomly some of the wrong predictions.

predictions = model[25].predict(x_test)
classes = np.argmax(predictions, axis=1)

pred_right = np.where(classes == y_test)[0]
pred_wrong = np.where(classes != y_test)[0]

# Showing 16 wrong predictions
rows = 4
cols = 4
axes = []
fig = plt.figure()

for a in range(rows*cols):
    ind = random.randint(0, len(pred_wrong))
    axes.append( fig.add_subplot(rows, cols, a+1))
    axes[-1].set_title("Correct: {}\nPredicted: {}".format(y_test[pred_wrong[ind]], classes[pred_wrong[ind]]))  
    img = plt.imshow(x_test[pred_wrong[ind]].reshape((28, 28)), cmap=plt.cm.gray_r)
    img.axes.get_xaxis().set_visible(False)
    img.axes.get_yaxis().set_visible(False)
plt.tight_layout()
plt.show()

This entire analysis considered the input data, a 28 x 28 pixels square, as a flattened array of 784 predictors. This was the easiest approach, but have some drawbacks.

One of them is how fast the number of parameters increases and how it becomes unpractical to train the model. Our final model, consisting of 2 layers, has 89,610 parameters. Also nearby pixels contain informations about the classification, but our model tries to uncorrelate the predictors, in order to minimize overfitting, and these are conflicting with each other, when trying to find the best model.

It is also important to mention that we purposely chose to investigate each hyperparameter analytically and individually. There are some tools for automatic hyperparameter optimization such as Grid Search and Random Search.

Our next step will be analyze the same digit classification problem with another neural network topology, convolutional neural network, which in fact is largely applied in image classification problems.