MNIST Digits Convolutional Neural Network

Convolutional neural networks are extremely powerful to extract spatial features. This network architecture allows us to build deep networks, with multiple convolution and pooling layers. Each convolution filter extracts important features regarding patterns, borders and neighborhood while the pooling kernels reduce the data size, applying pooling functions as max-pool or average-pool to mantain local informations.

Usually this type of network has several layers and filters and easily can sum up above hundreds of thousands of parameters. If the model is applied to analyze bigger sized colorful images, it can easily reach millions of parameters. It takes much more time to fit the model than the regular feedforward network and also may require thousands of epochs for training, which may take hours to days of training.

We will start by analyzing simpler classification problems and scale up to the full training set.

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import time
import random

mnist = tf.keras.datasets.mnist.load_data()

# Prepare data for training and testing sets
# Preprocess the data
(x_train, y_train), (x_test, y_test) = mnist
x_train = x_train.reshape(x_train.shape[0], x_train.shape[1], x_train.shape[2], 1).astype("float32") / 255.0
x_test = x_test.reshape(x_test.shape[0], x_test.shape[1], x_test.shape[2], 1).astype("float32") / 255.0

print(f"x_train shape: {x_train.shape}")
print(f"x_test shape: {x_test.shape}")

x_train shape: (60000, 28, 28, 1)
x_test shape: (10000, 28, 28, 1)

Initial analysis

Since we are creating and evaluating several neural network models, we define first some functions, for creating and training models and plotting their metrics.

def nn_compile(model, params):
    model.compile(
        loss=params[0],
        optimizer=params[1],
        metrics=[params[2]]
    )

    
def nn_fit(model, x, y, epochs=10, batch_size=64, validation_split=0.1, callbacks=None, verbose=2):
    print("Fit model on training data")
    
    history = model.fit(x, y,
                        epochs=epochs,
                        batch_size=batch_size,
                        validation_split=validation_split,
                        callbacks=callbacks,
                        verbose=verbose
                        )
    
    print("Training finished")
    return history


def nn_plot(history, epochs, subplot=False, loss=False, acc=False, merge=True, binary=False, title=""):
    """
    history: model fit output
    epochs: number of epochs to plot
    subplot: if True, uses subplot for comparison
    loss: plot loss graphic
    acc: plot accuracy graphic
    merge: if True plot train and validation together
    title: use of subplot
    """
    
    if binary==True:
        metric = ['loss', 'val_loss', 'binary_accuracy', 'val_binary_accuracy']
    else:
        metric = ['loss', 'val_loss', 'categorical_accuracy', 'val_categorical_accuracy']
    
    if loss == True:
        if subplot == False:
            if merge == True:
                pd.DataFrame(history.history, columns=metric[0:2]).plot(figsize=(8, 5))
                plt.grid(True)
                plt.legend(['Train loss', 'Validation loss'])
                plt.gca().set_xlim(0, epochs)
                plt.show()
            else:
                pd.DataFrame(history.history[metric[0]]).plot(figsize=(8, 5))
                plt.grid(True)
                plt.legend(['Train loss'])
                plt.gca().set_xlim(0, epochs)
                plt.show()
            
                pd.DataFrame(history.history[metric[1]]).plot(figsize=(8, 5))
                plt.grid(True)
                plt.legend(['Validation loss'])
                plt.gca().set_xlim(0, epochs)
                plt.show()
        else: # subplot merged
            subplot.plot(pd.DataFrame(history.history, columns=metric[0:2]))
            subplot.grid(True)
            subplot.set_title(title)
            subplot.legend(['Train loss', 'Validation loss'])
            subplot.set_xlim(0, epochs)
            
    if acc == True:
        if subplot == False:
            if merge == True:
                pd.DataFrame(history.history,
                             columns=metric[2:4]).plot(figsize=(8, 5))
                plt.grid(True)
                plt.legend(['Train accuracy', 'Validation accuracy'])
                plt.gca().set_xlim(0, epochs)
                plt.show()
            else:
                pd.DataFrame(history.history[metric[2]]).plot(figsize=(8, 5))
                plt.grid(True)
                plt.legend(['Train accuracy'])
                plt.gca().set_xlim(0, epochs)
                plt.show()
            
                pd.DataFrame(history.history[metric[3]]).plot(figsize=(8, 5))
                plt.grid(True)
                plt.legend(['Validation accuracy'])
                plt.gca().set_xlim(0, epochs)
                plt.show()
        else: # subplot merged
            subplot.plot(pd.DataFrame(history.history, columns=metric[2:4]))
            subplot.grid(True)
            subplot.set_title(title)
            subplot.legend(['Train accuracy', 'Validation accuracy'])
            subplot.set_xlim(0, epochs)

model = []
params = []
hist = []
start_time = []
end_time = []
results = []

Binary classification: digits 2 and 7

We start our convolutional network analysis by using it for a binary classification between digits 2 and 7. A binary classification involves simpler functions and less features than the entire dataset. Regarding our model, the significant differences between a binary and multi-class classification are:

• Output layer uses sigmoidal function instead of softmax
• Loss function Binary Cross-entropy instead of Categorical Cross-entropy
• Metrics Binary Accuracy instead of Categorical Accuracy

Our first model has: 3 convolutional layers, interleaved by 2 pooling layers, and a full connection to a binary classification, 1 for digit "2", 0 for digit "7".

The first convolutional layer has 16 filters with 3x3 kernel while the other two have 32 filters with 3x3 kernel. The pooling layers are max-pooling with 2x2 kernel. Before flattening the data, we have 32 kernels 3x3, which them are fully connected to the hidden layer. This layer has 32 nodes with activation function ReLU and binary classification with sigmoidal function. For optimizer we use the Adam algorithm, an extension to stochastic gradient descent method, which is the most recommended one for general problems.

# Mask for digits 2 and 7
ind = (y_train == 2) | (y_train == 7)
x_train_small = x_train[ind]
y_train_small = y_train[ind]
y_train_small = np.where(y_train_small == 2, 1, 0)

ind = (y_test == 2) | (y_test == 7)
x_test_small = x_test[ind]
y_test_small = y_test[ind]
y_test_small = np.where(y_test_small == 2, 1, 0)

print(f"x_train_small shape: {x_train_small.shape}")

x_train_small shape: (12223, 28, 28, 1)

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(16, (3, 3), input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Conv2D(32, (3, 3)),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Conv2D(32, (3, 3)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
]))

params.append([tf.keras.losses.BinaryCrossentropy(),
               tf.keras.optimizers.Adam(),
               tf.keras.metrics.BinaryAccuracy()])

nn_compile(model[0], params[0])

model[0].summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 26, 26, 16)        160       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 13, 13, 16)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 11, 11, 32)        4640      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 5, 5, 32)         0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 3, 3, 32)          9248      
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 32)                9248      
                                                                 
 dense_1 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 23,329
Trainable params: 23,329
Non-trainable params: 0
_________________________________________________________________

tf.keras.utils.plot_model(model[0], show_shapes=True, show_layer_names=True)

For the model training we apply the entire training set containing digits 2 and 7.