print("Execution time to fit 100 epochs: %s seconds" % (end_time[9] - start_time[9]))

Execution time to fit 100 epochs: 242.18311190605164 seconds

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[5], 100, subplot=ax1, loss=True, title="Reference: minibatch 64 samples / eta = 0.025")
nn_plot(hist[13], 100, subplot=ax2, loss=True, title="minibatch 64 samples / eta = 0.100")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[5], 100, subplot=ax1, acc=True, title="Reference: minibatch 64 samples / eta = 0.025")
nn_plot(hist[13], 100, subplot=ax2, acc=True, title="minibatch 64 samples / eta = 0.100")

It can be seen that in both cases, applying higher learning rate was not recommended for entire set. Minibatch of 32 samples took double the time to execute.

For next analysis, when studying the small training set, we will keep batch size 32 with $\eta$ 0.025, since the purpose here is to be simple and analyze individual behaviors. For the entire training set we will be applying batch size 64 with $\eta$ 0.025, which showed the best pair of behavior / execution time.

Activation function

Activation functions are responsible for inserting non-linearities into the neural network model and making it possible to fit any continuous function, linear or nonlinear. Choosing the right activation function on each layer is also an important task.

Until now we have been using the sigmoidal activation function for the internal layer of neurons. It is a very widely used and simple function to choose. However it has some drawbacks like being asymmetric and, most importantly, it looses sensitivity and efficiency with small gradient values, when approaching zero.

\begin{equation} f(x) = \frac{1}{1 + e^{-x}} \end{equation}

We will apply now another very widely used function, the ReLU (Rectified Linear Unit). This function helps making the network sparse, being efficient and easier to compute.

\begin{equation} f(x) = max(0, x) \end{equation}

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])