print("Execution time to fit 200 epochs: %s seconds" % (end_time[7] - start_time[7]))

Execution time to fit 200 epochs: 16.888010263442993 seconds

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[4], 200, subplot=ax1, loss=True, title="Reference: minibatch 10 samples / eta = 0.025")
nn_plot(hist[11], 200, subplot=ax2, loss=True, title="minibatch 64 samples / eta = 0.100")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[8], 200, subplot=ax1, acc=True, title="Reference: minibatch 10 samples / eta = 0.025")
nn_plot(hist[11], 200, subplot=ax2, acc=True, title="minibatch 64 samples / eta = 0.100")

In both cases, with higher learning rate, the models converged at the same speed as our reference, with batch of 10 samples. However we can see that higher the batch and higher the learning rate, more wiggly the metrics become.

For the entire training set, we can perform similar tests, with minibatches of 32 and 64 samples and higher learning rate. The execution time of batch 64, without higher learning rate was:

print("Execution time to fit 100 epochs: %s seconds" % (end_time[1] - start_time[1]))

Execution time to fit 100 epochs: 237.47249603271484 seconds

Entire training set: minibatch 32 / higher learning rate

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation=tf.nn.sigmoid, input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[4])