We can notice that there are some techniques that achieved individual accuracy better than the ensemble accuracy, with all techniques. These were the knn, gamLoess and qda techniques.
In order to find a better ensemble, we cannot use the accuracy over the test set. We will find the accuracy estimates, obtained from cross validation on the training set.

Now we build a new ensemble only with techniques with estimated accuracy greater or equal to 0.8.

In the case of outcomes 2 and 7, the Local Weighted Regression had the highest accuracy. This shows that the models obtained depend on random samples and depending on the case, a particular solution may have better accuracy.

Full data analysis

The train set of the full data compromises of 60,000 images, with 784 pixels (predictors) and the test set compromises of 10,000 images with 784 pixels.
Some analysis require more processing time so for these we work with a sample of 10,000 rows from the training set and 1,000 rows from the test set.

We first make an analysis and preprocessing of the data.

We start by looking into the distribution of the pixels variability. Using the standard deviation, we remove predictors (columns) associated with pixels that don't change much and thus can't provide much information for classification.

The image below shows the variance of the pixels, plotted by location. It is reasonable that higher variability happens in the central area, since rarely the digits are written on the boarders.

Filtering the pixels with bigger variance, we end up with 322 predictors of the original 784, those on the center.

We can make a similar histogram to see the distribution of all pixel values, separating the pixels with and without ink. We can then binarize the data.
We use a cutoff to zero out low values that likely smudges, since they represent unwritten space and convert all the remaining entries into 1.

Below is an example between a random image before and after it was binarized.

We will apply the knn and random forests, two very common applied techniques, and study their outcomes.

Knn method

We use 10-fold cross validation to tune the best k parameter tuning. Since the knn method is very suscetible to neighborhood pixels, we use the training dataset before binarizing.

With the optimized k parameter, we can fit the entire dataset for the model.

From the sensitivity and specificity, we can see that with this model, the 8 is the hardest to detect and the most commonly incorrect predicted digit is 9.

We show here 12 examples of correct predictions of the test set.

And below we have 12 examples of incorrect predictions, with their true classification and prediction.

Random Forest Rborist method

Since the fitting step is the slowest part, rather than predicting, as with knn, we will apply 5-fold cross validation.
We use only a small sample (nSamp) of the data first to build every tree and tune the best parameter.

Now we can use the best tuned parameter and the entire dataset to fit the model.

From the sensitivity and specificity, we can see that with this model, the 4 is the hardest to detect and the most commonly incorrect predicted digit is 9.

Below we have 12 examples of incorrect predictions, with their true classification and prediction.

In random forests we can investigate the importance of the predictors. By showing the importance of the pixels on the graphic, we can see that the important ones are located in the center, where the digits change more frequently.

Ensembles

Now we will combine both knn and random forest models in the ensemble, based on the highest average probability.

We can see that the ensemble accuracy improved from the two applied individually.
Below are 12 examples of incorrect predictions, with their true classification and prediction.