Regression Tree

In Regression Trees, each partition evaluates the mean value of the kernel.

fit_rpart <- rpart(margin ~ ., data = polls_2008)
plot(fit_rpart, margin = 0.1)
text(fit_rpart, cex = 1)

polls_2008 %>% 
  mutate(y_hat = predict(fit_rpart)) %>% 
  ggplot() +
  geom_point(aes(day, margin), size = 2, alpha = .5) +
  geom_step(aes(day, y_hat), col="red")

An overtraining example of the regression tree is when the algorithm reduces every partition to its own data. This can be avoided by setting the complexity parameter (cp) and the minimum number of observations in each partition. Below is an example of an overtrained tree, with cp = 0 and minsplit = 2.

fit_rpart <- rpart(margin ~ ., data = polls_2008, control = rpart.control(cp = 0, minsplit = 2))
polls_2008 %>% 
  mutate(y_hat = predict(fit_rpart)) %>% 
  ggplot() +
  geom_point(aes(day, margin), size = 2, alpha = .5) +
  geom_step(aes(day, y_hat), col="red")

There is an option to prune off branches, to make smaller trees, in search for better estimates. Below is an example over the previous tree.

pruned_fit <- prune(fit_rpart, cp = 0.01)
polls_2008 %>% 
  mutate(y_hat = predict(pruned_fit)) %>% 
  ggplot() +
  geom_point(aes(day, margin), size = 2, alpha = .5) +
  geom_step(aes(day, y_hat), col="red")

In search of the best complexity parameter for the model, the regression tree uses cross-validation and finds the best one by calculating the residual sum of squares.

train_rpart <- train(margin ~ .,
                     method = "rpart",
                     tuneGrid = data.frame(cp = seq(0, 0.05, len = 25)),
                     data = polls_2008)
ggplot(train_rpart, highlight = TRUE)

train_rpart$bestTune
train_rpart$finalModel

A data.frame: 1 × 1

	cp
	<dbl>
7	0.0125

n= 131 

node), split, n, deviance, yval
      * denotes terminal node

 1) root 131 0.109342100  0.042234730  
   2) day< -39.5 94 0.063168710  0.031616130  
     4) day>=-86.5 41 0.024493780  0.017587400  
       8) day< -49.5 32 0.018979240  0.012794270  
        16) day>=-62 12 0.003585867 -0.003659722 *
        17) day< -62 20 0.010195280  0.022666670  
          34) day< -71.5 13 0.004953632  0.012115380 *
          35) day>=-71.5 7 0.001106548  0.042261900 *
       9) day>=-49.5 9 0.002165432  0.034629630 *
     5) day< -86.5 53 0.024363840  0.042468550  
      10) day>=-117.5 25 0.009066667  0.035000000  
        20) day< -109.5 7 0.001542857  0.022857140 *
        21) day>=-109.5 18 0.006090278  0.039722220 *
      11) day< -117.5 28 0.012657610  0.049136900 *
   3) day>=-39.5 37 0.008647425  0.069211710 *

plot(train_rpart$finalModel, margin = 0.1)
text(train_rpart$finalModel, cex = 1)

polls_2008 %>% 
  mutate(y_hat = predict(train_rpart)) %>% 
  ggplot() +
  geom_point(aes(day, margin), size = 2, alpha = .5) +
  geom_step(aes(day, y_hat), col="red")