Session_9.knit

What are ensembles?

Ensembles are models made out of models
- The output from 2 or more models is the input into the ensemble model
Ex.: You train 3 models using different techniques, and each seems to work well in certain cases and poorly in others
- If you use the models in isolation, then any of them would do an OK (but not great) job
- If you make a model using all three, you can get better performance if their strengths all shine through
Ensembles range from simple to complex
- Simple: a (weighted) average of a few model’s predictions

When are ensembles useful?

You have multiple models that are all decent, but none are great
- And, ideally, the models’ predictions are not highly correlated

When are ensembles useful?

You have a really good model and a bunch of mediocre models
- And, ideally the mediocre models are not highly correlated

When are ensembles useful?

You really need to get just a bit more accuracy/less error out of the model, and you have some other models lying around
You want a more stable model
- It helps to stabilize predictions by limiting the effect of errors or outliers produced by any one model on your prediction
- Think: Diversification (like in finance)

A simple ensemble (averaging)

For continuous predictions, simple averaging is viable
- Often you may want to weight the best model a bit higher
For binary or categorical predictions, consider averaging ranks
- i.e., instead of using a probability from a logit, use ranks 1, 2, 3, etc.
- Ranks average a bit better, as scores on binary models (particularly when evaluated with measures like AUC) can have extremely different variances across models
  - In which case the ensemble is really just the most volatile model’s prediction…
    - Not much of an ensemble

A more complex ensemble (voting model)

If you have a model the is very good at predicting a binary outcome, ensembling can still help
- This is particularly true when you have other models that capture different aspects of the problem
Let the other models vote against the best model, and use their prediction if they are above some threshold of agreement

A lot more complex ensemble

Stacking models (2 layers)
1. Train models on subsets (folds) of the training data
2. Make predictions for each model on the folds it wasn’t applied to
3. Train a new model that takes those predictions as inputs (and optionally the dataset as well)
Blending (similar to stacking)
- Like stacking, but using predictions on a hold out sample instead of folds (and thus all models are using the same data for predictions)

Simple ensemble example

df <- readRDS('../../Data/Session_6_models.rds')
df2 <- df %>% mutate(AAER = ifelse(AAER==0,0,1))
#head(df) %>% select(-pred_F, -pred_S) %>% slice(1:2) %>% html_df()
library(xgboost)

# Prep data
train_x <- model.matrix(AAER ~ ., data=df2[df2$Test==0,-1])[,-1]
train_y <- model.frame(AAER ~ ., data=df2[df2$Test==0,])[,"AAER"]
test_x <- model.matrix(AAER ~ ., data=df2[df2$Test==1,-1])[,-1]
test_y <- model.frame(AAER ~ ., data=df2[df2$Test==1,])[,"AAER"]

set.seed(468435)  #for reproducibility
xgbCV <- xgb.cv(max_depth=5, eta=0.10, gamma=5, min_child_weight = 4,
                subsample = 0.5, objective = "binary:logistic", data=train_x,
                label=train_y, nrounds=100, eval_metric="auc", nfold=10,
                stratified=TRUE, verbose=0)

fit_ens <- xgboost(params=xgbCV$params, data = train_x, label = train_y,
                   nrounds = which.max(xgbCV$evaluation_log$test_auc_mean),
                   verbose = 0)

Simple ensemble results

aucs  # Out of sample

##           Ensemble        Logit (BCE) Lasso (lambda.min)            XGBoost 
##          0.8169036          0.7599594          0.7290185          0.8083503

What drives the ensemble?

xgb.train.data = xgb.DMatrix(train_x, label = train_y, missing = NA)
col_names = attr(xgb.train.data, ".Dimnames")[[2]]
imp = xgb.importance(col_names, fit_ens)
# Variable importance
xgb.plot.importance(imp)

Practicalities

Methods like stacking or blending are much more complex than a simple averaging or voting based ensemble
- But in practice they perform slightly better

Recall the tradeoff between complexity and accuracy!

As such, we may not prefer the complex ensemble in practice, unless we only care about accuracy

Example: In 2009, Netflix awarded a $1M prize to the BellKor’s Pragmatic Chaos team for beating Netflix’s own user preference algorithm by >10%. The alogorithm was so complex that Netflix never used it. It instead used a simpler algorithm with an 8% improvement.

[Geoff Hinton’s] Dark knowledge

Complex ensembles work well
Complex ensembles are exceedingly computationally intensive
- This is bad for running on small or constrained devices (like phones)

Dark knowledge

We can (almost) always create a simple model that approximates the complex model
- Interpret the above literally – we can train a model to fit the model

Dark knowledge

Train the simple model not on the actual DV from the training data, but on the best algorithm’s (softened) prediction for the training data
Somewhat surprisingly, this new, simple algorithm can work almost as well as the full thing!

An example of this dark knowledge

Google’s full model for interpretting human speech is >100GB
- As of October 2019
In Google’s Pixel 4 phone, they have human speech interpretation running locally on the phone
- Not in the cloud like it works on any other Android phone

How did they do this?

They can approximate the output of the complex speech model using a 0.5GB model
0.5GB isn’t small, but it’s small enough to run on a phone

Learning more about Ensembling

Geoff Hinton’s Dark Knowledge slides
- For more details on dark knowledge, applications, and the softening transform
- His interesting (though highly technical) Reddit AMA
A Kaggler’s Guide to Model Stacking in Practice
- A short guide on stacking with nice visualizations
Kaggle Ensembling Guide
- A comprehensive list of ensembling methods with some code samples and applications discussed
Ensemble Learning to Improve Machine Learning Results
- Nicely covers bagging and boosting (two other techniques)

There are many ways to ensemble, and there is no specific guide as to what is best. It may prove useful in the group project, however.

ACCT 420: Machine Learning and AI: Ethics

Session 9

Dr. Richard M. Crowleyrcrowley@smu.edu.sghttp://rmc.link/

Dr. Richard M. Crowley
rcrowley@smu.edu.sg
http://rmc.link/