df <- readRDS('../../Data/Session_6_models.rds')
df2 <- df %>% mutate(AAER = ifelse(AAER==0,0,1))
library(xgboost)
# Prep data
train_x <- model.matrix(AAER ~ ., data=df2[df2$Test==0,-1])[,-1]
train_y <- model.frame(AAER ~ ., data=df2[df2$Test==0,])[,"AAER"]
test_x <- model.matrix(AAER ~ ., data=df2[df2$Test==1,-1])[,-1]
test_y <- model.frame(AAER ~ ., data=df2[df2$Test==1,])[,"AAER"]
set.seed(468435) #for reproducibility
xgbCV <- xgb.cv(max_depth=5, eta=0.10, gamma=5, min_child_weight = 4,
subsample = 0.5, objective = "binary:logistic", data=train_x,
label=train_y, nrounds=100, eval_metric="auc", nfold=10,
stratified=TRUE, verbose=0)
fit_ens <- xgboost(params=xgbCV$params, data = train_x, label = train_y,
nrounds = which.max(xgbCV$evaluation_log$test_auc_mean),
verbose = 0)ACCT 420: Ensembling and Ethics
Learning objectives
-
Theory:
- Ensembling
- Ethics
-
Application:
- Fraud detection
- Fairness
- Data security
-
Methodology:
- Ensembling
- SHAP
- Critical thinking
What are ensembles?
- Ensembles are models made out of models
- Ex.: You train 3 models using different techniques, and each seems to work well in certain cases and poorly in others
- If you use the models in isolation, any of them would do an OK (but not great) job
- If you make a model using all three, you can get better performance if their strengths all shine through
- Ensembles range from simple to complex
- Simple: a (weighted) average of a few model’s predictions
When are ensembles useful?
- You have multiple models that are all decent, but none are great
- And, ideally, the models’ predictions are not highly correlated
When are ensembles useful?
- You have a really good model and a bunch of mediocre models
- And, ideally the mediocre models are not highly correlated
A simple ensemble (averaging)
- For continuous predictions, simple averaging is viable
- Often you may want to weight the best model a bit higher
- For binary or categorical predictions, consider averaging ranks
- i.e., instead of using a probability from a logit, use ranks 1, 2, 3, etc.
- Ranks average a bit better, as scores on binary models (particularly when evaluated with measures like AUC) can have extremely different variances across models
- In which case the ensemble is really just the most volatile model’s prediction…
- Not much of an ensemble
- In which case the ensemble is really just the most volatile model’s prediction…
A more complex ensemble (voting model)
- If you have a model the is very good at predicting a binary outcome, ensembling can still help
- This is particularly true when you have other models that capture different aspects of the problem
- Let the other models vote against the best model, and use their prediction if they are above some threshold of agreement
A lot more complex ensemble
- Stacking models (2 layers)
- Train models on subsets (folds) of the training data
- Make predictions for each model on the folds it wasn’t applied to
- Train a new model that takes those predictions as inputs (and optionally the dataset as well)
- Blending (similar to stacking)
- Like stacking, but using predictions on a hold out sample instead of folds (and thus all models are using the same data for predictions)
Simple ensemble results
aucs # Out of sample Ensemble Logit (BCE) Lasso (lambda.min) XGBoost
0.8169036 0.7599594 0.7290185 0.8083503 The ensemble has the best performance
What drives the ensemble?
xgb.train.data = xgb.DMatrix(train_x, label = train_y, missing = NA)
col_names = attr(xgb.train.data, ".Dimnames")[[2]]
imp = xgb.importance(col_names, fit_ens)
# Variable importance
xgb.plot.importance(imp)
Dark knowledge
- Train the simple model not on the actual DV from the training data, but on the best algorithm’s (softened) prediction for the training data
- Somewhat surprisingly, this new, simple algorithm can work almost as well as the full thing!
Examples of Reputational damage
- Microsoft’s Tay and their response
- Coca-Cola: Go make it happy
- Google: Google Photos mistakenly labels black people ‘gorillas’
-
Machine Bias
- ProPublica’s in depth look at racial bias in US courts’ risk assessment algorithms (as of May 2016)
- Note that the number of true positives divided by the number of all positives is more or less equal across ethnicities
Visualizing SHAP across the population
shap_long <- shap.prep(xgb_model = fit_xgb, X_train = train_x)
shap.plot.summary(shap_long)
Visualizing differences in gender within a job
shap.plot.dependence(data_long = shap_long, x = 'Female', y = 'Female',
color_feature = 'Job.TitlesPOLICE OFFICER', jitter_width = 0.05)
SHAP force plot: natural groupings in the data
plot_data <- shap.prep.stack.data(shap_contrib = vals$shap_score,
top_n = 5, n_groups = 6)
shap.plot.force_plot_bygroup(plot_data)
Research on fairness
- Excerpt below from Universal Sentence Encoder
Compares a variety of unintended associations (top) and intended associations (bottom) across Global Vectors (GloVe) and USE