Session 5 R practice

Logistic models and ROC

To run your code, click run. It will let you know if your code is correct or not, and will offer hints on what to fix if needed. If you are stuck, you can see the solution by clicking Solution.

Exercise 1: Running a logit regression

This is a continuation of Session_4’s exercises.

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6ImxpYnJhcnkoZHBseXIpICMgbG9hZHMgZHBseXIgZm9yIGRhdGEgbWFuaXB1bGF0aW9uXG5kZiA8LSBkYXRhLmZyYW1lKHRpYz1yZXAoXCJXTVRcIiwxOTQpLFxuICAgICAgICAgICAgICAgICBmeWVhcnE9YygxOTcwLDE5NzAsMTk3MCwxOTcwLDE5NzEsMTk3MSwxOTcxLDE5NzEsMTk3MiwxOTcyLDE5NzIsMTk3MiwxOTczLDE5NzMsMTk3MywxOTczLDE5NzQsMTk3NCwxOTc0LDE5NzQsMTk3NSwxOTc1LDE5NzUsMTk3NSwxOTc2LDE5NzYsMTk3NiwxOTc2LDE5NzcsMTk3NywxOTc3LDE5NzcsMTk3OCwxOTc4LDE5NzgsMTk3OCwxOTc5LDE5NzksMTk3OSwxOTc5LDE5ODAsMTk4MCwxOTgwLDE5ODAsMTk4MSwxOTgxLDE5ODEsMTk4MSwxOTgyLDE5ODIsMTk4MiwxOTgyLDE5ODMsMTk4MywxOTgzLDE5ODMsMTk4NCwxOTg0LDE5ODQsMTk4NCwxOTg1LDE5ODUsMTk4NSwxOTg1LDE5ODYsMTk4NiwxOTg2LDE5ODYsMTk4NywxOTg3LDE5ODcsMTk4NywxOTg4LDE5ODgsMTk4OCwxOTg4LDE5ODksMTk4OSwxOTg5LDE5ODksMTk5MCwxOTkwLDE5OTAsMTk5MCwxOTkxLDE5OTEsMTk5MSwxOTkxLDE5OTIsMTk5MiwxOTkyLDE5OTIsMTk5MywxOTkzLDE5OTMsMTk5MywxOTk0LDE5OTQsMTk5NCwxOTk0LDE5OTUsMTk5NSwxOTk1LDE5OTUsMTk5NiwxOTk2LDE5OTYsMTk5NiwxOTk3LDE5OTcsMTk5NywxOTk3LDE5OTgsMTk5OCwxOTk4LDE5OTgsMTk5OSwxOTk5LDE5OTksMTk5OSwyMDAwLDIwMDAsMjAwMCwyMDAwLDIwMDEsMjAwMSwyMDAxLDIwMDEsMjAwMiwyMDAyLDIwMDIsMjAwMiwyMDAzLDIwMDMsMjAwMywyMDAzLDIwMDQsMjAwNCwyMDA0LDIwMDQsMjAwNSwyMDA1LDIwMDUsMjAwNSwyMDA2LDIwMDYsMjAwNiwyMDA2LDIwMDcsMjAwNywyMDA3LDIwMDcsMjAwOCwyMDA4LDIwMDgsMjAwOCwyMDA5LDIwMDksMjAwOSwyMDA5LDIwMTAsMjAxMCwyMDEwLDIwMTAsMjAxMSwyMDExLDIwMTEsMjAxMSwyMDEyLDIwMTIsMjAxMiwyMDEyLDIwMTMsMjAxMywyMDEzLDIwMTMsMjAxNCwyMDE0LDIwMTQsMjAxNCwyMDE1LDIwMTUsMjAxNSwyMDE1LDIwMTYsMjAxNiwyMDE2LDIwMTYsMjAxNywyMDE3LDIwMTcsMjAxNywyMDE4LDIwMTgpLFxuICAgICAgICAgICAgICAgICBmcXRyPWMoMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyKSxcbiAgICAgICAgICAgICAgICAgcmV2dHE9Yyg3LjI2Nyw5LjkzOSwxMS40NywxNS42MSwxMi43NTgsMTguMjA3LDE5LjIwNSwyNy44NDQsMjEuMDExLDI5LjkxOCwzMi42ODMsNDEuMjc3LDMyLjczNywzNy42NTUsNDIuMDk4LDU1LjA3MSw0Ni4zNTUsNTYuMTc3LDU5LjQ2Miw3NC4yMTUsNjQuNjUsNzguODUyLDg2LjEwOCwxMTAuNzIxLDkzLjk3MSwxMTUuMTg2LDEyMS44NzUsMTQ3Ljc3NSwxMjIuNjU1LDE1Mi4zODEsMTcyLjYzNywyMzAuNzgzLDE2Ny4yNzQsMjA0LjA4NSwyMjEuOTgyLDMwNi45NTcsMjM4LjU2OSwyOTEuNjg1LDMwMC43NDcsNDE3LjE3NSwzMTQuNTc1LDM3Mi41OTgsNDAxLjg0Miw1NTQuMTg0LDQzNC4yNTEsNTMzLjY1Myw2MzQuNDc2LDg0Mi42MTcsNjY1LjIyOSw3ODguNzA1LDgyNy43NjcsMTA5NC41NTEsODU0LjczNCwxMDk4Ljk0MiwxMTY2LjU5NCwxNTQ2LjYzOSwxMjM0Ljc2OCwxNTA4LjUzMywxNTgzLjU3MywyMDczLjk4MywxNjU1LjY2MSwxOTM0LjM1OCwyMDg3LjUzMywyNzczLjkzNiwyMzQ0LjI1MywyNzcwLjI3NiwyOTQ5LjAwOCwzODQ1LjU1NywzMjExLjc2MSwzNzMyLjIzNyw0MDE1LjAxNyw1MDAwLjIzOCw0MjgyLjAxMiw0ODQ1LjEzMyw0OTkxLjExMyw2NTMwLjczNCw1MzczLjI1OCw2MDQ2LjQxLDYyODMuNDk2LDgxMDcuNDg0LDY3NjguMTkxLDc1NDMuNTA4LDc5MzAuOTQ5LDEwMzU4Ljg5OCw5MjgwLjU2NiwxMDMzOS44OTgsMTA2MjcuNSwxMzYzOC43OTcsMTE2NDkuMzk4LDEzMDI4LjM5OCwxMzY4My43OTcsMTcxMjIsMTM5MjAuMzk4LDE2MjM2LjUsMTY4MjYuODk4LDIwMzYwLjUsMTc2ODYuMDk4LDE5OTQyLjI5NywyMDQxNy42OTksMjQ0NDcuNzk3LDIwNDQwLDIyNzIzLDIyOTE0LDI3NTUwLDIyNzcyLDI1NTg3LDI1NjQ0LDMwODU2LDI1NDA5LDI4Mzg2LDI4Nzc3LDM1Mzg2LDI5ODE5LDMzNTIxLDMzNTA5LDQwNzg1LDM1MTI5LDM4OTEzLDQwODk5LDUxODY4LDQzNDQ3LDQ2NTg4LDQ2MTgxLDU3MDc5LDQ4NTY1LDUzMTg3LDUzMTg1LDY0NzM1LDUyMTI2LDU2NzgxLDU1NzY1LDY2OTA1LDU3MjI0LDYzMjMxLDYzMDM2LDc1MTkwLDY1NDM0LDcwNTE2LDY5MjgyLDgyOTAwLDcwNzU1LDc2Njk3LDc1Mzk3LDg5MjUyLDc5Njc1LDg1NDMwLDg0NDY3LDk5MDc4LDg2Mzc4LDkyODI3LDkxNzk0LDEwNzI4OSw5NDk0MCwxMDIzNDIsOTgzNDUsMTA4NzQ3LDk0MjE0LDEwMDg3Niw5OTM3MywxMTM2MjIsOTk4MTEsMTAzNzI2LDEwMTk1MiwxMTYzNjAsMTA0MTg5LDEwOTM2NiwxMTAyMjYsMTIzMTY5LDExMzAxMCwxMTQwNjUsMTEzODAwLDEyNzc3NiwxMTQwNzAsMTE2ODMwLDExNTY4OCwxMjk3MDYsMTE0OTYwLDEyMDEyNSwxMTkwMDEsMTMxNTY1LDExNDgyNiwxMjAyMjksMTE3NDA4LDEyOTY2NywxMTU5MDQsMTIwMzE5LDExODE3OSwxMzA5MzYsMTE3NTQyLDEyMjk2OCwxMjMxNzksMTM2MjY3LDEyMjY5MCwxMjgwMjgpLFxuICAgICAgICAgICAgICAgICByZWNlc3Npb249YygxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwKSlcbmRmIDwtIGRmICU+JSBtdXRhdGUocmV2dHFfeW95ID0gKHJldnRxIC0gbGFnKHJldnRxLCA0KSkgLyBsYWcocmV2dHEsIDQpLFxuICAgICAgICAgICAgICAgICAgICByZWNlc3Npb25fbGFnID0gbGFnKHJlY2Vzc2lvbiksXG4gICAgICAgICAgICAgICAgICAgIHJldnRxX2Rvd24gPSBmYWN0b3IoaWZlbHNlKHJldnRxX3lveTwwLDEsMCksIGxldmVscz1jKDAsMSkpLFxuICAgICAgICAgICAgICAgICAgICByZXZ0cV95b3lfbGFnID0gbGFnKHJldnRxX3lveSksXG4gICAgICAgICAgICAgICAgICAgIGRhdGUgPSBmeWVhcnEgKyAoZnF0ci0xKS80KSIsInNhbXBsZSI6IiMgVGhlIGRhdGFmcmFtZSBiZWxvdyBjb250YWlucyBoaXN0b3JpY2FsIHJldmVudWUgZm9yIFdhbG1hcnQgc2luY2UgMTk3MFxuIyBBbGwgZGF0YSBpcyBxdWFydGVybHlcbiMgVGhlIGZpbGUgdXNlcyBDb21wdXN0YXQncyBub21lbmNsYXR1cmU6XG4jICAgICAtIHJldmVudWUgaXMgcmV2dHFcbiMgICAgIC0gdGlja2VyIGlzIHRpY1xuIyAgICAgLSBmeWVhcnEgaXMgeWVhclxuIyAgICAgLSBmcXRyIGlzIHF1YXJ0ZXJcbiMgICAgIC0gcmVjZXNzaW9uIGlzIGFuIGluZGljYXRvciBmb3IgVVMgcmVjZXNzaW9uc1xuIyAgICAgLSByZXZ0cV95b3kgaXMgeWVhciBvdmVyIHllYXIgcmV2ZW51ZSBncm93dGhcbiMgICAgIC0gcmV2dHFfZG93biBpcyBhbiBpbmRpY2F0b3IgZXF1YWwgdG8gMSBpZiByZXZ0cV95b3kgaXMgbmVnYXRpdmUsIGVsc2UgMFxuIyAgICAgLSByZXZ0cV95b3lfbGFnIGlzIHRoZSBsYWcgb2YgcmV2dHFfeW95XG5kZlsxOjUsXVxuXG4jIENhbGN1bGF0ZSBhIGxvZ2lzdGljIG1vZGVsIG9mIGEgZGVjcmVhc2UgaW4geWVhciBvdmVyIHllYXIgcXVhcnRlcmx5IHJldmVudWVcbiMgb24gdGhlIHByZXZpb3VzIHF1YXJ0ZXIncyB5ZWFyIG92ZXIgeWVhciByZXZlbnVlIGdyb3d0aCwgc3RvcmUgaXQgaW4gbW9kMVxubW9kMSA8LSBnbG0oIH4gLCBkYXRhPSwgZmFtaWx5PSlcblxuIyBQcmludCBvdXQgYSBzdW1tYXJ5IG9mIHRoZSBtb2RlbCB1c2luZyBzdW1tYXJ5XG5zdW1tYXJ5KG1vZDEpXG5cbiNFTkQiLCJzb2x1dGlvbiI6IiMgVGhlIGRhdGFmcmFtZSBiZWxvdyBjb250YWlucyBoaXN0b3JpY2FsIHJldmVudWUgZm9yIFdhbG1hcnQgc2luY2UgMTk3MFxuIyBBbGwgZGF0YSBpcyBxdWFydGVybHlcbiMgVGhlIGZpbGUgdXNlcyBDb21wdXN0YXQncyBub21lbmNsYXR1cmU6XG4jICAgICAtIHJldmVudWUgaXMgcmV2dHFcbiMgICAgIC0gdGlja2VyIGlzIHRpY1xuIyAgICAgLSBmeWVhcnEgaXMgeWVhclxuIyAgICAgLSBmcXRyIGlzIHF1YXJ0ZXJcbiMgICAgIC0gcmVjZXNzaW9uIGlzIGFuIGluZGljYXRvciBmb3IgVVMgcmVjZXNzaW9uc1xuIyAgICAgLSByZXZ0cV95b3kgaXMgeWVhciBvdmVyIHllYXIgcmV2ZW51ZSBncm93dGhcbiMgICAgIC0gcmV2dHFfZG93biBpcyBhbiBpbmRpY2F0b3IgZXF1YWwgdG8gMSBpZiByZXZ0cV95b3kgaXMgbmVnYXRpdmUsIGVsc2UgMFxuIyAgICAgLSByZXZ0cV95b3lfbGFnIGlzIHRoZSBsYWcgb2YgcmV2dHFfeW95XG5kZlsxOjUsXVxuXG4jIENhbGN1bGF0ZSBhIGxvZ2lzdGljIG1vZGVsIG9mIGEgZGVjcmVhc2UgaW4geWVhciBvdmVyIHllYXIgcXVhcnRlcmx5IHJldmVudWVcbiMgb24gdGhlIHByZXZpb3VzIHF1YXJ0ZXIncyB5ZWFyIG92ZXIgeWVhciByZXZlbnVlIGdyb3d0aCwgc3RvcmUgaXQgaW4gbW9kMVxubW9kMSA8LSBnbG0ocmV2dHFfZG93biB+IHJldnRxX3lveV9sYWcsIGRhdGE9ZGYsIGZhbWlseT1iaW5vbWlhbClcblxuIyBQcmludCBvdXQgYSBzdW1tYXJ5IG9mIHRoZSBtb2RlbCB1c2luZyBzdW1tYXJ5XG5zdW1tYXJ5KG1vZDEpXG5cbiNFTkQiLCJzY3QiOiIjIFRlbXBsYXRlIGJhc2VkIG9uIGh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy90ZXN0d2hhdC92ZXJzaW9ucy80LjEuMVxuIyBDaGVjayBpZiBzb21ldGhpbmcgaXMgZXhwbGljaXRseSB0eXBlZFxuXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ3JldnRxX2Rvd24gfiByZXZ0cV95b3lfbGFnJyxub3RfdHlwZWRfbXNnPSdNYWtlIHN1cmUgdG8gcmVncmVzcyBgcmV2dHFfZG93bmAgb24gYHJldnRxX3lveV9sYWdgLicpXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ2RhdGE9ZGYnLG5vdF90eXBlZF9tc2c9J01ha2Ugc3VyZSB0byB1c2luZyB0aGUgYGRmYCBkYXRhIGZyYW1lIGZvciB0aGUgdGVzdC4nKVxudGVzdF9zdHVkZW50X3R5cGVkKCdmYW1pbHk9Ymlub21pYWwnLG5vdF90eXBlZF9tc2c9J01ha2Ugc3VyZSB0byBzZXQgZmFtaWx5IHRvIGJlIGBiaW5vbWlhbGAsIHNvIGFzIHRvIHJ1biBhIGxvZ2lzdGljIHJlZ3Jlc3Npb24uJylcbnRlc3RfZXhwcmVzc2lvbl9vdXRwdXQoXCJtb2QxXCIsIGluY29ycmVjdF9tc2c9XCJZb3VyIHJlZ3Jlc3Npb24gbW9kZWwgaXNuJ3QgcXVpdGUgcmlnaHQuXCIpXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ3N1bW1hcnkobW9kMSknLG5vdF90eXBlZF9tc2c9J1lvdXIgbW9kZWwgaXMgY29ycmVjdCwgYnV0IGRpZCB5b3UgcmVtZW1iZXIgdG8gc3VtbWFyaXplIGl0IHdpdGggYHN1bW1hcnkoKWA/JylcblxuIyB0ZXN0X3N0dWRlbnRfdHlwZWQoJ3ggPC0gMicsIG5vdF90eXBlZF9tc2c9JycpXG5cbiMgQ2hlY2sgaWYgZnVuY3Rpb24gd2FzIHVzZWQgaW4gaW5wdXQgY29kZVxuIyB0ZXN0X2Z1bmN0aW9uKCdjJyxpbmNvcnJlY3RfbXNnPScnKSAgXG5cbiMgUmVxdWlyZXMgYW4gb2JqZWN0IGB4YCB0byBoYXZlIHRoZSBzYW1lIHZhbHVlIGFzIHRoZSBzb2x1dGlvblxuIyB0ZXN0X29iamVjdChcInhcIixpbmNvcnJlY3RfbXNnID0gXCJcIix1bmRlZmluZWRfbXNnID0gXCJcIikgIFxuXG4jIFJlcXVpcmVzIGFuIG9uamVjdCB3aXRoIHRoZSBzYW1lIHZhbHVlIG9mIGB4YCBpbiB0aGUgc29sdXRpb25cbiMgdGVzdF9hbl9vYmplY3QoXCJ4XCIsdW5kZWZpbmVkX21zZz1cIlwiKVxuXG4jIENoZWNrcyBpZiBvdXRwdXQgb2Ygc3R1ZGVudCdzIGNvZGUgY29udGFpbnMgZ2l2ZW4gZXZhbHVhdGVkIGV4cHJlc3Npb25cbiMgdGVzdF9vdXRwdXRfY29udGFpbnMoXCJ4XCIsaW5jb3JyZWN0X21zZyA9IFwiXCIpXG5cbiMgQ2hlY2sgaWYgYSB2ZWN0b3Igb2YgcHJlZGVmaW5lZCBvYmplY3RzIGFyZSB1bmNoYW5nZWRcbiMgdGVzdF9wcmVkZWZpbmVkX29iamVjdHMoYygneCcsJ3knKSxpbmNvcnJlY3RfbXNnPVwiRG9uJ3Qgb252ZXJ3cml0ZSB0aGUgcHJlZGVmaW5lZCB2YXJpYWJsZXNcIilcblxuIyBDaGVja3MgZm9yIGEgcmVnZXggcGF0dGVybiBpbiB0cmhlIG91dHB1dFxuIyB0ZXN0X291dHB1dF9yZWdleChwYXR0ZXJuLGZpeGVkPUYsIHRpbWVzPTEsIGluY29ycmVjdF9tc2c9JycpXG5cbiMgQ2FuIGNoZWNrIGFuIGFyYml0cmFyeSBleHByZXNzaW9uIGFjcm9zcyBib3RoIHNvbHV0aW9uIGFuZCBzdHVkZW50IGNvZGVcbiN0ZXN0X2V4cHJlc3Npb25fb3V0cHV0KFwidHlwZW9mKGNvbXBhbnlfbmFtZSlcIiwgaW5jb3JyZWN0X21zZz1cIkRpZCB5b3Ugc3RvcmUgdGV4dHVhbCBkYXRhIGluIGBjb21wYW55X25hbWVgP1wiKVxuXG50ZXN0X2Vycm9yKClcbnN1Y2Nlc3NfbXNnKFwiQXdlc29tZSFcIilcblxuIyBPdGhlciBmdW5jdGlvbnMgdG8gbm90ZTpcbiMgICAgIC0gdGVzdF9vcihhLGIpIC0tIGNoZWNrcyBpZiBlaXRoZXIgdGVzdCBhIG9yIHRlc3QgYiBwYXNzXG4jICAgICAtIHRlc3RfZ2dwbG90KCkgLS0gY2FuIGNoZWNrIGlmIHBsb3RzIGFyZSBjb3JyZWN0XG4jICAgICAtIHRlc3RfZnVuY3Rpb24oKSAtLSBjYW4gYWxzbyBjaGVjayBpbmNsdWRlZCBwYXJhbWV0ZXJzXG4jICAgICAtIHRlc3RfbG9vcCgpIC0tIGNoZWNraW5nIGZvciBhbmQgd2hpbGUgbG9vcHNcbiMgICAgIC0gdGVzdF9saWJyYXJ5X2Z1bmN0aW9uKCdwYWNrYWdlJywgbm90X2NhbGxlZF9tc2c9JycsaW5jb3JyZWN0X21zZz0nJylcbiMgICAgIC0gdGVzdF9pZl9lbHNlKCkgLS0gY2hlY2tpbmcgaWYgc3RhdGVtZW50c1xuIyAgICAgLSB0ZXN0X2V4cHJlc3Npb25fZXJyb3IoKSAtLSBjYW4gY2hlY2sgaWYgZnVuY3Rpb25zIGFyZSBwcm9wZXJseSBkZWZpbmVkXG4jICAgICAtIHRlc3Rfb3BlcmF0b3IoJ29wZXJhdG9yJywpLCBub3RfY2FsbGVkX21zZz0nJyxpbmNvcnJlY3RfbXNnPScnKVxuIyAgICAgLSB0ZXN0X2Z1bmN0aW9uX2RlZmluaXRpb24oKSAtLSByaWdvcm91c2x5IGNoZWNrIGRlZmluZWQgZnVuY3Rpb25cbiMgICAgIC0gdGVzdF9kYXRhX2ZyYW1lKCkgLS0gY2hlY2sgaWYgZGF0YWZyYW1lIFtjb2x1bW5zXSBhcmUgZXF1aXZhbGVudFxuIyAgICAgLSB0ZXN0X2Z1bmN0aW9uX3Jlc3VsdCwgdGVzdF9leHByZXNzaW9uX3Jlc3VsdCJ9

Exercise 2: Exploring model predictions

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6ImxpYnJhcnkoZHBseXIpICMgbG9hZHMgZHBseXIgZm9yIGRhdGEgbWFuaXB1bGF0aW9uXG5saWJyYXJ5KGdncGxvdDIpXG5kZiA8LSBkYXRhLmZyYW1lKHRpYz1yZXAoXCJXTVRcIiwxOTQpLFxuICAgICAgICAgICAgICAgICBmeWVhcnE9YygxOTcwLDE5NzAsMTk3MCwxOTcwLDE5NzEsMTk3MSwxOTcxLDE5NzEsMTk3MiwxOTcyLDE5NzIsMTk3MiwxOTczLDE5NzMsMTk3MywxOTczLDE5NzQsMTk3NCwxOTc0LDE5NzQsMTk3NSwxOTc1LDE5NzUsMTk3NSwxOTc2LDE5NzYsMTk3NiwxOTc2LDE5NzcsMTk3NywxOTc3LDE5NzcsMTk3OCwxOTc4LDE5NzgsMTk3OCwxOTc5LDE5NzksMTk3OSwxOTc5LDE5ODAsMTk4MCwxOTgwLDE5ODAsMTk4MSwxOTgxLDE5ODEsMTk4MSwxOTgyLDE5ODIsMTk4MiwxOTgyLDE5ODMsMTk4MywxOTgzLDE5ODMsMTk4NCwxOTg0LDE5ODQsMTk4NCwxOTg1LDE5ODUsMTk4NSwxOTg1LDE5ODYsMTk4NiwxOTg2LDE5ODYsMTk4NywxOTg3LDE5ODcsMTk4NywxOTg4LDE5ODgsMTk4OCwxOTg4LDE5ODksMTk4OSwxOTg5LDE5ODksMTk5MCwxOTkwLDE5OTAsMTk5MCwxOTkxLDE5OTEsMTk5MSwxOTkxLDE5OTIsMTk5MiwxOTkyLDE5OTIsMTk5MywxOTkzLDE5OTMsMTk5MywxOTk0LDE5OTQsMTk5NCwxOTk0LDE5OTUsMTk5NSwxOTk1LDE5OTUsMTk5NiwxOTk2LDE5OTYsMTk5NiwxOTk3LDE5OTcsMTk5NywxOTk3LDE5OTgsMTk5OCwxOTk4LDE5OTgsMTk5OSwxOTk5LDE5OTksMTk5OSwyMDAwLDIwMDAsMjAwMCwyMDAwLDIwMDEsMjAwMSwyMDAxLDIwMDEsMjAwMiwyMDAyLDIwMDIsMjAwMiwyMDAzLDIwMDMsMjAwMywyMDAzLDIwMDQsMjAwNCwyMDA0LDIwMDQsMjAwNSwyMDA1LDIwMDUsMjAwNSwyMDA2LDIwMDYsMjAwNiwyMDA2LDIwMDcsMjAwNywyMDA3LDIwMDcsMjAwOCwyMDA4LDIwMDgsMjAwOCwyMDA5LDIwMDksMjAwOSwyMDA5LDIwMTAsMjAxMCwyMDEwLDIwMTAsMjAxMSwyMDExLDIwMTEsMjAxMSwyMDEyLDIwMTIsMjAxMiwyMDEyLDIwMTMsMjAxMywyMDEzLDIwMTMsMjAxNCwyMDE0LDIwMTQsMjAxNCwyMDE1LDIwMTUsMjAxNSwyMDE1LDIwMTYsMjAxNiwyMDE2LDIwMTYsMjAxNywyMDE3LDIwMTcsMjAxNywyMDE4LDIwMTgpLFxuICAgICAgICAgICAgICAgICBmcXRyPWMoMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyKSxcbiAgICAgICAgICAgICAgICAgcmV2dHE9Yyg3LjI2Nyw5LjkzOSwxMS40NywxNS42MSwxMi43NTgsMTguMjA3LDE5LjIwNSwyNy44NDQsMjEuMDExLDI5LjkxOCwzMi42ODMsNDEuMjc3LDMyLjczNywzNy42NTUsNDIuMDk4LDU1LjA3MSw0Ni4zNTUsNTYuMTc3LDU5LjQ2Miw3NC4yMTUsNjQuNjUsNzguODUyLDg2LjEwOCwxMTAuNzIxLDkzLjk3MSwxMTUuMTg2LDEyMS44NzUsMTQ3Ljc3NSwxMjIuNjU1LDE1Mi4zODEsMTcyLjYzNywyMzAuNzgzLDE2Ny4yNzQsMjA0LjA4NSwyMjEuOTgyLDMwNi45NTcsMjM4LjU2OSwyOTEuNjg1LDMwMC43NDcsNDE3LjE3NSwzMTQuNTc1LDM3Mi41OTgsNDAxLjg0Miw1NTQuMTg0LDQzNC4yNTEsNTMzLjY1Myw2MzQuNDc2LDg0Mi42MTcsNjY1LjIyOSw3ODguNzA1LDgyNy43NjcsMTA5NC41NTEsODU0LjczNCwxMDk4Ljk0MiwxMTY2LjU5NCwxNTQ2LjYzOSwxMjM0Ljc2OCwxNTA4LjUzMywxNTgzLjU3MywyMDczLjk4MywxNjU1LjY2MSwxOTM0LjM1OCwyMDg3LjUzMywyNzczLjkzNiwyMzQ0LjI1MywyNzcwLjI3NiwyOTQ5LjAwOCwzODQ1LjU1NywzMjExLjc2MSwzNzMyLjIzNyw0MDE1LjAxNyw1MDAwLjIzOCw0MjgyLjAxMiw0ODQ1LjEzMyw0OTkxLjExMyw2NTMwLjczNCw1MzczLjI1OCw2MDQ2LjQxLDYyODMuNDk2LDgxMDcuNDg0LDY3NjguMTkxLDc1NDMuNTA4LDc5MzAuOTQ5LDEwMzU4Ljg5OCw5MjgwLjU2NiwxMDMzOS44OTgsMTA2MjcuNSwxMzYzOC43OTcsMTE2NDkuMzk4LDEzMDI4LjM5OCwxMzY4My43OTcsMTcxMjIsMTM5MjAuMzk4LDE2MjM2LjUsMTY4MjYuODk4LDIwMzYwLjUsMTc2ODYuMDk4LDE5OTQyLjI5NywyMDQxNy42OTksMjQ0NDcuNzk3LDIwNDQwLDIyNzIzLDIyOTE0LDI3NTUwLDIyNzcyLDI1NTg3LDI1NjQ0LDMwODU2LDI1NDA5LDI4Mzg2LDI4Nzc3LDM1Mzg2LDI5ODE5LDMzNTIxLDMzNTA5LDQwNzg1LDM1MTI5LDM4OTEzLDQwODk5LDUxODY4LDQzNDQ3LDQ2NTg4LDQ2MTgxLDU3MDc5LDQ4NTY1LDUzMTg3LDUzMTg1LDY0NzM1LDUyMTI2LDU2NzgxLDU1NzY1LDY2OTA1LDU3MjI0LDYzMjMxLDYzMDM2LDc1MTkwLDY1NDM0LDcwNTE2LDY5MjgyLDgyOTAwLDcwNzU1LDc2Njk3LDc1Mzk3LDg5MjUyLDc5Njc1LDg1NDMwLDg0NDY3LDk5MDc4LDg2Mzc4LDkyODI3LDkxNzk0LDEwNzI4OSw5NDk0MCwxMDIzNDIsOTgzNDUsMTA4NzQ3LDk0MjE0LDEwMDg3Niw5OTM3MywxMTM2MjIsOTk4MTEsMTAzNzI2LDEwMTk1MiwxMTYzNjAsMTA0MTg5LDEwOTM2NiwxMTAyMjYsMTIzMTY5LDExMzAxMCwxMTQwNjUsMTEzODAwLDEyNzc3NiwxMTQwNzAsMTE2ODMwLDExNTY4OCwxMjk3MDYsMTE0OTYwLDEyMDEyNSwxMTkwMDEsMTMxNTY1LDExNDgyNiwxMjAyMjksMTE3NDA4LDEyOTY2NywxMTU5MDQsMTIwMzE5LDExODE3OSwxMzA5MzYsMTE3NTQyLDEyMjk2OCwxMjMxNzksMTM2MjY3LDEyMjY5MCwxMjgwMjgpLFxuICAgICAgICAgICAgICAgICByZWNlc3Npb249YygxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwKSlcbmRmIDwtIGRmICU+JSBtdXRhdGUocmV2dHFfeW95ID0gKHJldnRxIC0gbGFnKHJldnRxLCA0KSkgLyBsYWcocmV2dHEsIDQpLFxuICAgICAgICAgICAgICAgICAgICByZWNlc3Npb25fbGFnID0gbGFnKHJlY2Vzc2lvbiksXG4gICAgICAgICAgICAgICAgICAgIHJldnRxX2Rvd24gPSBpZmVsc2UocmV2dHFfeW95PDAsMSwwKSxcbiAgICAgICAgICAgICAgICAgICAgcmV2dHFfeW95X2xhZyA9IGxhZyhyZXZ0cV95b3kpLFxuICAgICAgICAgICAgICAgICAgICBkYXRlID0gZnllYXJxICsgKGZxdHItMSkvNClcbm1vZDEgPC0gZ2xtKHJldnRxX2Rvd24gfiByZXZ0cV95b3lfbGFnLCBkYXRhPWRmLCBmYW1pbHk9Ymlub21pYWwpIiwic2FtcGxlIjoiIyBUaGUgZGF0YWZyYW1lIGJlbG93IGNvbnRhaW5zIGhpc3RvcmljYWwgcmV2ZW51ZSBmb3IgV2FsbWFydCBzaW5jZSAxOTcwXG4jIEFsbCBkYXRhIGlzIHF1YXJ0ZXJseVxuIyBUaGUgZmlsZSB1c2VzIENvbXB1c3RhdCdzIG5vbWVuY2xhdHVyZTpcbiMgICAgIC0gcmV2dHFfeW95IGlzIHllYXIgb3ZlciB5ZWFyIHJldmVudWUgZ3Jvd3RoXG4jICAgICAtIHJldnRxX2Rvd24gaXMgYW4gaW5kaWNhdG9yIGVxdWFsIHRvIDEgaWYgcmV2dHFfeW95IGlzIG5lZ2F0aXZlLCBlbHNlIDBcbiMgICAgIC0gcmV2dHFfeW95X2xhZyBpcyB0aGUgbGFnIG9mIHJldnRxX3lveVxuIyAgICAgLSBkYXRlIGlzIGEgZnJhY3Rpb25hbCBkYXRlXG4jIG1vZDEgZnJvbSB0aGUgbGFzdCBleGVyY2lzZSBpcyBhbHNvIGxvYWRlZFxuXG4jIFVzZSBwcmVkaWN0KCkgdG8gY2FsY3VsYXRlIHRoZSBwcm9iYWJpbGl0eSBvZiBkZWNyZWFzaW5nIHllYXIgb3ZlciB5ZWFyXG4jIHJldmVudWUgZm9yIFdhbG1hcnQgYmFzZWQgb24gbW9kMVxuZGYkcHJlZCA8LSBwcmVkaWN0KG1vZDEsIGRmLCB0eXBlPVwicmVzcG9uc2VcIilcblxuIyBQbG90IG91dCB0aGUgYWN0dWFsIGRlY3JlYXNlcyBhbmQgcHJlZGljdGVkIGRlY3JlYXNlIHByb2JhYmlsaXRpZXMgYnkgeWVhclxuIyBnZ3Bsb3QyIGlzIGFscmVhZHkgbG9hZGVkXG5nZ3Bsb3QoZGF0YT1kZikgKyBcbiAgZ2VvbV9wb2ludChhZXMoeSA9ICwgeCA9ICwgY29sb3IgPSBcIkFjdHVhbFwiKSkgKyBcbiAgZ2VvbV9wb2ludChhZXMoeSA9ICwgeCA9ICwgY29sb3IgPSBcIlByZWRpY3RlZFwiKSkgKyBcbiAgeWxhYihcIkRlY3JlYXNlIGluIHJldmVudWVcIilcblxuXG4jRU5EIiwic29sdXRpb24iOiIjIFRoZSBkYXRhZnJhbWUgYmVsb3cgY29udGFpbnMgaGlzdG9yaWNhbCByZXZlbnVlIGZvciBXYWxtYXJ0IHNpbmNlIDE5NzBcbiMgQWxsIGRhdGEgaXMgcXVhcnRlcmx5XG4jIFRoZSBmaWxlIHVzZXMgQ29tcHVzdGF0J3Mgbm9tZW5jbGF0dXJlOlxuIyAgICAgLSByZXZ0cV95b3kgaXMgeWVhciBvdmVyIHllYXIgcmV2ZW51ZSBncm93dGhcbiMgICAgIC0gcmV2dHFfZG93biBpcyBhbiBpbmRpY2F0b3IgZXF1YWwgdG8gMSBpZiByZXZ0cV95b3kgaXMgbmVnYXRpdmUsIGVsc2UgMFxuIyAgICAgLSByZXZ0cV95b3lfbGFnIGlzIHRoZSBsYWcgb2YgcmV2dHFfeW95XG4jICAgICAtIGRhdGUgaXMgYSBmcmFjdGlvbmFsIGRhdGVcbiMgbW9kMSBmcm9tIHRoZSBsYXN0IGV4ZXJjaXNlIGlzIGFsc28gbG9hZGVkXG5cbiMgVXNlIHByZWRpY3QoKSB0byBjYWxjdWxhdGUgdGhlIHByb2JhYmlsaXR5IG9mIGRlY3JlYXNpbmcgeWVhciBvdmVyIHllYXJcbiMgcmV2ZW51ZSBmb3IgV2FsbWFydCBiYXNlZCBvbiBtb2QxXG5kZiRwcmVkIDwtIHByZWRpY3QobW9kMSwgZGYsIHR5cGU9XCJyZXNwb25zZVwiKVxuXG4jIFBsb3Qgb3V0IHRoZSBhY3R1YWwgZGVjcmVhc2VzIGFuZCBwcmVkaWN0ZWQgZGVjcmVhc2UgcHJvYmFiaWxpdGllcyBieSBkYXRlXG4jIGdncGxvdDIgaXMgYWxyZWFkeSBsb2FkZWRcbmdncGxvdChkYXRhPWRmKSArIFxuICBnZW9tX3BvaW50KGFlcyh5ID0gcmV2dHFfZG93biwgeCA9IGRhdGUsIGNvbG9yID0gXCJBY3R1YWxcIikpICsgXG4gIGdlb21fcG9pbnQoYWVzKHkgPSBwcmVkLCB4ID0gZGF0ZSwgY29sb3IgPSBcIlByZWRpY3RlZFwiKSkgKyBcbiAgeWxhYihcIkRlY3JlYXNlIGluIHJldmVudWVcIilcblxuXG4jRU5EIiwic2N0IjoiIyBUZW1wbGF0ZSBiYXNlZCBvbiBodHRwczovL3d3dy5yZG9jdW1lbnRhdGlvbi5vcmcvcGFja2FnZXMvdGVzdHdoYXQvdmVyc2lvbnMvNC4xLjFcbiMgQ2hlY2sgaWYgc29tZXRoaW5nIGlzIGV4cGxpY2l0bHkgdHlwZWRcblxudGVzdF9zdHVkZW50X3R5cGVkKCd4ID0gZGF0ZScsbm90X3R5cGVkX21zZz0nTWFrZSBzdXJlIHRvIHJlZ3Jlc3MgYHJldnRxX2Rvd25gIG9uIGByZXZ0cV95b3lfbGFnYC4nKVxudGVzdF9zdHVkZW50X3R5cGVkKCd5ID0gcmV2dHFfZG93bicsbm90X3R5cGVkX21zZz0nTWFrZSBzdXJlIHRvIHNldCBgeSA9IHJldnRxX2Rvd25gIHdpdGhpbiBgYWVzKClgIGZvciB0aGUgYWN0dWFsIGRlY3JlYXNlIGluIHJldmVudWUgc2VyaWVzLicpXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ3kgPSBwcmVkJyxub3RfdHlwZWRfbXNnPSdNYWtlIHN1cmUgdG8gc2V0IGB5ID0gcHJlZGAgd2l0aGluIGBhZXMoKWAgZm9yIHRoZSBwcmVkaWN0ZWQgZGVjcmVhc2UgaW4gcmV2ZW51ZSBzZXJpZXMuJylcbnRlc3Rfc3R1ZGVudF90eXBlZCgndHlwZSA9IFwicmVzcG9uc2VcIicsbm90X3R5cGVkX21zZz0nTWFrZSBzdXJlIHRvIHNldCBgdHlwZSA9IFwicmVzcG9uc2VcImAgd2hlbiB1c2luZyBgcHJlZGljdCgpYC4nKVxudGVzdF9leHByZXNzaW9uX291dHB1dChcImRmJHByZWRcIiwgaW5jb3JyZWN0X21zZz1cIllvdXIgcHJlZGljdGlvbnMgaW4gYGRmJHByZWRgIGRvbid0IHNlZW0gcXVpdGUgcmlnaHQuXCIpXG5cbiMgdGVzdF9zdHVkZW50X3R5cGVkKCd4IDwtIDInLCBub3RfdHlwZWRfbXNnPScnKVxuXG4jIENoZWNrIGlmIGZ1bmN0aW9uIHdhcyB1c2VkIGluIGlucHV0IGNvZGVcbiMgdGVzdF9mdW5jdGlvbignYycsaW5jb3JyZWN0X21zZz0nJykgIFxuXG4jIFJlcXVpcmVzIGFuIG9iamVjdCBgeGAgdG8gaGF2ZSB0aGUgc2FtZSB2YWx1ZSBhcyB0aGUgc29sdXRpb25cbiMgdGVzdF9vYmplY3QoXCJ4XCIsaW5jb3JyZWN0X21zZyA9IFwiXCIsdW5kZWZpbmVkX21zZyA9IFwiXCIpICBcblxuIyBSZXF1aXJlcyBhbiBvbmplY3Qgd2l0aCB0aGUgc2FtZSB2YWx1ZSBvZiBgeGAgaW4gdGhlIHNvbHV0aW9uXG4jIHRlc3RfYW5fb2JqZWN0KFwieFwiLHVuZGVmaW5lZF9tc2c9XCJcIilcblxuIyBDaGVja3MgaWYgb3V0cHV0IG9mIHN0dWRlbnQncyBjb2RlIGNvbnRhaW5zIGdpdmVuIGV2YWx1YXRlZCBleHByZXNzaW9uXG4jIHRlc3Rfb3V0cHV0X2NvbnRhaW5zKFwieFwiLGluY29ycmVjdF9tc2cgPSBcIlwiKVxuXG4jIENoZWNrIGlmIGEgdmVjdG9yIG9mIHByZWRlZmluZWQgb2JqZWN0cyBhcmUgdW5jaGFuZ2VkXG4jIHRlc3RfcHJlZGVmaW5lZF9vYmplY3RzKGMoJ3gnLCd5JyksaW5jb3JyZWN0X21zZz1cIkRvbid0IG9udmVyd3JpdGUgdGhlIHByZWRlZmluZWQgdmFyaWFibGVzXCIpXG5cbiMgQ2hlY2tzIGZvciBhIHJlZ2V4IHBhdHRlcm4gaW4gdHJoZSBvdXRwdXRcbiMgdGVzdF9vdXRwdXRfcmVnZXgocGF0dGVybixmaXhlZD1GLCB0aW1lcz0xLCBpbmNvcnJlY3RfbXNnPScnKVxuXG4jIENhbiBjaGVjayBhbiBhcmJpdHJhcnkgZXhwcmVzc2lvbiBhY3Jvc3MgYm90aCBzb2x1dGlvbiBhbmQgc3R1ZGVudCBjb2RlXG4jdGVzdF9leHByZXNzaW9uX291dHB1dChcInR5cGVvZihjb21wYW55X25hbWUpXCIsIGluY29ycmVjdF9tc2c9XCJEaWQgeW91IHN0b3JlIHRleHR1YWwgZGF0YSBpbiBgY29tcGFueV9uYW1lYD9cIilcblxudGVzdF9lcnJvcigpXG5zdWNjZXNzX21zZyhcIkF3ZXNvbWUhXCIpXG5cbiMgT3RoZXIgZnVuY3Rpb25zIHRvIG5vdGU6XG4jICAgICAtIHRlc3Rfb3IoYSxiKSAtLSBjaGVja3MgaWYgZWl0aGVyIHRlc3QgYSBvciB0ZXN0IGIgcGFzc1xuIyAgICAgLSB0ZXN0X2dncGxvdCgpIC0tIGNhbiBjaGVjayBpZiBwbG90cyBhcmUgY29ycmVjdFxuIyAgICAgLSB0ZXN0X2Z1bmN0aW9uKCkgLS0gY2FuIGFsc28gY2hlY2sgaW5jbHVkZWQgcGFyYW1ldGVyc1xuIyAgICAgLSB0ZXN0X2xvb3AoKSAtLSBjaGVja2luZyBmb3IgYW5kIHdoaWxlIGxvb3BzXG4jICAgICAtIHRlc3RfbGlicmFyeV9mdW5jdGlvbigncGFja2FnZScsIG5vdF9jYWxsZWRfbXNnPScnLGluY29ycmVjdF9tc2c9JycpXG4jICAgICAtIHRlc3RfaWZfZWxzZSgpIC0tIGNoZWNraW5nIGlmIHN0YXRlbWVudHNcbiMgICAgIC0gdGVzdF9leHByZXNzaW9uX2Vycm9yKCkgLS0gY2FuIGNoZWNrIGlmIGZ1bmN0aW9ucyBhcmUgcHJvcGVybHkgZGVmaW5lZFxuIyAgICAgLSB0ZXN0X29wZXJhdG9yKCdvcGVyYXRvcicsKSwgbm90X2NhbGxlZF9tc2c9JycsaW5jb3JyZWN0X21zZz0nJylcbiMgICAgIC0gdGVzdF9mdW5jdGlvbl9kZWZpbml0aW9uKCkgLS0gcmlnb3JvdXNseSBjaGVjayBkZWZpbmVkIGZ1bmN0aW9uXG4jICAgICAtIHRlc3RfZGF0YV9mcmFtZSgpIC0tIGNoZWNrIGlmIGRhdGFmcmFtZSBbY29sdW1uc10gYXJlIGVxdWl2YWxlbnRcbiMgICAgIC0gdGVzdF9mdW5jdGlvbl9yZXN1bHQsIHRlc3RfZXhwcmVzc2lvbl9yZXN1bHQifQ==

Exercise 3: ROC curve and AUC

Note: This isn’t using the yardstick library directly, but instead implements a small subset of the functionality of it.

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6ImxpYnJhcnkoZHBseXIpICMgbG9hZHMgZHBseXIgZm9yIGRhdGEgbWFuaXB1bGF0aW9uXG5saWJyYXJ5KGdncGxvdDIpXG5kZiA8LSBkYXRhLmZyYW1lKHRpYz1yZXAoXCJXTVRcIiwxOTQpLFxuICAgICAgICAgICAgICAgICBmeWVhcnE9YygxOTcwLDE5NzAsMTk3MCwxOTcwLDE5NzEsMTk3MSwxOTcxLDE5NzEsMTk3MiwxOTcyLDE5NzIsMTk3MiwxOTczLDE5NzMsMTk3MywxOTczLDE5NzQsMTk3NCwxOTc0LDE5NzQsMTk3NSwxOTc1LDE5NzUsMTk3NSwxOTc2LDE5NzYsMTk3NiwxOTc2LDE5NzcsMTk3NywxOTc3LDE5NzcsMTk3OCwxOTc4LDE5NzgsMTk3OCwxOTc5LDE5NzksMTk3OSwxOTc5LDE5ODAsMTk4MCwxOTgwLDE5ODAsMTk4MSwxOTgxLDE5ODEsMTk4MSwxOTgyLDE5ODIsMTk4MiwxOTgyLDE5ODMsMTk4MywxOTgzLDE5ODMsMTk4NCwxOTg0LDE5ODQsMTk4NCwxOTg1LDE5ODUsMTk4NSwxOTg1LDE5ODYsMTk4NiwxOTg2LDE5ODYsMTk4NywxOTg3LDE5ODcsMTk4NywxOTg4LDE5ODgsMTk4OCwxOTg4LDE5ODksMTk4OSwxOTg5LDE5ODksMTk5MCwxOTkwLDE5OTAsMTk5MCwxOTkxLDE5OTEsMTk5MSwxOTkxLDE5OTIsMTk5MiwxOTkyLDE5OTIsMTk5MywxOTkzLDE5OTMsMTk5MywxOTk0LDE5OTQsMTk5NCwxOTk0LDE5OTUsMTk5NSwxOTk1LDE5OTUsMTk5NiwxOTk2LDE5OTYsMTk5NiwxOTk3LDE5OTcsMTk5NywxOTk3LDE5OTgsMTk5OCwxOTk4LDE5OTgsMTk5OSwxOTk5LDE5OTksMTk5OSwyMDAwLDIwMDAsMjAwMCwyMDAwLDIwMDEsMjAwMSwyMDAxLDIwMDEsMjAwMiwyMDAyLDIwMDIsMjAwMiwyMDAzLDIwMDMsMjAwMywyMDAzLDIwMDQsMjAwNCwyMDA0LDIwMDQsMjAwNSwyMDA1LDIwMDUsMjAwNSwyMDA2LDIwMDYsMjAwNiwyMDA2LDIwMDcsMjAwNywyMDA3LDIwMDcsMjAwOCwyMDA4LDIwMDgsMjAwOCwyMDA5LDIwMDksMjAwOSwyMDA5LDIwMTAsMjAxMCwyMDEwLDIwMTAsMjAxMSwyMDExLDIwMTEsMjAxMSwyMDEyLDIwMTIsMjAxMiwyMDEyLDIwMTMsMjAxMywyMDEzLDIwMTMsMjAxNCwyMDE0LDIwMTQsMjAxNCwyMDE1LDIwMTUsMjAxNSwyMDE1LDIwMTYsMjAxNiwyMDE2LDIwMTYsMjAxNywyMDE3LDIwMTcsMjAxNywyMDE4LDIwMTgpLFxuICAgICAgICAgICAgICAgICBmcXRyPWMoMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyLDMsNCwxLDIsMyw0LDEsMiwzLDQsMSwyKSxcbiAgICAgICAgICAgICAgICAgcmV2dHE9Yyg3LjI2Nyw5LjkzOSwxMS40NywxNS42MSwxMi43NTgsMTguMjA3LDE5LjIwNSwyNy44NDQsMjEuMDExLDI5LjkxOCwzMi42ODMsNDEuMjc3LDMyLjczNywzNy42NTUsNDIuMDk4LDU1LjA3MSw0Ni4zNTUsNTYuMTc3LDU5LjQ2Miw3NC4yMTUsNjQuNjUsNzguODUyLDg2LjEwOCwxMTAuNzIxLDkzLjk3MSwxMTUuMTg2LDEyMS44NzUsMTQ3Ljc3NSwxMjIuNjU1LDE1Mi4zODEsMTcyLjYzNywyMzAuNzgzLDE2Ny4yNzQsMjA0LjA4NSwyMjEuOTgyLDMwNi45NTcsMjM4LjU2OSwyOTEuNjg1LDMwMC43NDcsNDE3LjE3NSwzMTQuNTc1LDM3Mi41OTgsNDAxLjg0Miw1NTQuMTg0LDQzNC4yNTEsNTMzLjY1Myw2MzQuNDc2LDg0Mi42MTcsNjY1LjIyOSw3ODguNzA1LDgyNy43NjcsMTA5NC41NTEsODU0LjczNCwxMDk4Ljk0MiwxMTY2LjU5NCwxNTQ2LjYzOSwxMjM0Ljc2OCwxNTA4LjUzMywxNTgzLjU3MywyMDczLjk4MywxNjU1LjY2MSwxOTM0LjM1OCwyMDg3LjUzMywyNzczLjkzNiwyMzQ0LjI1MywyNzcwLjI3NiwyOTQ5LjAwOCwzODQ1LjU1NywzMjExLjc2MSwzNzMyLjIzNyw0MDE1LjAxNyw1MDAwLjIzOCw0MjgyLjAxMiw0ODQ1LjEzMyw0OTkxLjExMyw2NTMwLjczNCw1MzczLjI1OCw2MDQ2LjQxLDYyODMuNDk2LDgxMDcuNDg0LDY3NjguMTkxLDc1NDMuNTA4LDc5MzAuOTQ5LDEwMzU4Ljg5OCw5MjgwLjU2NiwxMDMzOS44OTgsMTA2MjcuNSwxMzYzOC43OTcsMTE2NDkuMzk4LDEzMDI4LjM5OCwxMzY4My43OTcsMTcxMjIsMTM5MjAuMzk4LDE2MjM2LjUsMTY4MjYuODk4LDIwMzYwLjUsMTc2ODYuMDk4LDE5OTQyLjI5NywyMDQxNy42OTksMjQ0NDcuNzk3LDIwNDQwLDIyNzIzLDIyOTE0LDI3NTUwLDIyNzcyLDI1NTg3LDI1NjQ0LDMwODU2LDI1NDA5LDI4Mzg2LDI4Nzc3LDM1Mzg2LDI5ODE5LDMzNTIxLDMzNTA5LDQwNzg1LDM1MTI5LDM4OTEzLDQwODk5LDUxODY4LDQzNDQ3LDQ2NTg4LDQ2MTgxLDU3MDc5LDQ4NTY1LDUzMTg3LDUzMTg1LDY0NzM1LDUyMTI2LDU2NzgxLDU1NzY1LDY2OTA1LDU3MjI0LDYzMjMxLDYzMDM2LDc1MTkwLDY1NDM0LDcwNTE2LDY5MjgyLDgyOTAwLDcwNzU1LDc2Njk3LDc1Mzk3LDg5MjUyLDc5Njc1LDg1NDMwLDg0NDY3LDk5MDc4LDg2Mzc4LDkyODI3LDkxNzk0LDEwNzI4OSw5NDk0MCwxMDIzNDIsOTgzNDUsMTA4NzQ3LDk0MjE0LDEwMDg3Niw5OTM3MywxMTM2MjIsOTk4MTEsMTAzNzI2LDEwMTk1MiwxMTYzNjAsMTA0MTg5LDEwOTM2NiwxMTAyMjYsMTIzMTY5LDExMzAxMCwxMTQwNjUsMTEzODAwLDEyNzc3NiwxMTQwNzAsMTE2ODMwLDExNTY4OCwxMjk3MDYsMTE0OTYwLDEyMDEyNSwxMTkwMDEsMTMxNTY1LDExNDgyNiwxMjAyMjksMTE3NDA4LDEyOTY2NywxMTU5MDQsMTIwMzE5LDExODE3OSwxMzA5MzYsMTE3NTQyLDEyMjk2OCwxMjMxNzksMTM2MjY3LDEyMjY5MCwxMjgwMjgpLFxuICAgICAgICAgICAgICAgICByZWNlc3Npb249YygxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAxLCAxLCAxLCAxLCAxLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwKSlcbmRmIDwtIGRmICU+JSBtdXRhdGUocmV2dHFfeW95ID0gKHJldnRxIC0gbGFnKHJldnRxLCA0KSkgLyBsYWcocmV2dHEsIDQpLFxuICAgICAgICAgICAgICAgICAgICByZWNlc3Npb25fbGFnID0gbGFnKHJlY2Vzc2lvbiksXG4gICAgICAgICAgICAgICAgICAgIHJldnRxX2Rvd24gPSBpZmVsc2UocmV2dHFfeW95PDAsMSwwKSxcbiAgICAgICAgICAgICAgICAgICAgcmV2dHFfeW95X2xhZyA9IGxhZyhyZXZ0cV95b3kpLFxuICAgICAgICAgICAgICAgICAgICBkYXRlID0gZnllYXJxICsgKGZxdHItMSkvNClcbm1vZDEgPC0gZ2xtKHJldnRxX2Rvd24gfiByZXZ0cV95b3lfbGFnLCBkYXRhPWRmLCBmYW1pbHk9Ymlub21pYWwpXG5kZiRwcmVkIDwtIHByZWRpY3QobW9kMSwgZGYsIHR5cGU9XCJyZXNwb25zZVwiKVxuXG5hdXJvYyA8LSBmdW5jdGlvbihzY29yZSwgYm9vbCkge1xuICBuMSA8LSBzdW0oIWJvb2wpXG4gIG4yIDwtIHN1bShib29sKVxuICBVICA8LSBzdW0ocmFuayhzY29yZSlbIWJvb2xdKSAtIG4xICogKG4xICsgMSkgLyAyXG4gIHJldHVybigxIC0gVSAvIG4xIC8gbjIpXG59XG5cbmRmIDwtIGRmICU+JSBmaWx0ZXIoIWlzLm5hKHJldnRxX2Rvd24pLCAhaXMubmEocHJlZCkpXG5cbnJvY19hdWMgPC0gZnVuY3Rpb24oZGYsIHRydXRoPU5VTEwsIGVzdGltYXRlPU5VTEwsIGV2ZW50X2xldmVsPSdmaXJzdCcpIHtcbiAgdHJ1dGggPC0gcmxhbmc6OmVucXVvKHRydXRoKVxuICBlc3RpbWF0ZSA8LSBybGFuZzo6ZW5xdW8oZXN0aW1hdGUpXG4gIHRydWVfWSA9IGRmICU+JSBzZWxlY3QocmV2dHFfZG93bilcbiAgcHJvYnMgPSBkZiAlPiUgc2VsZWN0KHByZWQpXG4gIGF1YyA9IGF1cm9jKHByb2JzLCB0cnVlX1kpXG4gIGlmIChldmVudF9sZXZlbD09J2ZpcnN0Jykge1xuICAgIGF1YyA9IDEtYXVjXG4gIH1cbiAgZGF0YS5mcmFtZSgubWV0cmljPWMoXCJyb2NfYXVjXCIpLCAuZXN0aW1hdG9yPWMoXCJiaW5hcnlcIiksIC5lc3RpbWF0ZT1jKGF1YykpXG59Iiwic2FtcGxlIjoiIyBkZiwgaW5jbHVkaW5nIHByZWQsIGlzIGxvYWRlZFxuXG4jIENhbGN1bGF0ZSBST0MgQVVDICgnYXVjJykgdXNpbmcgeWFyZHN0aWNrJ3Mgcm9jX2F1YygpXG5BVUMgPC0gZGYgJT4lIHJvY19hdWModHJ1dGg9LCBlc3RpbWF0ZT0sIGV2ZW50X2xldmVsPSlcblxuIyBEaXNwbGF5IHRoZSBBVUMgdmFsdWUgLS0gdGhpcyBsaW5lIGlzIGFscmVhZHkgZG9uZVxuQVVDXG4jRU5EIiwic29sdXRpb24iOiIjIGRmLCBpbmNsdWRpbmcgcHJlZCwgaXMgbG9hZGVkXG5cbiMgQ2FsY3VsYXRlIFJPQyBBVUMgKCdhdWMnKSB1c2luZyB5YXJkc3RpY2sncyByb2NfYXVjKClcbkFVQyA8LSBkZiAlPiUgcm9jX2F1Yyh0cnV0aD1yZXZ0cV9kb3duLCBlc3RpbWF0ZT1wcmVkLCBldmVudF9sZXZlbD0nc2Vjb25kJylcblxuIyBEaXNwbGF5IHRoZSBBVUMgdmFsdWUgLS0gdGhpcyBsaW5lIGlzIGFscmVhZHkgZG9uZVxuQVVDXG4jRU5EIiwic2N0IjoiIyBUZW1wbGF0ZSBiYXNlZCBvbiBodHRwczovL3d3dy5yZG9jdW1lbnRhdGlvbi5vcmcvcGFja2FnZXMvdGVzdHdoYXQvdmVyc2lvbnMvNC4xLjFcbiMgQ2hlY2sgaWYgc29tZXRoaW5nIGlzIGV4cGxpY2l0bHkgdHlwZWRcblxudGVzdF9zdHVkZW50X3R5cGVkKCdzZWNvbmQnLG5vdF90eXBlZF9tc2c9J01ha2Ugc3VyZSB0byBzZXQgZXZlbnRfbGV2ZWwgYXMgXFwnc2Vjb25kXFwnLicpXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ3JldnRxX2Rvd24nLG5vdF90eXBlZF9tc2c9J01ha2Ugc3VyZSB0byBzZXQgdHJ1dGggYXMgYHJldnRxX2Rvd25gLicpXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ3ByZWQnLG5vdF90eXBlZF9tc2c9J01ha2Ugc3VyZSB0byBzZXQgZXN0aW1hdGUgYXMgYHByZWRgLicpXG50ZXN0X2V4cHJlc3Npb25fb3V0cHV0KFwiQVVDXCIsIGluY29ycmVjdF9tc2c9XCJZb3VyIEFVQyBvdXRwdXQgaXNuJ3QgcXVpdGUgcmlnaHQuXCIpXG5cbiN0ZXN0X3N0dWRlbnRfdHlwZWQoJ3JldnRxX2Rvd24gfiByZWNlc3Npb25fbGFnJyxub3RfdHlwZWRfbXNnPSdNYWtlIHN1cmUgdG8gcmVncmVzcyBgcmV2dHFfZG93bmAgb24gYHJlY2Vzc2lvbl9sYWdgLicpXG5cblxuIyB0ZXN0X3N0dWRlbnRfdHlwZWQoJ3ggPC0gMicsIG5vdF90eXBlZF9tc2c9JycpXG5cbiMgQ2hlY2sgaWYgZnVuY3Rpb24gd2FzIHVzZWQgaW4gaW5wdXQgY29kZVxuIyB0ZXN0X2Z1bmN0aW9uKCdjJyxpbmNvcnJlY3RfbXNnPScnKSAgXG5cbiMgUmVxdWlyZXMgYW4gb2JqZWN0IGB4YCB0byBoYXZlIHRoZSBzYW1lIHZhbHVlIGFzIHRoZSBzb2x1dGlvblxuIyB0ZXN0X29iamVjdChcInhcIixpbmNvcnJlY3RfbXNnID0gXCJcIix1bmRlZmluZWRfbXNnID0gXCJcIikgIFxuXG4jIFJlcXVpcmVzIGFuIG9uamVjdCB3aXRoIHRoZSBzYW1lIHZhbHVlIG9mIGB4YCBpbiB0aGUgc29sdXRpb25cbiMgdGVzdF9hbl9vYmplY3QoXCJ4XCIsdW5kZWZpbmVkX21zZz1cIlwiKVxuXG4jIENoZWNrcyBpZiBvdXRwdXQgb2Ygc3R1ZGVudCdzIGNvZGUgY29udGFpbnMgZ2l2ZW4gZXZhbHVhdGVkIGV4cHJlc3Npb25cbiMgdGVzdF9vdXRwdXRfY29udGFpbnMoXCJ4XCIsaW5jb3JyZWN0X21zZyA9IFwiXCIpXG5cbiMgQ2hlY2sgaWYgYSB2ZWN0b3Igb2YgcHJlZGVmaW5lZCBvYmplY3RzIGFyZSB1bmNoYW5nZWRcbiMgdGVzdF9wcmVkZWZpbmVkX29iamVjdHMoYygneCcsJ3knKSxpbmNvcnJlY3RfbXNnPVwiRG9uJ3Qgb252ZXJ3cml0ZSB0aGUgcHJlZGVmaW5lZCB2YXJpYWJsZXNcIilcblxuIyBDaGVja3MgZm9yIGEgcmVnZXggcGF0dGVybiBpbiB0cmhlIG91dHB1dFxuIyB0ZXN0X291dHB1dF9yZWdleChwYXR0ZXJuLGZpeGVkPUYsIHRpbWVzPTEsIGluY29ycmVjdF9tc2c9JycpXG5cbiMgQ2FuIGNoZWNrIGFuIGFyYml0cmFyeSBleHByZXNzaW9uIGFjcm9zcyBib3RoIHNvbHV0aW9uIGFuZCBzdHVkZW50IGNvZGVcbiN0ZXN0X2V4cHJlc3Npb25fb3V0cHV0KFwidHlwZW9mKGNvbXBhbnlfbmFtZSlcIiwgaW5jb3JyZWN0X21zZz1cIkRpZCB5b3Ugc3RvcmUgdGV4dHVhbCBkYXRhIGluIGBjb21wYW55X25hbWVgP1wiKVxuXG50ZXN0X2Vycm9yKClcbnN1Y2Nlc3NfbXNnKFwiQXdlc29tZSFcIilcblxuIyBPdGhlciBmdW5jdGlvbnMgdG8gbm90ZTpcbiMgICAgIC0gdGVzdF9vcihhLGIpIC0tIGNoZWNrcyBpZiBlaXRoZXIgdGVzdCBhIG9yIHRlc3QgYiBwYXNzXG4jICAgICAtIHRlc3RfZ2dwbG90KCkgLS0gY2FuIGNoZWNrIGlmIHBsb3RzIGFyZSBjb3JyZWN0XG4jICAgICAtIHRlc3RfZnVuY3Rpb24oKSAtLSBjYW4gYWxzbyBjaGVjayBpbmNsdWRlZCBwYXJhbWV0ZXJzXG4jICAgICAtIHRlc3RfbG9vcCgpIC0tIGNoZWNraW5nIGZvciBhbmQgd2hpbGUgbG9vcHNcbiMgICAgIC0gdGVzdF9saWJyYXJ5X2Z1bmN0aW9uKCdwYWNrYWdlJywgbm90X2NhbGxlZF9tc2c9JycsaW5jb3JyZWN0X21zZz0nJylcbiMgICAgIC0gdGVzdF9pZl9lbHNlKCkgLS0gY2hlY2tpbmcgaWYgc3RhdGVtZW50c1xuIyAgICAgLSB0ZXN0X2V4cHJlc3Npb25fZXJyb3IoKSAtLSBjYW4gY2hlY2sgaWYgZnVuY3Rpb25zIGFyZSBwcm9wZXJseSBkZWZpbmVkXG4jICAgICAtIHRlc3Rfb3BlcmF0b3IoJ29wZXJhdG9yJywpLCBub3RfY2FsbGVkX21zZz0nJyxpbmNvcnJlY3RfbXNnPScnKVxuIyAgICAgLSB0ZXN0X2Z1bmN0aW9uX2RlZmluaXRpb24oKSAtLSByaWdvcm91c2x5IGNoZWNrIGRlZmluZWQgZnVuY3Rpb25cbiMgICAgIC0gdGVzdF9kYXRhX2ZyYW1lKCkgLS0gY2hlY2sgaWYgZGF0YWZyYW1lIFtjb2x1bW5zXSBhcmUgZXF1aXZhbGVudFxuIyAgICAgLSB0ZXN0X2Z1bmN0aW9uX3Jlc3VsdCwgdGVzdF9leHByZXNzaW9uX3Jlc3VsdCJ9

Exercise 4: Plotting a ROC curve

As the autoplot() method is unsupported here, code is given below to visualize the ROC curve for the model you have built.

eyJsYW5ndWFnZSI6InIiLCJwcmVfZXhlcmNpc2VfY29kZSI6ImxpYnJhcnkoZ2dwbG90MilcbmRmX1JPQyA8LSBkYXRhLmZyYW1lKEZhbHNlUG9zaXRpdmU9YygwLCAwLjAwNTQzNDc4MjYwODY5NTY1LCAwLjAxMDg2OTU2NTIxNzM5MTMsIDAuMDEwODY5NTY1MjE3MzkxMywgMC4wMTA4Njk1NjUyMTczOTEzLCAwLjAxNjMwNDM0NzgyNjA4NywgMC4wMjE3MzkxMzA0MzQ3ODI2LCAwLjAyMTczOTEzMDQzNDc4MjYsIDAuMDI3MTczOTEzMDQzNDc4MywgMC4wMzI2MDg2OTU2NTIxNzM5LCAwLjAzODA0MzQ3ODI2MDg2OTYsIDAuMDQzNDc4MjYwODY5NTY1MiwgMC4wNDg5MTMwNDM0NzgyNjA5LCAwLjA1NDM0NzgyNjA4Njk1NjUsIDAuMDU0MzQ3ODI2MDg2OTU2NSwgMC4wNTk3ODI2MDg2OTU2NTIyLCAwLjA1OTc4MjYwODY5NTY1MjIsIDAuMDY1MjE3MzkxMzA0MzQ3OCwgMC4wNzA2NTIxNzM5MTMwNDM1LCAwLjA3NjA4Njk1NjUyMTczOTEsIDAuMDgxNTIxNzM5MTMwNDM0OCwgMC4wODY5NTY1MjE3MzkxMzA0LCAwLjA5MjM5MTMwNDM0NzgyNjEsIDAuMDk3ODI2MDg2OTU2NTIxNywgMC4xMDMyNjA4Njk1NjUyMTcsIDAuMTA4Njk1NjUyMTczOTEzLCAwLjExNDEzMDQzNDc4MjYwOSwgMC4xMTk1NjUyMTczOTEzMDQsIDAuMTI1LCAwLjEzMDQzNDc4MjYwODY5NiwgMC4xMzU4Njk1NjUyMTczOTEsIDAuMTQxMzA0MzQ3ODI2MDg3LCAwLjE0NjczOTEzMDQzNDc4MywgMC4xNTIxNzM5MTMwNDM0NzgsIDAuMTU3NjA4Njk1NjUyMTc0LCAwLjE2MzA0MzQ3ODI2MDg3LCAwLjE2ODQ3ODI2MDg2OTU2NSwgMC4xNzM5MTMwNDM0NzgyNjEsIDAuMTc5MzQ3ODI2MDg2OTU3LCAwLjE4NDc4MjYwODY5NTY1MiwgMC4xOTAyMTczOTEzMDQzNDgsIDAuMTk1NjUyMTczOTEzMDQzLCAwLjIwMTA4Njk1NjUyMTczOSwgMC4yMDY1MjE3MzkxMzA0MzUsIDAuMjExOTU2NTIxNzM5MTMsIDAuMjE3MzkxMzA0MzQ3ODI2LCAwLjIyMjgyNjA4Njk1NjUyMiwgMC4yMjgyNjA4Njk1NjUyMTcsIDAuMjMzNjk1NjUyMTczOTEzLCAwLjIzOTEzMDQzNDc4MjYwOSwgMC4yNDQ1NjUyMTczOTEzMDQsIDAuMjUsIDAuMjU1NDM0NzgyNjA4Njk2LCAwLjI2MDg2OTU2NTIxNzM5MSwgMC4yNjYzMDQzNDc4MjYwODcsIDAuMjcxNzM5MTMwNDM0NzgzLCAwLjI3NzE3MzkxMzA0MzQ3OCwgMC4yODI2MDg2OTU2NTIxNzQsIDAuMjg4MDQzNDc4MjYwODcsIDAuMjkzNDc4MjYwODY5NTY1LCAwLjI5ODkxMzA0MzQ3ODI2MSwgMC4zMDQzNDc4MjYwODY5NTcsIDAuMzA5NzgyNjA4Njk1NjUyLCAwLjMxNTIxNzM5MTMwNDM0OCwgMC4zMjA2NTIxNzM5MTMwNDMsIDAuMzI2MDg2OTU2NTIxNzM5LCAwLjMzMTUyMTczOTEzMDQzNSwgMC4zMzY5NTY1MjE3MzkxMywgMC4zNDIzOTEzMDQzNDc4MjYsIDAuMzQ3ODI2MDg2OTU2NTIyLCAwLjM1MzI2MDg2OTU2NTIxNywgMC4zNTg2OTU2NTIxNzM5MTMsIDAuMzY0MTMwNDM0NzgyNjA5LCAwLjM2OTU2NTIxNzM5MTMwNCwgMC4zNzUsIDAuMzgwNDM0NzgyNjA4Njk2LCAwLjM4NTg2OTU2NTIxNzM5MSwgMC4zOTEzMDQzNDc4MjYwODcsIDAuMzk2NzM5MTMwNDM0NzgzLCAwLjQwMjE3MzkxMzA0MzQ3OCwgMC40MDc2MDg2OTU2NTIxNzQsIDAuNDEzMDQzNDc4MjYwODcsIDAuNDE4NDc4MjYwODY5NTY1LCAwLjQyMzkxMzA0MzQ3ODI2MSwgMC40MjkzNDc4MjYwODY5NTcsIDAuNDM0NzgyNjA4Njk1NjUyLCAwLjQ0MDIxNzM5MTMwNDM0OCwgMC40NDU2NTIxNzM5MTMwNDMsIDAuNDUxMDg2OTU2NTIxNzM5LCAwLjQ1NjUyMTczOTEzMDQzNSwgMC40NjE5NTY1MjE3MzkxMywgMC40NjczOTEzMDQzNDc4MjYsIDAuNDcyODI2MDg2OTU2NTIyLCAwLjQ3ODI2MDg2OTU2NTIxNywgMC40ODM2OTU2NTIxNzM5MTMsIDAuNDg5MTMwNDM0NzgyNjA5LCAwLjQ5NDU2NTIxNzM5MTMwNCwgMC41LCAwLjUwNTQzNDc4MjYwODY5NiwgMC41MTA4Njk1NjUyMTczOTEsIDAuNTE2MzA0MzQ3ODI2MDg3LCAwLjUyMTczOTEzMDQzNDc4MywgMC41MjcxNzM5MTMwNDM0NzgsIDAuNTMyNjA4Njk1NjUyMTc0LCAwLjUzODA0MzQ3ODI2MDg3LCAwLjU0MzQ3ODI2MDg2OTU2NSwgMC41NDg5MTMwNDM0NzgyNjEsIDAuNTU0MzQ3ODI2MDg2OTU3LCAwLjU1OTc4MjYwODY5NTY1MiwgMC41NjUyMTczOTEzMDQzNDgsIDAuNTcwNjUyMTczOTEzMDQzLCAwLjU3NjA4Njk1NjUyMTczOSwgMC41ODE1MjE3MzkxMzA0MzUsIDAuNTg2OTU2NTIxNzM5MTMsIDAuNTkyMzkxMzA0MzQ3ODI2LCAwLjU5NzgyNjA4Njk1NjUyMiwgMC42MDMyNjA4Njk1NjUyMTcsIDAuNjA4Njk1NjUyMTczOTEzLCAwLjYxNDEzMDQzNDc4MjYwOSwgMC42MTk1NjUyMTczOTEzMDQsIDAuNjI1LCAwLjYzMDQzNDc4MjYwODY5NiwgMC42MzU4Njk1NjUyMTczOTEsIDAuNjQxMzA0MzQ3ODI2MDg3LCAwLjY0NjczOTEzMDQzNDc4MywgMC42NTIxNzM5MTMwNDM0NzgsIDAuNjU3NjA4Njk1NjUyMTc0LCAwLjY2MzA0MzQ3ODI2MDg3LCAwLjY2ODQ3ODI2MDg2OTU2NSwgMC42NzM5MTMwNDM0NzgyNjEsIDAuNjc5MzQ3ODI2MDg2OTU3LCAwLjY4NDc4MjYwODY5NTY1MiwgMC42OTAyMTczOTEzMDQzNDgsIDAuNjk1NjUyMTczOTEzMDQzLCAwLjcwMTA4Njk1NjUyMTczOSwgMC43MDY1MjE3MzkxMzA0MzUsIDAuNzExOTU2NTIxNzM5MTMsIDAuNzE3MzkxMzA0MzQ3ODI2LCAwLjcyMjgyNjA4Njk1NjUyMiwgMC43MjgyNjA4Njk1NjUyMTcsIDAuNzMzNjk1NjUyMTczOTEzLCAwLjczOTEzMDQzNDc4MjYwOSwgMC43NDQ1NjUyMTczOTEzMDQsIDAuNzUsIDAuNzU1NDM0NzgyNjA4Njk2LCAwLjc2MDg2OTU2NTIxNzM5MSwgMC43NjYzMDQzNDc4MjYwODcsIDAuNzcxNzM5MTMwNDM0NzgzLCAwLjc3NzE3MzkxMzA0MzQ3OCwgMSksXG4gICAgICAgICAgICAgICAgICAgICBUcnVlUG9zaXRpdmU9YygwLCAwLCAwLCAwLjIsIDAuNCwgMC40LCAwLjQsIDAuNiwgMC42LCAwLjYsIDAuNiwgMC42LCAwLjYsIDAuNiwgMC44LCAwLjgsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEsIDEpKSIsInNhbXBsZSI6IiMgUGxvdCB0aGUgUk9DIGN1cnZlIC0tIHRoaXMgY29kZSBpcyBkb25lIGZvciB5b3VcbmdncGxvdChkYXRhPWRmX1JPQywgYWVzKHg9RmFsc2VQb3NpdGl2ZSwgeT1UcnVlUG9zaXRpdmUpKSArXG4gIGdlb21fbGluZSgpICtcbiAgZ2VvbV9hYmxpbmUoc2xvcGU9MSlcblxuI0VORCIsInNvbHV0aW9uIjoiIyBQbG90IHRoZSBST0MgY3VydmUgLS0gdGhpcyBjb2RlIGlzIGRvbmUgZm9yIHlvdVxuZ2dwbG90KGRhdGE9ZGZfUk9DLCBhZXMoeD1GYWxzZVBvc2l0aXZlLCB5PVRydWVQb3NpdGl2ZSkpICtcbiAgZ2VvbV9saW5lKCkgK1xuICBnZW9tX2FibGluZShzbG9wZT0xKVxuXG4jRU5EIiwic2N0IjoiIyBUZW1wbGF0ZSBiYXNlZCBvbiBodHRwczovL3d3dy5yZG9jdW1lbnRhdGlvbi5vcmcvcGFja2FnZXMvdGVzdHdoYXQvdmVyc2lvbnMvNC4xLjFcbiMgQ2hlY2sgaWYgc29tZXRoaW5nIGlzIGV4cGxpY2l0bHkgdHlwZWRcblxudGVzdF9zdHVkZW50X3R5cGVkKCdkYXRhPWRmX1JPQycsbm90X3R5cGVkX21zZz0nTWFrZSBzdXJlIHRvIHVzZSBgZGZfUk9DYCBhcyB0aGUgZGF0YSBmcmFtZS4nKVxudGVzdF9zdHVkZW50X3R5cGVkKCd4PUZhbHNlUG9zaXRpdmUnLG5vdF90eXBlZF9tc2c9J01ha2Ugc3VyZSB0byBwdXQgYEZhbHNlUG9zaXRpdmVgIGFzIHRoZSBkYXRhIHBsb3R0ZWQgb24gdGhlIHgtYXhpcycpXG50ZXN0X3N0dWRlbnRfdHlwZWQoJ3k9RmFsc2VQb3NpdGl2ZScsbm90X3R5cGVkX21zZz0nTWFrZSBzdXJlIHRvIHB1dCBgVHJ1ZVBvc2l0aXZlYCBhcyB0aGUgZGF0YSBwbG90dGVkIG9uIHRoZSB5LWF4aXMnKVxuXG4jIHRlc3Rfc3R1ZGVudF90eXBlZCgneCA8LSAyJywgbm90X3R5cGVkX21zZz0nJylcblxuIyBDaGVjayBpZiBmdW5jdGlvbiB3YXMgdXNlZCBpbiBpbnB1dCBjb2RlXG4jIHRlc3RfZnVuY3Rpb24oJ2MnLGluY29ycmVjdF9tc2c9JycpICBcblxuIyBSZXF1aXJlcyBhbiBvYmplY3QgYHhgIHRvIGhhdmUgdGhlIHNhbWUgdmFsdWUgYXMgdGhlIHNvbHV0aW9uXG4jIHRlc3Rfb2JqZWN0KFwieFwiLGluY29ycmVjdF9tc2cgPSBcIlwiLHVuZGVmaW5lZF9tc2cgPSBcIlwiKSAgXG5cbiMgUmVxdWlyZXMgYW4gb25qZWN0IHdpdGggdGhlIHNhbWUgdmFsdWUgb2YgYHhgIGluIHRoZSBzb2x1dGlvblxuIyB0ZXN0X2FuX29iamVjdChcInhcIix1bmRlZmluZWRfbXNnPVwiXCIpXG5cbiMgQ2hlY2tzIGlmIG91dHB1dCBvZiBzdHVkZW50J3MgY29kZSBjb250YWlucyBnaXZlbiBldmFsdWF0ZWQgZXhwcmVzc2lvblxuIyB0ZXN0X291dHB1dF9jb250YWlucyhcInhcIixpbmNvcnJlY3RfbXNnID0gXCJcIilcblxuIyBDaGVjayBpZiBhIHZlY3RvciBvZiBwcmVkZWZpbmVkIG9iamVjdHMgYXJlIHVuY2hhbmdlZFxuIyB0ZXN0X3ByZWRlZmluZWRfb2JqZWN0cyhjKCd4JywneScpLGluY29ycmVjdF9tc2c9XCJEb24ndCBvbnZlcndyaXRlIHRoZSBwcmVkZWZpbmVkIHZhcmlhYmxlc1wiKVxuXG4jIENoZWNrcyBmb3IgYSByZWdleCBwYXR0ZXJuIGluIHRyaGUgb3V0cHV0XG4jIHRlc3Rfb3V0cHV0X3JlZ2V4KHBhdHRlcm4sZml4ZWQ9RiwgdGltZXM9MSwgaW5jb3JyZWN0X21zZz0nJylcblxuIyBDYW4gY2hlY2sgYW4gYXJiaXRyYXJ5IGV4cHJlc3Npb24gYWNyb3NzIGJvdGggc29sdXRpb24gYW5kIHN0dWRlbnQgY29kZVxuI3Rlc3RfZXhwcmVzc2lvbl9vdXRwdXQoXCJ0eXBlb2YoY29tcGFueV9uYW1lKVwiLCBpbmNvcnJlY3RfbXNnPVwiRGlkIHlvdSBzdG9yZSB0ZXh0dWFsIGRhdGEgaW4gYGNvbXBhbnlfbmFtZWA/XCIpXG5cbnRlc3RfZXJyb3IoKVxuc3VjY2Vzc19tc2coXCJBd2Vzb21lIVwiKVxuXG4jIE90aGVyIGZ1bmN0aW9ucyB0byBub3RlOlxuIyAgICAgLSB0ZXN0X29yKGEsYikgLS0gY2hlY2tzIGlmIGVpdGhlciB0ZXN0IGEgb3IgdGVzdCBiIHBhc3NcbiMgICAgIC0gdGVzdF9nZ3Bsb3QoKSAtLSBjYW4gY2hlY2sgaWYgcGxvdHMgYXJlIGNvcnJlY3RcbiMgICAgIC0gdGVzdF9mdW5jdGlvbigpIC0tIGNhbiBhbHNvIGNoZWNrIGluY2x1ZGVkIHBhcmFtZXRlcnNcbiMgICAgIC0gdGVzdF9sb29wKCkgLS0gY2hlY2tpbmcgZm9yIGFuZCB3aGlsZSBsb29wc1xuIyAgICAgLSB0ZXN0X2xpYnJhcnlfZnVuY3Rpb24oJ3BhY2thZ2UnLCBub3RfY2FsbGVkX21zZz0nJyxpbmNvcnJlY3RfbXNnPScnKVxuIyAgICAgLSB0ZXN0X2lmX2Vsc2UoKSAtLSBjaGVja2luZyBpZiBzdGF0ZW1lbnRzXG4jICAgICAtIHRlc3RfZXhwcmVzc2lvbl9lcnJvcigpIC0tIGNhbiBjaGVjayBpZiBmdW5jdGlvbnMgYXJlIHByb3Blcmx5IGRlZmluZWRcbiMgICAgIC0gdGVzdF9vcGVyYXRvcignb3BlcmF0b3InLCksIG5vdF9jYWxsZWRfbXNnPScnLGluY29ycmVjdF9tc2c9JycpXG4jICAgICAtIHRlc3RfZnVuY3Rpb25fZGVmaW5pdGlvbigpIC0tIHJpZ29yb3VzbHkgY2hlY2sgZGVmaW5lZCBmdW5jdGlvblxuIyAgICAgLSB0ZXN0X2RhdGFfZnJhbWUoKSAtLSBjaGVjayBpZiBkYXRhZnJhbWUgW2NvbHVtbnNdIGFyZSBlcXVpdmFsZW50XG4jICAgICAtIHRlc3RfZnVuY3Rpb25fcmVzdWx0LCB0ZXN0X2V4cHJlc3Npb25fcmVzdWx0In0=