---
title:
output:
  revealjs::revealjs_presentation:
    theme: white
    css: white-edits.css
    highlight: tango
    transition: slide
    self_contained: false
    reveal_plugins: ["chalkboard","search"]
    transitionSpead: fast
    mathjax: "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS_SVG"
    reveal_options:
      minScale: 1.0
      maxScale: 1.0
      width: 1536
      height: 864
      slideNumber: true
      controlsTutorial: true
      chalkboard:
        theme: whiteboard
        readOnly: false
---

## ACCT 420: ML and AI for visual data {data-background-video="../Backgrounds/title.mp4" data-background-video-loop=true class="header no-toc-progress present-only" data-state="no-toc-progress"}

<!-- "reveal_plugins:..." and "chalkboard:" are handled by prep38.py -->
```{r helpers, echo=FALSE, warning=FALSE, message=F}
library(knitr)
library(kableExtra)
html_df <- function(text, cols=NULL, col1=FALSE, full=F) {
  if(!length(cols)) {
    cols=colnames(text)
  }
  if(!col1) {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full)
  } else {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full) %>%
      column_spec(1,bold=T)
  }
}
```

<h3 class="date"><br>Session 11</h3>
<h2 class="author"><br>Dr. Richard M. Crowley<br>rcrowley@smu.edu.sg<br><a href="http://rmc.link/">http://rmc.link/</a></h2>

## ACCT 420: ML and AI for visual data {data-background="../Backgrounds/title.jpg" class="header no-toc-progress present-not" data-state="no-toc-progress"}

<h3 class="date"><br>Session 11</h3>
<h2 class="author"><br>Dr. Richard M. Crowley<br>rcrowley@smu.edu.sg<br><a href="http://rmc.link/">http://rmc.link/</a></h2>

<!-- Load in primary data set -->
```{r, include=FALSE}
library(tidyverse)
```

# Front matter {data-background="../Backgrounds/section.jpg" class="header"}

## Learning objectives {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Course_Diagrams/Map-9.png" width="500px">
</div>
<div class="right objectives">
- <b>Theory:</b>
    - Neural Networks for...
        - Images
        - Audio
        - Video
- <b>Application:</b>
    - Handwriting recognition
    - Identifying financial information in images
- <b>Methodology:</b>
    - Neural networks
        - CNNs
</div>
</div>

## Group project {data-background="../Backgrounds/presentation.jpg" class="default"}

- Next class you will have an opportunity to present your work
    - ~12-15 minutes per group
- You will also need to submit your report & code
    - Please submit as a zip file
    - Be sure to include your report **AND** code **AND** slides
        - Code should cover your final model
            - Covering more is fine though
    - Do not include the data!
- Competitions close Sunday night!

# Image data {data-background="../Backgrounds/section.jpg" class="header"}

## Thinking about images as data {data-background="../Backgrounds/default.jpg" class="default"}

- Images **are** data, but they are very unstructured
    - No instructions to say what is in them
    - No common grammar across images
    - Many, many possible subjects, objects, styles, etc.
- From a computer's perspective, images are just 3-dimensional matrices
    - Rows (pixels)
    - Columns (pixels)
    - Color channels (usually Red, Green, and Blue)

## Using images as data {data-background="../Backgrounds/default.jpg" class="default"}

- We can definitely use numeric matrices as data
    - We did this plenty with XGBoost, for instance
- However, images have a lot of different numbers tied to each observation.

<div class="clearfix">
<div class="left">
<img src="../Figures/tweet_nonfin_3.jpeg" width="400px">

- Source: Twitter
</div>
<div class="right">
- 798 rows
- 1200 columns
- 3 color channels
- 798 $\times$ 1,200 $\times$ 3 $=$ 2,872,800
    - The number of 'variables' per image like this!
</div>
</div>

## Using images in practice {data-background="../Backgrounds/default.jpg" class="default"}

- There are a number of strategies to shrink images' dimensionality
    1. Downsample the image to a smaller resolution like 256x256x3
    2. Convert to grayscale
    3. Cut the image up and use sections of the image as variables instead of individual numbers in the matrix
        - Often done with convolutions in neural networks
    4. Drop variables that aren't needed, like LASSO

# Images in R using Keras {data-background="../Backgrounds/section.jpg" class="header"}

## R interface to Keras {data-background="../Backgrounds/default.jpg" class="default"}

> By R Studio: <a href="https://keras.rstudio.com/index.html">details here</a>

- Install with: `devtools::install_github("rstudio/keras")`
- Finish the install in one of two ways:

<div class="clearfix">
<div class="left">
**For those using <a href="https://docs.conda.io/en/latest/">Conda</a>**

- CPU Based, works on *any* computer

```{r, eval=F}
library(keras)
install_keras()
```

- Nvidia GPU based
    - Install the <a href="https://www.tensorflow.org/install/gpu">Software requirements</a> first

```{r, eval = F}
library(keras)
install_keras(tensorflow = "gpu")
```
</div>
<div class="right">
**Using your own python setup**

- Follow Google's <a href="https://www.tensorflow.org/install">install instructions for Tensorflow</a>
- Install keras from a terminal with `pip install keras`
- R Studio's keras package will automatically find it
    - May require a reboot to work on Windows
</div>
</div>

## The "hello world" of neural networks {data-background="../Backgrounds/lab.jpg" class="default"}

- A "Hello  world" is the standard first program one writes in a language
- In R, that could be:

```{r}
print("Hello world!")
```

- For neural networks, the "Hello world" is writing a handwriting classification script
    - We will use the MNIST database, which contains many writing samples and the answers
    - Keras provides this for us :)

```{r, eval=F}
library(keras)
mnist <- dataset_mnist()
```

```{r, echo=F, warning=F, message=F}
library(keras)
model <- load_model_hdf5("../../Data/Session_11-mnist_model.h5")
history <- readRDS('../../Data/Session_11-mnist_history.rds')
eval <- readRDS('../../Data/Session_11-mnist_eval.rds')
```

## Set up and pre-processing {data-background="../Backgrounds/lab.jpg" class="default"}

- We still do training and testing samples
    - It is just as important here as before!

```{r, eval = F}
x_train <- mnist$train$x
y_train <- mnist$train$y
x_test <- mnist$test$x
y_test <- mnist$test$y
```

<br>

- Shape and scale the data into a big matrix with every value between 0 and 1

```{r, eval=F}
# reshape
x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))
# rescale
x_train <- x_train / 255
x_test <- x_test / 255
```

## Building a Neural Network {data-background="../Backgrounds/lab.jpg" class="default"}

```{r, eval=F}
model <- keras_model_sequential() # Open an interface to tensorflow
# Set up the neural network
model %>% 
    layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>% 
    layer_dropout(rate = 0.4) %>% 
    layer_dense(units = 128, activation = 'relu') %>%
    layer_dropout(rate = 0.3) %>%
    layer_dense(units = 10, activation = 'softmax')
```

> That's it.  Keras makes it easy.

- Relu is the same as a call option payoff: $max(x, 0)$
- Softmax approximates the $argmax$ function
    - Which input was highest?

## The model {data-background="../Backgrounds/lab.jpg" class="default"}

- We can just call `summary()` on the model to see what we built

```{r, warning=F}
summary(model)
```

## Compile the model {data-background="../Backgrounds/lab.jpg" class="default"}

- Tensorflow doesn't compute anything until you tell it to
- After we have set up the instructions for the model, we compile it to build our actual model

```{r, eval=F}
model %>% compile(
    loss = 'sparse_categorical_crossentropy',
    optimizer = optimizer_rmsprop(),
    metrics = c('accuracy')
)
```

## Running the model {data-background="../Backgrounds/lab.jpg" class="default"}

- It takes about 1 minute to run on an Nvidia GTX 1080

<div class="clearfix">
<div class="left">
```{r, eval=F}
history <- model %>% fit(
    x_train, y_train, 
    epochs = 30, batch_size = 128, 
    validation_split = 0.2
)
```
</div>
<div class="right">
```{r, fig.height=4.5, fig.width=4}
plot(history)
```
</div>
</div>

## Out of sample testing {data-background="../Backgrounds/lab.jpg" class="default"}

```{r, eval=F}
eval <- model %>% evaluate(x_test, y_test)
eval
```

```{r, echo=F}
eval
```

## Saving the model {data-background="../Backgrounds/lab.jpg" class="default"}

- Saving:

```{r, eval=F}
model %>% save_model_hdf5("../../Data/Session_11-mnist_model.h5")
```

<br>

- Loading an already trained model:

```{r, eval=F}
model <- load_model_hdf5("../../Data/Session_11-mnist_model.h5")
```

# More advanced image techniques {data-background="../Backgrounds/section.jpg" class="header"}

## How CNNs work {data-background="../Backgrounds/reference.jpg" class="default"}

- CNNs use repeated convolution, usually looking at slightly bigger chunks of data each iteration
- But what is convolution?  It is illustrated by the following graphs (from <a href="https://en.wikipedia.org/wiki/Convolution">Wikipedia</a>):

<img src="../Figures/conv.gif" width="400px">
<img src="../Figures/conv2.gif" width="400px">

> <a href="http://colah.github.io/posts/2014-07-Understanding-Convolutions/">Further reading</a>


## CNN {data-background="../Backgrounds/default.jpg" class="default" data-transition="slide-in fade-out"}

- AlexNet (<a href="http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf">paper</a>)

<div class="clearfix">
<div class="left">
Example output of AlexNet

<img src="../Figures/alexnet_ex.png" width="400px">
</div>
<div class="right">
The first (of 5) layers learned

<img src="../Figures/alexnet_conv.png" width="400px">
</div>
</div>

## {data-background="../Figures/transfer_learning_slide.jpg" data-transition-speed="slow" data-transition="fade-in fade-out"}

## {data-background="../Figures/transfer_learning_slide2.jpg" data-transition-speed="slow" data-transition="fade-in"}

## Transfer Learning {data-background="../Backgrounds/default.jpg" class="default"}

- The previous slide is an example of *style transfer*
- This is also done using CNNs
- <a href="https://medium.com/data-science-group-iitr/artistic-style-transfer-with-convolutional-neural-network-7ce2476039fd">More details here</a>

<img src="../Figures/cnn_picasso.gif" width="600px">

## What is transfer learning? {data-background="../Backgrounds/default.jpg" class="default"}

- It is a method of training an algorithm on one domain and then applying the algorithm on another domain
- It is useful when...
    - You don't have enough data for your primary task
        - And you have enough for a related task
    - You want to augment a model with even more 

## Try it out! {data-background="../Backgrounds/lab.jpg" class="default"}

- Colab file available at <a href="https://colab.research.google.com/drive/1fepwhtxIyqE9VQ02Hb7A7RpMpVKBFGkp">this link</a>
    - Largely based off of <a href="https://github.com/dsgiitr/Neural-Style-Transfer">dsgiitr/Neural-Style-Transfer</a>
    - It just took a few tweaks to get it working in a Google Colaboratory environment properly

<div class="clearfix">
<div class="left">
Inputs:

<img src="../Figures/merlion.jpg" width="200px">

<img src="../Figures/still_life_with_guitar.jpg" width="100px">
</div>
<div class="right">
<img src="../Figures/merlion_st.png" width="400px">
</div>
</div>

## Image generation with VAE {data-background="../Backgrounds/default.jpg" class="default never"}

- Example from <a href="https://github.com/yzwxx/vae-celebA">yzwxx/vae-celeb</a>

<div class="clearfix">
<div class="left">
Input and autoencoder

<img src="../Figures/vae_celeba.gif" width="400px">
</div>
<div class="right">
Generated celebrity images

<img src="../Figures/vae_celeba.png" width="400px">
</div>
</div>

## Note on VAE {data-background="../Backgrounds/default.jpg" class="default never"}

- VAE doesn't just work with image data
- It can also handle sound, such as <a href="https://magenta.tensorflow.org/music-vae">MusicVAE</a>

<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/Ir_AFDKOc-I" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

> <a href="https://colab.research.google.com/notebooks/magenta/music_vae/music_vae.ipynb">Code for trying on your own</a>

## Another generative use: Photography {data-background="../Backgrounds/default.jpg" class="default"}

- <a href="https://google.github.io/creatism/">Creatism: Generating photography from Google Earth Panoramas</a>

<div class="clearfix">
<div class="left">
**Input**

<img src="../Figures/Creatism_in.jpg" width="400px">
</div>
<div class="right">
**Output**

<img src="../Figures/Creatism_out.jpg" width="400px">
</div>
</div>

## Try out a CNN in your browser! {data-background="../Backgrounds/lab.jpg" class="default"}

- <a href="https://colab.research.google.com/drive/1U_GYz2NP1yVDKkqTTXa_GQN7OvY8CpnJ?usp=sharing">Fashion MNIST with Keras and TPUs</a>
    - <a href="https://github.com/zalandoresearch/fashion-mnist">Fashion MNIST</a>: A dataset of clothing pictures
    - Keras: An easier API for TensorFlow
    - TPU: A "Tensor Processing Unit" -- A custom processor built by Google
    - Python code

<img src="../Figures/fashion-mnist.png" height="200px">

## Recent attempts at explaining CNNs {data-background="../Backgrounds/default.jpg" class="default"}

- Google & Stanford's "Automated Concept-based Explanation"

<a href="https://venturebeat.com/2019/10/14/googles-ai-explains-how-image-classifiers-made-their-decisions/"><img src="../Figures/Google_ACE.jpg" height="500px"></a>


# Detecting financial content {data-background="../Backgrounds/section.jpg" class="header"}

## The data {data-background="../Backgrounds/case_digital.jpg" class="default"}

- 5,000 images that should not contain financial information
- 2,777 images that should contain financial information
- 500 of each type are held aside for testing

> Goal: Build a classifier based on the images' content

## Examples: Financial {data-background="../Backgrounds/case_digital.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Figures/tweet_fin_1.jpeg" width="300px">

<img src="../Figures/tweet_fin_2.jpeg" width="300px">
</div>
<div class="right">
<img src="../Figures/tweet_fin_3.jpeg" width="300px">

<img src="../Figures/tweet_fin_4.jpeg" width="300px">
</div>
</div>

## Examples: Non-financial {data-background="../Backgrounds/case_digital.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Figures/tweet_nonfin_1.jpeg" width="300px">

<img src="../Figures/tweet_nonfin_2.jpeg" width="300px">
</div>
<div class="right">
<img src="../Figures/tweet_nonfin_3.jpeg" width="300px">

<img src="../Figures/tweet_nonfin_4.jpeg" width="300px">
</div>
</div>

```{r, echo=F, warning=F, message=F}
model <- load_model_hdf5("../../Data/Session_11-tweet_model.h5")
history <- readRDS('../../Data/Session_11-tweet_history.rds')
eval <- readRDS('../../Data/Session_11-tweet_eval.rds')
```

## The CNN {data-background="../Backgrounds/case_digital.jpg" class="default"}

```{r, message=F}
summary(model)
```

## Running the model {data-background="../Backgrounds/case_digital.jpg" class="default"}

- It takes about 10 minutes to run on an Nvidia GTX 1080

<div class="clearfix">
<div class="left">
```{r, eval=F}
history <- model %>% fit_generator(
  img_train, # training data
  epochs = 10, # epoch
  steps_per_epoch =
   as.integer(train_samples/batch_size),
  # print progress
  verbose = 2,
)
```
</div>
<div class="right">
```{r, fig.height=4.5, fig.width=4, message=F}
plot(history)
```
</div>
</div>

## Out of sample testing {data-background="../Backgrounds/case_digital.jpg" class="default"}

```{r, eval=F}
eval <- model %>%
  evaluate_generator(img_test,
                     steps = as.integer(test_samples / batch_size),
                     workers = 4)
eval
```

```{r, echo=F}
eval
```

## Optimizing the CNN {data-background="../Backgrounds/lab.jpg" class="default"}

- The model we saw was run for 10 epochs (iterations)
- Why not more?  Why not less?

```{r, message=F, warning=F, fig.height=4}
history <- readRDS('../../Data/Session_11-tweet_history-30epoch.rds')
plot(history)
```

## AlexNet variant {data-background="../Backgrounds/lab.jpg" class="default"}

```{r, echo=F, message=F, warning=F}
model <- load_model_hdf5("../../Data/Session_11-tweet_model-alexnet.h5")
history <- readRDS('../../Data/Session_11-tweet_history-alexnet.rds')
```

```{r, message=F}
summary(model)
```

## AlexNet performance {data-background="../Backgrounds/lab.jpg" class="default"}

```{r, message=F, fig.height=4.5}
plot(history)
```

# Video data {data-background="../Backgrounds/section.jpg" class="header"}

## Working with video {data-background="../Backgrounds/default.jpg" class="default"}

- Video data is challenging -- very storage intensive
    - Ex.: Uber's self driving cars would generate >100GB of data *per hour* **per car**
- Video data is very promising
    - Think of how many task involve vision!
        - Driving
        - Photography
        - Warehouse auditing...
- At the end of the day though, video is just a sequence of images


## One method for video {data-background="../Backgrounds/default.jpg" class="default"}

> YOLOv3

>- You
>- Only
>- ____
>- Once

<div class="fragment exclude">

> You Only Look Once: Because the algorithm only does one pass (looks once) to classify any and all objects

</div>

## {data-background="../Backgrounds/case_digital.jpg"}

<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/MPU2HistivI" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

> <a href="https://www.youtube.com/watch?v=MPU2HistivI&feature=youtu.be">Video link</a>

## What does YOLO do? {data-background="../Backgrounds/default.jpg" class="default"}

- It spots objects in videos and labels them
    - It also figures out a *bounding box* -- a box containing the object inside the video frame
- It can spot *overlapping* objects
- It can spot multiple of the same or different object types
- The baseline model (using the COCO dataset) can detect 80 different object types
    - There are other datasets with more objects

## How does Yolo do it? Map of Tiny YOLO {data-background="../Backgrounds/default.jpg" class="default"}

<div id="scroller" style="overflow:scroll; height:400px;">
<img src="../Figures/tinyyolo.png">
</div>

> Yolo model and graphing tool from <a href="https://github.com/lutzroeder/netron">lutzroeder/netron</a>

## How does Yolo do it? {data-background="../Backgrounds/default.jpg" class="default"}

<a href="https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b"><img src="https://cdn-images-1.medium.com/max/1000/1*d4Eg17IVJ0L41e7CTWLLSg.png" width="600px"></a>

> Diagram from <a href="https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b">*What's new in YOLO v3*</a> by Ayoosh Kathuria

## Final word on object detection {data-background="../Backgrounds/default.jpg" class="default"}

- An algorithm like YOLO v3 is somewhat tricky to run
- Preparing the algorithm takes a long time
    - The final output, though, can run on much cheaper hardware
- These algorithms just recently became feasible so their impact has yet to be felt so strongly

> Think about how facial recognition showed up everywhere for images over the past few years

## Where to get video data {data-background="../Backgrounds/default.jpg" class="default"}

- One extensive source is <a href="https://research.google.com/youtube8m/">Youtube-8M</a>
    - 6.1M videos, 3-10 minutes each
    - Each video has >1,000 views
    - 350,000 hours of video
    - 237,000 labeled 5 second segments
    - 1.3B video features that are machine labeled
    - 1.3B audio features that are machine labeled

# End matter {data-background="../Backgrounds/ending.jpg" class="header"}

## Final discussion {data-background="../Backgrounds/group.jpg" class="default"}

> What creative uses for the techniques discussed today do you expect to see become reality in accounting in the next 3-5 years?

- 1 example: Using image recognition techniques, warehouse counting for audit can be automated
    - Strap a camera to a drone, have it fly all over the warehouse, and process the video to get item counts

## Recap {data-background="../Backgrounds/ending.jpg" class="default"}

Today, we:

- Learned about using images as data
- Constructed a simple handwriting recognition system
- Learned about more advanced image methods
- Applied CNNs to detect financial information in images on Twitter
- Learned about object detection in videos

## For next week {data-background="../Backgrounds/ending.jpg" class="default"}

- For next week:
    - Finish the group project!
        1. Kaggle submission closes Sunday!
        2. Turn in your code, presentation, and report through eLearn's dropbox
        3. Prepare a short (~12-15 minute) presentation for class
- Survey on the class session at <a href="https://rmc.link/420survey11">rmc.link/420survey11</a>

## More fun examples {data-background="../Backgrounds/default.jpg" class="default"}

- Interactive:
    - <a href="https://magenta.tensorflow.org/demos/performance_rnn/index.html#2|2,0,1,0,1,1,0,1,0,1,0,1|1,1,1,1,1,1,1,1,1,1,1,1|1,1,1,1,1,1,1,1,1,1,1,1|false">Performance RNN</a>
    - <a href="https://js.tensorflow.org/">TensorFlow.js examples</a>
- Others:
    - <a href="https://github.com/google/deepdream">Google's deepdream</a>
    - <a href="https://github.com/googlecreativelab/open-nsynth-super">Open NSynth Super</a>




## Fun machine learning examples {data-background="../Backgrounds/default.jpg" class="default"}

- Interactive:
    - <a href="https://magenta.tensorflow.org/sketch-rnn-demo">Draw together with a neural network</a>
        - click the images to try it out yourself!
    - <a href="https://quickdraw.withgoogle.com/">Google's Quickdraw</a>
    - <a href="https://teachablemachine.withgoogle.com/">Google's Teachable Machine</a>
    - <a href="https://distill.pub/2016/handwriting/">Four experiments in handwriting with a neural network</a>

## Bonus: Neural networks in interactive media {data-background="../Backgrounds/default.jpg" class="default"}

- <a href="">Super Mario using MarI/O</a>
- <a href="https://www.twitch.tv/sethbling/clip/FrigidBillowingBasenjiCopyThis?filter=clips&range=all&sort=time">Mario Kart using an RNN for controller prediction</a>
- <a href="https://openai.com/five/">Open AI's Five tops Dota 2</a>
    - Trained on 180 years of play
- <a href="https://deepmind.com/blog/article/AlphaStar-Grandmaster-level-in-StarCraft-II-using-multi-agent-reinforcement-learning">Google Deepmind's Alphastar AI on StarCraft II</a>
    - Trained on 200 years of play

## Packages used for these slides {data-background="../Backgrounds/code.jpg" class="default"}

- `package:kableExtra`
- `package:keras`
- `package:knitr`
- `package:tidyverse`

## References {data-background="../Backgrounds/books-old.jpg" class="default stack never"}

## Custom code {data-background="../Backgrounds/code.jpg" class="default stack never"}

## NOTES {class="never"}
