---
title: "ACCT 420: ML/AI for visual data"
author: 
  - name: Dr. Richard M. Crowley
    email: rcrowley@smu.edu.sg
    affiliation: https://rmc.link/
title-slide-attributes:
  data-background-video: "../Backgrounds/title.mp4"
  data-background-video-loop: "true"
  data-background-size: stretch
format:
  revealjs:
    theme: default
    css: quarto_white.css
    navigation-mode: vertical
    controls: true
    transition: slide
    background-transition: fade
    transition-speed: default
    width: 1536
    height: 864
    logo: ../Logos/logo2.png
    code-link: true
    code-line-numbers: false
    chalkboard:
      theme: whiteboard
    auto-stretch: false
revealjs-plugins:
  - attribution
execute:
  echo: true
  warning: false
  message: false
editor: source
---

# Front Matter {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

<!--# Note that the downlit and xml2 pacakges are required to compile -->
<!--# Installed extensions include `attribution`, `qrcode`, and `material-icons` -->

```{r helpers, echo=FALSE, warning=FALSE, message=F}
library(knitr)
library(kableExtra)

#' Pretty formatted HTML tables
#' 
#' @param df The dataframe to format as a table
#' @param names Names to use for columns if not using the df column names. Vector
#' @param highlight_cols Columns to highlight. Default is c(1) for highlighting the first row. Use c() to remove.
#' @param highlight_rows Rows to highlight. Default is c() to not highlight any rows.
#' @param color_rows Rows to highlight in color. Default is c() to not highlight any rows.
#' @param highlight_end_rows Rows to highlight. Uses a reverse indexing where 1 is the last row, 2 is second to last, etc.
#' @param full Use to span the full width of a slide. FALSE by default.
#' @param fixed_header Whether the first row should be treated as a header row. TRUE by default.
html_df <- function(df, names=NULL, highlight_cols=c(1), highlight_rows=c(), color_rows=c(), highlight_end_rows=c(), full=F, fixed_header=T) {
  if(!length(names)) {
    names=colnames(df)
  }
  kbl(df,"html", col.names = names, align = c("l",rep('c',length(cols)-1))) %>%
    kable_paper(c("striped","hover"), full_width=full, fixed_thead=F, html_font = "\"Source Sans Pro\", Helvetica, sans-serif") %>%
    {if(length(highlight_cols)) column_spec(., highlight_cols,bold=T, background = "#ffffff99") else .} %>%
    {if(length(highlight_rows)) row_spec(., highlight_rows,bold=T, background = "#ffffff99") else .} %>%
    {if(length(color_rows)) row_spec(., color_rows,bold=T, background = "#aaaaff99") else .} %>%
    {if(length(highlight_end_rows)) row_spec(., nrow(df) + 1 - highlight_end_rows,bold=T, background = "#ffffff99") else .}
}

library(tidyverse)
```

<!--# List packages here that are not being correctly highlighted -->
<!--# The usual cause is calling the library from a echo=FALSE cell -->

```{r}
#| include: false
options(downlit.attached = c("knitr", "kableExtra", "tidyverse"))
```

## Learning objectives {data-background="../Backgrounds/default.jpg"}

:::: columns

::: {.column width="50%"}

![](../Course_Diagrams/Map-9.png){fig-alt="Roadmap for the course"}

:::

::: {.column width="50%"}

- **Theory:**
    - Neural Networks for...
        - Images
        - Audio
        - Video
- **Application:**
    - Handwriting recognition
    - Identifying financial information in images
- **Methodology:**
    - Neural networks
        - CNNs
        - Transformers

:::

::::

## Group project {data-background="../Backgrounds/presentation.jpg"}

- Next class you will have an opportunity to present your work
    - ~12-15 minutes per group
- You will also need to submit your report & code
    - Please submit as a zip file
    - Be sure to include your report **AND** code **AND** slides
        - Code should cover your final model
            - Covering more is fine though
    - **Do not include the data!**
- Competitions close Monday at 12 noon!

# Image data {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

## Thinking about images as data {data-background="../Backgrounds/default.jpg"}

- Images **are** data, but they are very unstructured
    - No instructions to say what is in them
    - No common grammar across images
    - Many, many possible subjects, objects, styles, etc.
- From a computer's perspective, images are just 3-dimensional matrices
    - Rows (pixels)
    - Columns (pixels)
    - Color channels (usually Red, Green, and Blue)

## Using images as data {data-background="../Backgrounds/default.jpg"}

- We can definitely use numeric matrices as data
    - We did this plenty with XGBoost, for instance
- However, images have a lot of different numbers tied to each observation (image).

:::: {.columns}

::: {.column width="50%"}

![](../Figures/tweet_nonfin_3.jpeg){width="400px" fig-align="center"}

- Source: Twitter
:::

::: {.column width="50%"}

- 798 rows
- 1200 columns
- 3 color channels
- 798 $\times$ 1,200 $\times$ 3 $=$ 2,872,800
    - The number of 'variables' per image like this!

:::

::::

## Using images in practice {data-background="../Backgrounds/default.jpg"}

- There are a number of strategies to shrink images' dimensionality
    1. Downsample the image to a smaller resolution like 256x256x3
    2. Convert to grayscale
    3. Cut the image up and use sections of the image as variables instead of individual numbers in the matrix
        - Often done with convolutions in neural networks
    4. Drop variables that aren't needed, like LASSO

# Images in R using Keras {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

## R interface to Keras {data-background="../Backgrounds/default.jpg"}

> By R Studio: <a href="https://keras.rstudio.com/index.html">details here</a>

- Install with: `devtools::install_github("rstudio/keras")`
- Finish the install in one of two ways:

:::: {.columns}

::: {.column width="50%"}

**For those using <a href="https://docs.conda.io/en/latest/">Conda</a>**

- CPU Based, works on *any* computer

```{r, eval=F}
library(keras)
install_keras()
```

- Nvidia GPU based
    - Install the <a href="https://www.tensorflow.org/install/gpu">Software requirements</a> first

```{r, eval = F}
library(keras)
install_keras(tensorflow = "gpu")
```

:::

::: {.column width="50%"}

**Using your own python setup**

- Follow Google's <a href="https://www.tensorflow.org/install">install instructions for Tensorflow</a>
- Install keras from a terminal with `pip install keras`
- R Studio's keras package will automatically find it
    - May require a reboot to work on Windows

:::

::::

## The "hello world" of neural networks {data-background="../Backgrounds/lab.jpg"}

- A "Hello  world" is the standard first program one writes in a language
- In R, that could be:

```{r}
print("Hello world!")
```

- For neural networks, the "Hello world" is writing a handwriting classification script
    - We will use the MNIST database, which contains many writing samples and the answers
    - Keras provides this for us :)

```{r, eval=F}
library(keras)
mnist <- dataset_mnist()
```

```{r, echo=F, warning=F, message=F}
library(keras)
model <- load_model_hdf5("../../Data/Session_11-mnist_model.h5")
history <- readRDS('../../Data/Session_11-mnist_history.rds')
eval <- readRDS('../../Data/Session_11-mnist_eval.rds')
```

## Set up and pre-processing {data-background="../Backgrounds/lab.jpg"}

- We still do training and testing samples
    - It is just as important here as before!

```{r, eval = F}
x_train <- mnist$train$x
y_train <- mnist$train$y
x_test <- mnist$test$x
y_test <- mnist$test$y
```

<br>

- Shape and scale the data into a big $784 \times 1$ matrix with every value between 0 and 1

```{r, eval=F}
# reshape
x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))
# rescale
x_train <- x_train / 255
x_test <- x_test / 255
```

## Building a Neural Network {data-background="../Backgrounds/lab.jpg"}

```{r, eval=F}
model <- keras_model_sequential() # Open an interface to tensorflow
# Set up the neural network
model %>% 
    layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>% 
    layer_dropout(rate = 0.4) %>% 
    layer_dense(units = 128, activation = 'relu') %>%
    layer_dropout(rate = 0.3) %>%
    layer_dense(units = 10, activation = 'softmax')
```

> That's it.  Keras makes it easy.

- Relu is the same as a call option payoff: $max(x, 0)$
- Softmax approximates the $argmax$ function
    - Which input was highest?
    - Note that the `units = 10` maps to the number of categories in the data

## The model {data-background="../Backgrounds/lab.jpg"}

- We can just call `summary()` on the model to see what we built

```{r, warning=F}
summary(model)
```

## Compile the model {data-background="../Backgrounds/lab.jpg"}

- Tensorflow doesn't compute anything until you tell it to
- After we have set up the instructions for the model, we compile it to build our actual model

```{r, eval=F}
model %>% compile(
    loss = 'sparse_categorical_crossentropy',
    optimizer = optimizer_rmsprop(),
    metrics = c('accuracy')
)
```

## Running the model {data-background="../Backgrounds/lab.jpg"}

- It takes about 1 minute to run on an Nvidia GTX 1080

:::: {.columns}

::: {.column width="50%"}

```{r, eval=F}
history <- model %>% fit(
    x_train, y_train, 
    epochs = 30, batch_size = 128, 
    validation_split = 0.2
)
```

:::

::: {.column width="50%"}

```{r, fig.height=4.5, fig.width=4}
plot(history)
```

:::

::::

## Out of sample testing {data-background="../Backgrounds/lab.jpg"}

```{r, eval=F}
eval <- model %>% evaluate(x_test, y_test)
eval
```

```{r, echo=F}
eval
```

> 98% accurate!  Random chance would only be 10%

## Saving the model {data-background="../Backgrounds/lab.jpg"}

- Saving:

```{r, eval=F}
model %>% save_model_hdf5("../../Data/Session_11-mnist_model.h5")
```

<br>

- Loading an already trained model:

```{r, eval=F}
model <- load_model_hdf5("../../Data/Session_11-mnist_model.h5")
```

# More advanced image techniques {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

## How CNNs work {data-background="../Backgrounds/reference.jpg"}

- CNNs use repeated convolution, usually looking at slightly bigger chunks of data each iteration
- But what is convolution?  It is illustrated by the following graphs (from <a href="https://en.wikipedia.org/wiki/Convolution">Wikipedia</a>):

<img src="../Figures/conv.gif" width="400px">
<img src="../Figures/conv2.gif" width="400px">

> <a href="http://colah.github.io/posts/2014-07-Understanding-Convolutions/">Further reading</a>


## CNN example: Alexnet {data-background="../Backgrounds/default.jpg" data-transition="slide-in fade-out"}

:::: {.columns}

::: {.column width="50%"}
Example output of AlexNet

<img src="../Figures/alexnet_ex.png" width="550px">
:::

::: {.column width="50%"}
The first (of 5) layers learned

<img src="../Figures/alexnet_conv.png" width="550px">
:::

::::

## Recent attempts at explaining CNNs {data-background="../Backgrounds/default.jpg"}

- Google & Stanford's "Automated Concept-based Explanation"

<a href="https://venturebeat.com/2019/10/14/googles-ai-explains-how-image-classifiers-made-their-decisions/"><img src="../Figures/Google_ACE.jpg" height="500px"></a>

## Try out a CNN in your browser! {data-background="../Backgrounds/lab.jpg"}

- <a href="https://colab.research.google.com/drive/1U_GYz2NP1yVDKkqTTXa_GQN7OvY8CpnJ?usp=sharing">Fashion MNIST with Keras and TPUs</a>
    - <a href="https://github.com/zalandoresearch/fashion-mnist">Fashion MNIST</a>: A dataset of clothing pictures
    - Keras: An easier API for TensorFlow
    - TPU: A "Tensor Processing Unit" -- A custom processor built by Google
    - Python code

<img src="../Figures/fashion-mnist.png" height="200px">

# Detecting financial content with a CNN {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

## The data {data-background="../Backgrounds/case_digital.jpg"}

- 5,000 images that should not contain financial information
- 2,777 images that should contain financial information
- 500 of each type are held aside for testing

> Goal: Build a classifier based on the images' content

## Examples: Financial {data-background="../Backgrounds/case_digital.jpg"}

:::: {.columns}

::: {.column width="50%"}
<img src="../Figures/tweet_fin_1.jpeg" width="300px">

<img src="../Figures/tweet_fin_2.jpeg" width="300px">
:::

::: {.column width="50%"}
<img src="../Figures/tweet_fin_3.jpeg" width="300px">

<img src="../Figures/tweet_fin_4.jpeg" width="300px">
:::

::::

## Examples: Non-financial {data-background="../Backgrounds/case_digital.jpg"}


:::: {.columns}

::: {.column width="50%"}
<img src="../Figures/tweet_nonfin_1.jpeg" width="300px">

<img src="../Figures/tweet_nonfin_2.jpeg" width="300px">
:::

::: {.column width="50%"}
<img src="../Figures/tweet_nonfin_3.jpeg" width="300px">

<img src="../Figures/tweet_nonfin_4.jpeg" width="300px">
:::

::::

```{r, echo=F, warning=F, message=F}
model <- load_model_hdf5("../../Data/Session_11-tweet_model.h5")
history <- readRDS('../../Data/Session_11-tweet_history.rds')
eval <- readRDS('../../Data/Session_11-tweet_eval.rds')
```

## The CNN {data-background="../Backgrounds/case_digital.jpg"}

```{r, message=F}
summary(model)
```

## Running the model {data-background="../Backgrounds/case_digital.jpg"}

- It takes about 10 minutes to run on an Nvidia GTX 1080

:::: {.columns}

::: {.column width="50%"}
```{r, eval=F}
history <- model %>% fit_generator(
  img_train, # training data
  epochs = 10, # epoch
  steps_per_epoch =
   as.integer(train_samples/batch_size),
  # print progress
  verbose = 2,
)
```
:::

::: {.column width="50%"}
```{r, fig.height=4.5, fig.width=4, message=F}
plot(history)
```
:::

::::


## Out of sample testing {data-background="../Backgrounds/case_digital.jpg"}

```{r, eval=F}
eval <- model %>%
  evaluate_generator(img_test,
                     steps = as.integer(test_samples / batch_size),
                     workers = 4)
eval
```

```{r, echo=F}
eval
```

## Optimizing the CNN {data-background="../Backgrounds/lab.jpg"}

- The model we saw was run for 10 epochs (iterations)
- Why not more?  Why not less?

```{r, message=F, warning=F, fig.height=4}
history <- readRDS('../../Data/Session_11-tweet_history-30epoch.rds')
plot(history)
```

# Video data {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

## Working with video {data-background="../Backgrounds/default.jpg"}

- Video data is challenging -- very storage intensive
    - Ex.: Uber's self driving cars would generate >100GB of data *per hour* **per car**
- Video data is very promising
    - Think of how many task involve vision!
        - Driving
        - Photography
        - Warehouse auditing...
- At the end of the day though, video is just a sequence of images


## One method for video {data-background="../Backgrounds/default.jpg"}

> YOLOv3

>- You
>- Only
>- ____
>- Once

<div class="fragment exclude">

> You Only Look Once: Because the algorithm only does one pass (looks once) to classify any and all objects

</div>

## {data-background="../Backgrounds/case_digital.jpg"}

<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/MPU2HistivI" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

> [Video link](https://www.youtube.com/watch?v=MPU2HistivI)

## What does YOLO do? {data-background="../Backgrounds/default.jpg"}

- It spots objects in videos and labels them
    - It also figures out a *bounding box* -- a box containing the object inside the video frame
- It can spot *overlapping* objects
- It can spot multiple of the same or different object types
- The baseline model (using the COCO dataset) can detect 80 different object types
    - There are other datasets with more objects

## How does Yolo do it? Map of Tiny YOLO {data-background="../Backgrounds/default.jpg"}

<div id="scroller" style="overflow:scroll; height:400px;">
![](../Figures/tinyyolo.png){width="600px" fig-align="center"}
</div>

> Yolo model and graphing tool from [lutzroeder/netron](https://github.com/lutzroeder/netron)

## How does Yolo do it? {data-background="../Backgrounds/default.jpg"}

[![](https://cdn-images-1.medium.com/max/1000/1*d4Eg17IVJ0L41e7CTWLLSg.png){width="600px" fig-align="center"}](https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b)

> Diagram from [*What's new in YOLO v3*](https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b) by Ayoosh Kathuria

## Final word on object detection {data-background="../Backgrounds/default.jpg"}

- An algorithm like YOLO v3 is somewhat tricky to run
- Preparing the algorithm takes a long time
    - The final output, though, can run on much cheaper hardware
- These algorithms just recently became feasible so their impact has yet to be felt so strongly

> Think about how facial recognition showed up everywhere for images over the past few years

## Where to get video data {data-background="../Backgrounds/default.jpg"}

- One extensive source is <a href="https://research.google.com/youtube8m/">Youtube-8M</a>
    - 6.1M videos, 3-10 minutes each
    - Each video has >1,000 views
    - 350,000 hours of video
    - 237,000 labeled 5 second segments
    - 1.3B video features that are machine labeled
    - 1.3B audio features that are machine labeled
    
## A word on ethics of object detection {data-background="../Backgrounds/default.jpg"}

![](../Figures/yolo_stopped.png){width="500px" fig-align="center"}

> From Redmon and Farhadi (2018) [The YOLO v3 paper]



# Combining images and text in 1 model {data-background-video="../Backgrounds/section.mp4" data-background-video-loop="true"}

## Large language models + Images {data-background="../Backgrounds/default.jpg"}

- Multiple impactful models were released since 2021 that merge text and image processing into a single model
    - CLIP: Contrastive Language-Image Pre-training
        - Pairs images with captions
    - Stable Diffusion
        - Image generation from text

> These work by embedding images and text into the same embedding space

## CLIP {data-background="../Backgrounds/default.jpg"}

- Code for this is available at: [rmc.link/colab_clip](https://rmc.link/colab_clip)

![](../Figures/CLIP_example.png){width="600px" fig-align="center"}

## Stable diffusion: Content {data-background="../Backgrounds/default.jpg"}

- Code to implement as a Telegram bot: [rmcrowley2000/StableDiffBot](https://github.com/rmcrowley2000/StableDiffBot)

:::: {.columns}

::: {.column width="50%"}

"A photo of the Singapore skyline including Marina Bay Sands"

![](../Figures/SDB_9.jpg){width=400px fig-align="center"}

:::

::: {.column width="50%"}

"Singapore Management University"

![](../Figures/SDB_5.jpg){width=400px fig-align="center"}

:::

::::

## Stable diffusion: Style {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}

"Lithograph of a camel eating a pear"

![](../Figures/SDB_1.jpg){width=400px fig-align="center"}

:::

::: {.column width="50%"}

"A cartoon icon of a dog getting a hair cut."

![](../Figures/SDB_3.jpg){width=400px fig-align="center"}

:::

::::

## Stable diffusion: Problems {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}

"Sustainability data"

![](../Figures/SDB_4.jpg){width=400px fig-align="center"}

:::

::: {.column width="50%"}

"A cavapoo enjoying a nice warm cup of tea"

![](../Figures/SDB_8.jpg){width=400px fig-align="center"}

:::

::::

## Stable diffusion: Complexity {data-background="../Backgrounds/default.jpg"}

"Tiny cute isometric living room in a cutaway box, soft smooth lighting, soft colors, purple and blue color scheme, soft colors, 100mm lens, 3d blender render"

:::: {.columns}

::: {.column width="50%"}

![](../Figures/SDB_6.jpg){width=400px fig-align="center"}

:::

::: {.column width="50%"}

![](../Figures/SDB_10.jpg){width=400px fig-align="center"}

:::

::::






# End Matter {data-background-video="../Backgrounds/section.mp4"}

## Recap {data-background="../Backgrounds/ending.jpg"}

Today, we:

- Learned about using images as data
- Constructed a simple handwriting recognition system
- Learned about more advanced image methods
- Applied CNNs to detect financial information in images on Twitter
- Learned about object detection in videos
- Learned about methods combining images and text

## Wrap up {data-background="../Backgrounds/ending.jpg"}

- For next week:
    - Finish the group project!
        1. Kaggle submission closes Monday!
        2. Turn in your code, presentation, and report through eLearn's dropbox
        3. Prepare a short (~12-15 minute) presentation for class
- Survey on the class session at this QR code:

{{< qrcode https://rmc.link/420survey11 qrsurvey width=200 height=200 >}}

## More fun examples {data-background="../Backgrounds/default.jpg"}

- Interactive:
    - <a href="https://magenta.tensorflow.org/demos/performance_rnn/index.html#2|2,0,1,0,1,1,0,1,0,1,0,1|1,1,1,1,1,1,1,1,1,1,1,1|1,1,1,1,1,1,1,1,1,1,1,1|false">Performance RNN</a>
    - <a href="https://js.tensorflow.org/">TensorFlow.js examples</a>
- Others:
    - <a href="https://github.com/google/deepdream">Google's deepdream</a>
    - <a href="https://github.com/googlecreativelab/open-nsynth-super">Open NSynth Super</a>

## Bonus: Neural networks in interactive media {data-background="../Backgrounds/default.jpg"}

- <a href="">Super Mario using MarI/O</a>
- <a href="https://www.twitch.tv/sethbling/clip/FrigidBillowingBasenjiCopyThis?filter=clips&range=all&sort=time">Mario Kart using an RNN for controller prediction</a>
- <a href="https://openai.com/five/">Open AI's Five tops Dota 2</a>
    - Trained on 180 years of play
- <a href="https://deepmind.com/blog/article/AlphaStar-Grandmaster-level-in-StarCraft-II-using-multi-agent-reinforcement-learning">Google Deepmind's Alphastar AI on StarCraft II</a>
    - Trained on 200 years of play

## Packages used for these slides {data-background="../Backgrounds/code.jpg" visibility="uncounted"}


- `{DT}`
- `{downlit}`
- `{kableExtra}`
- `{keras}`
- `{knitr}`
- `{plotly}`
- `{quarto}`
- `{revealjs}`
- `{tidyverse}`



## NOTES {class="never"}
