---
title:
output:
  revealjs::revealjs_presentation:
    theme: white
    css: white-edits.css
    highlight: tango
    transition: slide
    self_contained: false
    reveal_plugins: ["chalkboard","search"]
    transitionSpead: fast
    mathjax: "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS_SVG"
    reveal_options:
      minScale: 1.0
      maxScale: 1.0
      width: 1024
      height: 768
      slideNumber: true
      controlsTutorial: true
      chalkboard:
        theme: whiteboard
        readOnly: false
---

## ACCT 420: ML and AI for numeric and text data {data-background="../Backgrounds/title.jpg" class="header no-toc-progress" data-state="no-toc-progress"}

<!-- "reveal_plugins:..." and "chalkboard:" are handled by prep.py -->
```{r helpers, echo=FALSE, warning=FALSE}
library(knitr)
library(kableExtra)
html_df <- function(text, cols=NULL, col1=FALSE, full=F) {
  if(!length(cols)) {
    cols=colnames(text)
  }
  if(!col1) {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full)
  } else {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full) %>%
      column_spec(1,bold=T)
  }
}
```


<h3 class="date"><br>Session 10</h3>
<h2 class="author"><br>Dr. Richard M. Crowley</h2>

<!-- Load in primary data set -->
```{r, include=FALSE}
library(tidyverse)
```

# Front matter {data-background="../Backgrounds/section.jpg" class="header"}

## Learning objectives {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Course_Diagrams/Map-9.png">
</div>
<div class="right objectives">
- <b>Theory:</b>
    - Neural Networks (broad overview)
    - Vector space methods
- <b>Application:</b>
    - Neural networks for understanding textual data
    - Top managements' tweets
- <b>Methodology:</b>
    - Vector methods
    - 6 types of neural networks
    - Others
</div>
</div>

# Languages for ML/AI {data-background="../Backgrounds/section.jpg" class="header"}

## R for ML/AI {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
Older methods

- `package:caret`
- `package:randomForest`
- `package:nnet`
- `package:e1071`
</div>
<div class="right">
Best-in-class

- `package:glmnet`: LASSO and elastic nets
- `package:xgboost`: XGBoost
- `package:Prophet`: ML for time series forecasting
- `package:keras`: Plugs into python's Keras
- `package:H2O4GPU`: Plugs into python's H2O
- `package:spacyr`: Plugs into python's SpaCy
</div>
</div>

## Python for ML/AI {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
Older methods

- Sci-kit learn -- one stop shop for most older libraries
- RPy2
- scipy + numpy + pandas + statsmodels
    - Add <a href="http://deeplearning.net/software/theano/">Theano</a> in for GPU compute
</div>
<div class="right">
Best-in-class

- <a href="https://www.tensorflow.org/">**TENSORFLOW**</a> (Google)
    - Can do everything
- <a href="https://pytorch.org/">pytorch</a> -- python specific Torch port
- <a href="https://radimrehurek.com/gensim/">gensim</a>: "Topic modelling for humans"
- <a href="https://www.h2o.ai/">H2O</a> (H2O)
- <a href="http://caffe.berkeleyvision.org/">caffe</a> (Berkley)
- <a href="https://caffe2.ai/">caffe2</a> (Facebook)
- <a href="https://spacy.io/">SpaCy</a> -- Fast NLP processing
- <a href="https://stanfordnlp.github.io/CoreNLP/other-languages.html">CoreNLP</a> -- through various wrappers to the Java library

</div>
</div>

## Others for ML/AI {data-background="../Backgrounds/default.jpg" class="default"}

- C/C++: Also a first class language for TensorFlow!
    - Really fast -- precompiled
    - Much more difficult to code in
- Swift: Strong TensorFlow support
- Javascript: Improving support from TensorFlow and others

## Why do I keep mentioning TensorFlow? {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
- It can run almost ANY ML/AI/NN algorithm
- It has APIs for easier access like Keras
- Comparatively easy GPU setup
- It can deploy anywhere
    - Python & C/C++ built in
    - Swift, R Haskell, and Rust bindings
    - TensorFlow light for mobile deployment
    - TensorFlow.js for web deployment
</div>
<div class="right">
<a href="https://www.tensorflow.org/lite/"><img src="../Figures/tf_light.png" width="200px"></a>
<a href="https://js.tensorflow.org/"><img src="../Figures/tf_js.png" width="200px"></a>
<a href="https://magenta.tensorflow.org/"><img src="../Figures/tf_magenta.png" width="200px"></a>
<a href="https://tfhub.dev/"><img src="../Figures/tf_hub.png" width="200px"></a>
</div>
</div>

## Why do I keep mentioning TensorFlow? {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
- It has strong support from Google and others
    - <a href="https://tfhub.dev/">TensorFlow Hub</a> -- Premade algorithms for text, image, and video
    - <a href="https://github.com/tensorflow/models">tensorflow/models</a> -- Premade code examples
        -  The <a href="https://github.com/tensorflow/models/tree/master/research">research</a> folder contains an amazing set of resources
    - <a href="https://github.com/google/trax">trax</a> -- AI research models from Google Brain
</div>
<div class="right">
<a href="https://www.tensorflow.org/lite/"><img src="../Figures/tf_light.png" width="200px"></a>
<a href="https://js.tensorflow.org/"><img src="../Figures/tf_js.png" width="200px"></a>
<a href="https://magenta.tensorflow.org/"><img src="../Figures/tf_magenta.png" width="200px"></a>
<a href="https://tfhub.dev/"><img src="../Figures/tf_hub.png" width="200px"></a>
</div>
</div>

## Other notable frameworks {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
- <a href="http://caffe.berkeleyvision.org/">Caffe</a>
    - Python, C/C++, Matlab
    - Good for image processing
- <a href="https://caffe2.ai/">Caffe2</a>
    - C++ and Python
    - Still largely image oriented
- <a href="https://www.microsoft.com/en-us/cognitive-toolkit/">Microsoft Cognitive Toolkit</a>
    - Python, C++
    - Scales well, good for NLP
- <a href="http://torch.ch/">Torch</a> and <a href="https://pytorch.org/">Pytorch</a>
    - For Lua and python
    - <a href="https://www.fast.ai/">fast.ai</a>, <a href="https://github.com/pytorch/elf">ELF</a>, and <a href="https://allennlp.org/">AllenNLP</a>
- <a href="https://www.h2o.ai/">H20</a>
    - Python based
    - Integration with R, Scala...
</div>
<div class="right">
<a href="https://caffe2.ai/"><img src="../Figures/caffe2.svg" width="200px"></a>
<a href="https://www.microsoft.com/en-us/cognitive-toolkit/"><img src="../Figures/cntk.png" width="200px"></a>
<a href="https://pytorch.org/"><img src="../Figures/pytorch.svg" width="200px"></a>
<a href="https://www.h2o.ai/"><img src="../Figures/h2o_logo.svg" width="200px"></a>
</div>
</div>

```{r ending, echo=FALSE, warning = FALSE, message = FALSE, include=F}
library(visNetwork)
nodes <- list(id=list.files('../Figures/Logos/TensorFlow'))
nodes$group <- "TF"
nodes$id <- c(nodes$id,list.files('../Figures/Logos/PyTorch'))
nodes$group[nodes$group != 'TF'] = 'PT'
edges = list("to"=c(), "from"=c())
nodes <- as.data.frame(nodes,stringsAsFactors=FALSE)
edges <- as.data.frame(edges,stringsAsFactors=FALSE)
nodes$image.unselected <- paste('Icons-',nodes$group,'/',nodes$id,sep='')
nodes$shape <- "image"
visNetwork(nodes, edges, width="100%", height="600px")
```

# Neural Networks {data-background="../Backgrounds/section.jpg" class="header"}

## What are neural networks? {data-background="../Backgrounds/reference.jpg" class="default"}

- The phrase *neural network* is thrown around almost like a buzz word
- *Neural networks* are actually a specific type class algorithms
    - There are many implementations with different primary uses

<img src="../Figures/analytics_ml.png" height="400px">

## What are neural networks? {data-background="../Backgrounds/reference.jpg" class="default"}

- Originally, the goal was to construct an algorithm that behaves like a human brain
    - Thus the name
- Current methods don't quite reflect human brains, however:
    1. We don't fully understand how our brains work, which makes replication rather difficult
    2. Most neural networks are constructed for specialized tasks (not general tasks)
    3. Some (but not all) neural networks use tools our brain may not have
        - I.e., **backpropogation** is <a href="https://www.frontiersin.org/articles/10.3389/fncom.2016.00094/full">potentially possible in brains</a>, but it is not pinned down how such a function occurs (if it does occur)

## What are neural networks? {data-background="../Backgrounds/reference.jpg" class="default"}

- Neural networks are a method by which a computer can learn from observational data
- In practice:
    - They were not computationally worthwhile until the mid 2000s
    - They have been known since the 1950s (perceptrons)
    - They can be used to construct algorithms that, at times, perform better than humans themselves
        - But these algorithms are often quite computationally intense, complex, and difficult to understand
    - Much work has been and is being done to make them more accessible

## Types of neural networks {data-background="../Backgrounds/default.jpg" class="default"}

- There are *a lot* of neural network types
    - See The <a href="http://www.asimovinstitute.org/neural-network-zoo/">"Neural Network Zoo"</a>
- Some of the more interesting ones which we will see or have seen:
    - RNN: Recurrent Neural Network
    - LSTM: Long/Short Term Memory
    - CNN: Convolutional Neural Network
    - DAN: Deep Averaging Network
    - GAN: Generative Adversarial Network
- Others worth noting
    - VAE (Variational Autoencoder): Generating *new* data from datasets
- Not in the Zoo, but of note:
    - <a href="http://jalammar.github.io/illustrated-transformer/">Transformer</a>: Networks with "attention"
        - From <a href="https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf">Attention is All You Need</a>

## RNN: Recurrent NN {data-background="../Backgrounds/neural-net.jpg" class="default"}

- Recurrent neural networks embed a history of information in the network
    - The previous computation affects the next one
    - Leads to a *short term memory*
- Used for speech recognition, image captioning, anomaly detection, and many others
    - Also the foundation of LSTM
    - <a href="https://ai.googleblog.com/2017/04/teaching-machines-to-draw.html">SketchRNN</a> (<a href="https://magenta.tensorflow.org/assets/sketch_rnn_demo/index.html">live demo</a>)

<img src="../Figures/NN_RNN.png" width="675px">

## LSTM: Long Short Term Memory {data-background="../Backgrounds/neural-net.jpg" class="default"}

- LSTM improves the *long term memory* of the network while explicitly modeling a *short term memory*
- Used wherever RNNs are used, and then some
    - Ex.: <a href="https://google.github.io/seq2seq/">Seq2seq</a> (machine translation)

<img src="../Figures/NN_LSTM.png">

## CNN: Convolutional NN {data-background="../Backgrounds/neural-net.jpg" class="default"}

- Networks that excel at object detection (in images)
- Can be applied to other data as well
- Ex.: <a href="https://ai.googleblog.com/2016/03/train-your-own-image-classifier-with.html">Inception-v3</a>

<img src="../Figures/NN_CNN.png" width="450px">

## DAN: Deep Averaging Network {data-background="../Backgrounds/neural-net.jpg" class="default"}

- DANs are simple networks that simply average their inputs
- Averaged inputs are then processed a few times
- These networks have found a home in NLP
    - Ex.: <a href="https://tfhub.dev/google/universal-sentence-encoder/2">Universal Sentence Encoder</a>

<img src="../Figures/NN_DAN.png" width="675px">

## GAN: Generative Adversarial Network {data-background="../Backgrounds/neural-net.jpg" class="default"}

- Feature two networks working against each other
- Many novel uses
    - Ex.: The anonymization GAN we saw
    - Ex.: <a href="https://medium.com/syncedreview/face-aging-with-conditional-generative-adversarial-networks-d41076379047">Aging images</a>

<img src="../Figures/NN_GAN.png" width="375px">

## VAE: Variational Autoencoder {data-background="../Backgrounds/neural-net.jpg" class="default"}

- An autoencoder (AE) is an algorithm that can recreate input data
- Variational means this type of AE can vary other aspects to generate completely new output
    - Good for creating <a href="https://github.com/yzwxx/vae-celebA">fake data</a>
- Like a simpler, noisier GAN

<img src="../Figures/NN_VAE.png" width="650px">

## Transformer {data-background="../Backgrounds/neural-net.jpg" class="default"}

- Shares some similarities with RNN and LSTM: Focuses on attention
- Currently being applied to solve many types of problems
- Examples: BERT, GPT-3, XLNEt

<img src="../Figures/Transformer.png" width="650px">

# Vector space models {data-background="../Backgrounds/section.jpg" class="header"}

## Motivating examples {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<a href="https://research.google.com/semantris/"><img src="../Figures/semantris.png" width="400px"></a>
<a href="https://books.google.com/talktobooks/"><img src="../Figures/talk_to_books.png" width="400px"></a>
</div>

## What are "vector space models" {data-background="../Backgrounds/default.jpg" class="default"}

- Different ways of converting some abstract information into numeric information
    - Focus on maintaining some of the underlying structure of the abstract information
- Examples (in chronological order):
    - Word vectors:
        - <a href="https://www.tensorflow.org/tutorials/representation/word2vec">Word2vec</a>
        - <a href="https://nlp.stanford.edu/projects/glove/">GloVe</a>
    - Paragraph/document vectors:
        - <a href="https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5">Doc2Vec</a>
    - Sentence vectors:
        - <a href="https://tfhub.dev/google/universal-sentence-encoder/2">Universal Sentence Encoder</a>

## Word vectors {data-background="../Backgrounds/default.jpg" class="default"}

- Instead of coding individual words, encode word meaning
- The idea:
    - Our old way (encode words as IDs from 1 to N) doesn't understand relationships such as:
        - Spatial
        - Categorical
        - Grammatical (weakly when using stemming)
        - Social
        - etc.
    - Word vectors try to encapsulate all of the above
        - They do this by encoding words as a vector of different features

## Word vectors: Simple example {data-background="../Backgrounds/default.jpg" class="default"}

```{r, echo=F}
wv <- data.frame(words = c('dog', 'cat', 'Bill', 'turkey', 'Turkey', "Singapore"),
                 f_animal = c(0.5, 0.5, 0.1, 0.5, -0.5, -0.5),
                 f_people = c(0.3, 0.1, 0.9, -0.2, 0.1, 0.1),
                 f_location = c(-0.3, -0.3, -0.4, -0.3, 0.7, 0.8))
html_df(wv)
```

- The above is an idealized example
- Notice how we can tell apart different animals based on their relationship with people
- Notice how we can distinguish turkey (the animal) from Turkey (the country) as well

## What it retains: <a href="https://www.tensorflow.org/tutorials/representation/word2vec#visualizing_the_learned_embeddings">word2vec</a> {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Figures/word2vec_gender.png" width="220px" style="margin:0px">
<img src="../Figures/word2vec_tense.png" width="220px" style="margin:0px">
</div>
<div class="right">
<img src="../Figures/word2vec_country.png" width="250px" style="margin:0px">
</div>
</div>

## What it retains: <a href="https://nlp.stanford.edu/projects/glove/">GloVe</a> {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Figures/glove_gender.jpg" width="350px" style="margin:0px">
<img src="../Figures/glove_grammar.jpg" width="350px" style="margin:0px">
</div>
<div class="right">
<img src="../Figures/glove_ceo.jpg" width="350px" style="margin:0px">
<img src="../Figures/glove_zip.jpg" width="350px" style="margin:0px">
</div>
</div>

## How to build word vectors {data-background="../Backgrounds/default.jpg" class="default"}

- Two ways:
    1. Word co-occurrence (like how LDA worked)
        - Global Vectors (GloVe) works this way
            - Available from the `package:text2vec` package
    2. Word order (using an NN)
        - word2vec works this way
            - Available from the `package:rword2vec` package
            - Uses a 2 layer neural network

## How does word order work? {data-background="../Backgrounds/default.jpg" class="default"}

> Infer a word's meaning from the words around it

<img src="../Figures/cbow.png" width="800px">

> Refered to as CBOW (continuous bag of words)

## How else can word order work? {data-background="../Backgrounds/default.jpg" class="default"}

> Infer a word's meaning by *generating* words around it

<img src="../Figures/skipgram.png" width="800px">

> Refered to as the Skip-gram model

## An example of using word2vec {data-background="../Backgrounds/lab.jpg" class="default"}

- In the BCE paper from Session 6, word2vec was used to provide assurance that the LDA model works reasonably well on annual reports
    1. We trained a word2vec model on random issues of the Wall Street Journal (247.8M words)
    2. The resulting model "understood" words in the context of the WSJ
    3. We then ran a psychology experiment (word intrusion task) on the algorithm

## Word intrusion task {data-background="../Backgrounds/lab.jpg" class="default"}

- The task is to find which word doesn't belong
- Each question consisted of 3 words from 1 topic and 1 *intruded* from another random topic
    - Ex.:
        - <span class="fragment highlight-red">Laser</span>, Drug, Viral, Therapeutic
        - Supply, Steel, Capacity, <span class="fragment highlight-red">Losses</span>
        - Relief, Lousisiana, <span class="fragment highlight-red">Cargo</span>, Assisted

## Results {data-background="../Backgrounds/lab.jpg" class="default"}

<img src="../Figures/word2vec_validation.png" height="500px">

## Implementing in R {data-background="../Backgrounds/default.jpg" class="default"}

- A few options:
    - The <a href="https://github.com/mukul13/rword2vec">rword2vec package</a> for word2vec
    - The <a href="http://text2vec.org/glove.html">text2vec package</a> for GloVe
    - Rolling your own neural network for word2vec with `package:keras` (<a href="https://blogs.rstudio.com/tensorflow/posts/2017-12-22-word-embeddings-with-keras/">guide here</a>)

## When are vector embeddings useful? {data-background="../Backgrounds/default.jpg" class="default"}

1. You care about the words used, by not stylistic choices
2. You want to crunch down a bunch of words into a smaller number of dimensions without running any bigger models (like LDA) on the text.

> <a href="https://wordsimilarity.com/en/king">An interactive demo of word similarity</a>

# Understanding phrases (or larger) {data-background="../Backgrounds/section.jpg" class="header"}

## Document vectors {data-background="../Backgrounds/default.jpg" class="default"}

- Document vectors work very similarly to word vectors
    - 1 added twist: a document/paragraph/sentence level factor variable
        - This is used to learn a vector representation of each text chunk
        - Learned simultaneously with the word vectors
            - Caveat: it can also be learned independently using <a href="https://cs.stanford.edu/~quocle/paragraph_vector.pdf">PV-DBOW</a>
- This is quite related to what we learned with LDA as well!
    - Both can tell us the topics discussed

## Wikipedia article categorization {data-background="../Backgrounds/default.jpg" class="default"}

<img src="../Figures/word2vec_colah.png" height="450px">

> <a href="https://colah.github.io/posts/2015-01-Visualizing-Representations/big_vis/wiki.html">Source article</a> (colah.github.io)

## Universal Sentence Encoder (USE) {data-background="../Backgrounds/default.jpg" class="default"}

- We saw this briefly last week
    - This is the algorithm with less bias
- Focused on representing sentence-length chunks of text

<img src="../Figures/gmail_demo.gif" width="600px">

## A fun example of with USE {data-background="../Backgrounds/lab.jpg" class="default"}

- <a href="https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/shakespeare_with_tpu_and_keras.ipynb">Predict Shakespeare with Cloud TPUs and Keras</a>

<img src="../Figures/shakespeare_USE.png" width="800px">

## Cavaet on using USE {data-background="../Backgrounds/lab.jpg" class="default"}

- One big caveat: USE only knows what it's trained on
    - Ex.: Feeding the same USE algorithm WSJ text

> Samsung Electronics Co., suffering a handset sales slide, revealed a foldable-screen smartphone that folds like a book and opens up to tablet size.  <span class="hi1">Ah, horror? I play Thee to her alone;<br>    And when we have withdrom him, good all.<br>      Come, go with no less through.<br><br>      Enter Don Pedres. A flourish and my money. I will tarry. Well, you do!<br><br>LADY CAPULET.<br>Farewell; and you are

## How does USE work? {data-background="../Backgrounds/reference.jpg" class="default"}

- USE is based on DAN and Transformer
    - There is another specification as well
    - Learns the meaning of sentences via words' meanings
- Learn more: <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46808.pdf">Original paper</a> and <a href="https://tfhub.dev/google/universal-sentence-encoder/4">TensorFlow site</a>
- In practice, it works quite well

<img src="../Figures/USE_ex.png" height="350px">

## Try it out! {data-background="../Backgrounds/lab.jpg" class="default"}

- Run on <a href="https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb">Google Colab</a>
    - Python code
    - Just click the cells in order, and click run
    - Colab provides free servers to run the code on
        - It still takes a few minutes to run though

## Bringing this into accounting {data-background="../Backgrounds/case_digital.jpg" class="default case_digital"}

> Understanding why stock markets respond strongly to CEOs and CFOs tweets:

- Crowley, Huang, and Lu 2020, "Executive Tweets"
- Data: Tweets for ~100 executives and their firms from 2011 through 2017
- Premise: Markets respond more strongly to executives' tweets than firms' tweets
- Idea: Do markets *trust* executives more or do executives post new *useful information*?

## How can USE help us to solve this? {data-background="../Backgrounds/case_digital.jpg" class="default case_digital"}

> Use USE to determine if there is new content

<div class="clearfix">
<div class="left">

1. Use USE to extract each tweet's meaning
2. See how similar executives' tweets are to their firms' tweets
    - Using the great RANN library in R to efficiently calculate this
3. See how markets respond conditional on tweet similarity

</div>
<div class="right">

<img src="../Figures/USE_similarity.png">

</div>
</div>

## Market reaction mechanism {data-background="../Backgrounds/case_digital.jpg" class="default"}

$$
\begin{array}{l c c}
\hline
\text{Variable} & \left|MMR_{(+1)}\right| & t\text{-value} \\ \hline
\text{Exec fin tweets} & -0.015 & (-1.57) \\
\text{Similarity} \times \text{Exec fin tweets} & 0.038^{*} & (1.91) \\
\text{Controls} & \text{Yes} & \\
\text{Firm FE} & \text{Yes} & \\
\text{Exec FE} & \text{Yes} & \\
\text{Year FE} & \text{Yes} & \\
\text{Month FE} & \text{Yes} & \\
\hline
\end{array}
$$

> Result is consistent with *Trust* driving investor reaction to executive financial tweets.

## Other Transformer models {data-background="../Backgrounds/default.jpg" class="default"}

- **BERT**
    - Optimized to mimic question and answer behavior (<a href="https://rajpurkar.github.io/SQuAD-explorer/">examples</a>)
    - Now used in <a href="https://www.blog.google/products/search/search-language-understanding-bert/">Google Search</a> for at least 70 languages
    - <a href="https://towardsdatascience.com/a-review-of-bert-based-models-4ffdc0f15d58">Additional reading</a>
    - Available in <a href="https://tfhub.dev/google/collections/bert/1">TensorFlow Hub</a>
- **XLNet**
    - Similar objective to BERT, but with a focus on word order
- **T5**
    - A more extensible transformer model
    - <a href="https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html">Details</a>

## Other Transformer models {data-background="../Backgrounds/default.jpg" class="default"}

- **GPT-2**
    - A pretty good model for mimicking human speech patterns
    - Considered dangerous enough to not release initially (<a href="https://openai.com/blog/better-language-models/">source</a>)
        - Released 9 months later alongside a model to detect GPT-2 text
    - Demo: <a href="https://app.inferkit.com/demo">Talk to Transformer</a>
- **GPT-3**
    - Follow-up to GPT-2, remarkably good at generating human-like text
    - A massive model containing 175 billion parameters inside
    - <a href="https://blogs.microsoft.com/blog/2020/09/22/microsoft-teams-up-with-openai-to-exclusively-license-gpt-3-language-model/">Exclusively licensed by Microsoft</a> and available as an API

## {data-background-iframe="https://app.inferkit.com/demo"}

# End matter {data-background="../Backgrounds/ending.jpg" class="header"}

## Discussion {data-background="../Backgrounds/group.jpg" class="default"}

> What creative uses for the techniques discussed today do you expect to see become reality in accounting in the next 3-5 years?

- Brainstorm with your group and try to come up with 1 good use for some technique discussed today
- Each group will be asked to share 1 use

## Recap {data-background="../Backgrounds/ending.jpg" class="default"}

Today, we:

- Learned formally what neural networks (NNs) are
- Discussed a variety of NN-based algorithms
- Saw uses for word and sentence vectors in a financial context

## For next week {data-background="../Backgrounds/ending.jpg" class="default"}

- For next week:
    - Work on the group project!
        - Definitely try to get a submission in on Kaggle
    - We'll keep talking about neural networks
        - A bit more theory
        - A lot more examples
        - Some real neural networks coded in **R**
- Survey on the class session at <a href="https://rmc.link/420survey10">rmc.link/420survey10</a>

## More fun examples {data-background="../Backgrounds/default.jpg" class="default"}

- Interactive:
    - <a href="https://js.tensorflow.org/demos">TensorFlow.js examples</a>
    - <a href="https://research.google.com/semantris/">Semantris</a>
        - A game based on the Universal Sentence Encoder
- Non-interactive
    - <a href="https://blog.insightdatascience.com/hero2vec-d42d6838c941">Predicting e-sports winners with Machine Learning</a>

## Now hiring! {data-background="../Backgrounds/people.jpg" class="default never"}

- Currently looking for **3-4* part time hires
- Supported by an MOE grant -- paid at \$10/hour + CPF
    - Restricted to SMU students
    - 3 month duration (renewable)

> We are trying to: 1) update the identifiers used for tracking companies and executives on Twitter, along with 2) identifying user types on Twitter.

- If you are interested, just email me your resume

## Packages used for these slides {data-background="../Backgrounds/coding.jpg" class="default"}

- `package:kableExtra`
- `package:knitr`
- `package:tidyverse`
    - `package:dplyr`, `package:magrittr`, `package:readr`

## Generating Shakespeare {data-background="../Backgrounds/coding.jpg" class="default"}

```{python, eval=F}
seed_txt = 'Looks it not like the king?  Verily, we must go! '  # Original code
seed_txt = 'SCENE I. Elsinore. A platform before the Castle.\n\n Enter Francisco and Barnardo, two sentinels.\n\nBARNARDO.\nWho’s there?\n\nFRANCISCO.\nNay, answer me. Stand and unfold yourself.\n\nBARNARDO.\nLong live the King!\n\n'  # Hamlet
seed_txt = 'Samsung Electronics Co., suffering a handset sales slide, revealed a foldable-screen smartphone that folds like a book and opens up to tablet size.'  # WSJ article
# From: https://www.wsj.com/articles/samsung-unveils-foldable-screen-smartphone-1541632221
```

## NOTES {class="never"}

- Enron
- Videos & sound to show in class
    - Emergence of Locomotion Behaviours in Rich Environments: https://www.youtube.com/watch?v=hx_bgoTF7bs
    - Performance RNN: https://magenta.tensorflow.org/demos/performance_rnn/index.html#2|2,0,1,0,1,1,0,1,0,1,0,1|1,1,1,1,1,1,1,1,1,1,1,1|1,1,1,1,1,1,1,1,1,1,1,1|false
    - Multi-Agent Hide and Seek: https://www.youtube.com/watch?v=kopoLzvh5jY
