---
title:
output:
  revealjs::revealjs_presentation:
    theme: white
    css: white-edits.css
    highlight: tango
    transition: slide
    self_contained: false
    reveal_plugins: ["chalkboard","search"]
    transitionSpead: fast
    mathjax: "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS_SVG"
    reveal_options:
      minScale: 1.0
      maxScale: 1.0
      width: 1536
      height: 864
      slideNumber: true
      controlsTutorial: true
      chalkboard:
        theme: whiteboard
        readOnly: false
---

## ACCT 420: Textual analysis {data-background-video="../Backgrounds/title.mp4" data-background-video-loop=true class="header no-toc-progress present-only" data-state="no-toc-progress"}

<!-- "reveal_plugins:..." and "chalkboard:" are handled by prep38.py -->
```{r helpers, echo=FALSE, warning=FALSE, message=F}
library(knitr)
library(kableExtra)
html_df <- function(text, cols=NULL, col1=FALSE, full=F) {
  if(!length(cols)) {
    cols=colnames(text)
  }
  if(!col1) {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full)
  } else {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full) %>%
      column_spec(1,bold=T)
  }
}
```

<h3 class="date"><br>Session 7</h3>
<h2 class="author"><br>Dr. Richard M. Crowley<br>rcrowley@smu.edu.sg<br><a href="http://rmc.link/">http://rmc.link/</a></h2>

## ACCT 420: Textual analysis {data-background="../Backgrounds/title.jpg" class="header no-toc-progress present-not" data-state="no-toc-progress"}

<h3 class="date"><br>Session 7</h3>
<h2 class="author"><br>Dr. Richard M. Crowley<br>rcrowley@smu.edu.sg<br><a href="http://rmc.link/">http://rmc.link/</a></h2>

<!-- Load in primary data set -->
```{r, include=FALSE}
library(tidyverse)

```

```{r, include=F, eval=F}
df <- read.csv("../../Data/Session_7.csv")
```

# Front matter {data-background="../Backgrounds/section.jpg" class="header"}

## Group projects {data-background="../Backgrounds/people.jpg" class="default present-only"}

> Decide on who you would like to work with by the end of tonight

> Submit your group preferences at [https://rmc.link/420groups2021](https://rmc.link/420groups2021)

- If you have a full group, that's great
- If you have a partial group, that's also fine
- If you need a group, there's an option for that on the form as well

## Group projects {data-background="../Backgrounds/people.jpg" class="default present-only"}

> Pre-defined projects available

1. Fraud detection of all frauds, not just those that got AAERs
    - Based on 10-K amendments
2. Forecasting of social media follower counts for S&P 1500 firms
    - Time series problem -- use 6 months of data to predict the next 3 months

> Or: Build your own from available data

- Annual and quarterly reports in the US, press releases, news articles
- Social media: company tweets, executive tweets, COVID tweets, Wall Street Bets tweets and Reddit posts

## Learning objectives {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
<img src="../Course_Diagrams/Map-7.png" width="500px">
</div>
<div class="right objectives">
- <b>Theory:</b>
    - Natural Language Processing
- <b>Application:</b>
    - Analyzing a Citigroup annual report
- <b>Methodology:</b>
    - Text analysis
    - Machine learning
</div>
<div class="clearfix">

## Datacamp {data-background="../Backgrounds/default.jpg" class="default"}

> - <a href="https://www.datacamp.com/courses/sentiment-analysis-in-r-the-tidy-way">Sentiment analysis in R the Tidy way</a>
>     - Just the first chapter is required
>     - You are welcome to do more, of course
> - I will generally follow the same "tidy text" principles as the Datacamp course does -- the structure keeps things easy to manage
>     - We will sometimes deviate to make use of certain libraries, which, while less tidy, make our work easier than the corresponding tidy-oriented packages (if they even exist!)

# Textual data and textual analysis {data-background="../Backgrounds/section.jpg" class="header"}

## Review of Session 6 {data-background="../Backgrounds/default.jpg" class="default"}

- Last session we saw that textual measures can help improve our fraud detection algorithm
- We actually looked at a bunch of textual measures:
    - Sentiment
    - Readability
    - Topic/content
- We didn't see how to make these though...
    - Instead, we had a nice premade dataset with everything already done

> We'll get started on these today -- sentiment and readability

> We'll cover topic modeling next session

## Why is textual analysis harder? {data-background="../Backgrounds/default.jpg" class="default"}

- Thus far, everything we've worked with is what is known as *structured data*
    - Structured data is numeric, nicely indexed, and easy to use
- Text data is *unstructured*
    - If we get an annual report with 200 pages of text...
        - Where is the information we want?
        - What do we want?
        - How do we crunch 200 pages into something that is...
            1. Manageable? (Structured)
            2. Meaningful?

> This is what we will work on today, and we will revist some of this in the remaining class sessions

## Structured data {data-background="../Backgrounds/default.jpg" class="default"}

- Our long or wide format data

<div class="clearfix">
<div class="left">
Wide format
```{r, echo=F, message=F}
# Note: read_csv is like read.csv, but loads as dplyr's format (tibble)
df_long <- read_csv("../../Data/general-business-expectations-by-detailed-services-industry-quarterly.csv")
df_long[1:3,c(1,4:5)]
```
</div>
<div class="right">
Long format
```{r, echo=F, message=F}
# Note: read_csv is like read.csv, but loads as dplyr's format (tibble)
df_wide <- read_csv("../../Assignments/Assignment_1/Zip_Zhvi_SingleFamilyResidence.csv")
df_wide[1:3,c(1,8:10)]
```
</div>
</div>

> The structure is given by the IDs, dates, and variables

## Unstructured data {data-background="../Backgrounds/default.jpg" class="default"}

- Text
    - Open responses to question, reports, etc.
    - What it isn't:
        - `"JANUARY"`, `"ONE"`, `"FEMALE"`
        - Months, numbers
        - Anything with clear and concise categories
- Images
    - Satellite imagery
- Audio
    - Phone call recordings 
- Video
    - Security camera footage

> All of these require us to determine and *impose* structure

## Some ideas of what we can do {data-background="../Backgrounds/default.jpg" class="default"}

1. Text extraction
    - Find all references to the CEO
    - Find if the company talked about global warming
    - Pull all telephone numbers or emails from a document
2. Text characteristics
    - How varied is the vocabulary?
    - Is it positive or negative (sentiment)
    - Is it written in a strong manner?
3. Text summarization or meaning
    - What is the content of the document?
    - What is the most important content of the document?
    - What other documents discuss similar issues?

## Where might we encounter text data in business {data-background="../Backgrounds/default.jpg" class="default"}

1. Business contracts
2. Legal documents
3. Any paperwork
4. News
5. Customer reviews or feedback
    - Including transcription (call centers)
6. Consumer social media posts
7. Chatbots and AI assistants

## Natural Language Processing (NLP) {data-background="../Backgrounds/default.jpg" class="default"}

- NLP is the subfield of computer science focused on analyzing large amounts of unstructured textual information
    - Much of the work builds from computer science, linguistics, and statistics
- Unstructured text actually has some structure derived from language itself
    - Word selection
    - Grammar
    - Phrases
    - Implicit orderings
- NLP utilizes this implicit structure to better understand textual data


## NLP in everyday life {data-background="../Backgrounds/default.jpg" class="default"}

- Autocomplete of the next word in phone keyboards
    - Demo below from <a href="https://www.blog.google/products/search/gboard-now-on-android/">Google's blog</a>
- Voice assistants like Google Assistant, Siri, Cortana, and Alexa
- Article suggestions on websites
- Search engine queries
- Email features like missing attachment detection

<img src="../Figures/gmail_demo.gif" height="350px">

## {data-background="../Figures/Gartner_AI_2020.png" data-background-size="contain"}

## Case: How leveraging NLP helps call centers {data-background="../Backgrounds/case.jpg" class="default"}

- <a href="https://medium.com/syncedreview/natural-language-processing-in-call-centres-b572da4da5dc">Natural Language Processing in Call Centres</a>
- Short link: <a href="https://rmc.link/420class7">rmc.link/420class7</a>


> What are call centers using NLP for?

> How does NLP help call centers with their business?

## Brainstorming... {data-background="../Backgrounds/rain.jpg" class="default present-only"}

<iframe allowfullscreen frameborder="0" height="600px" mozallowfullscreen src="https://app.wooclap.com/events/OYLTZU/questions/5f72e15c2f1630116bb2d6b3" width="100%"></iframe>

## Consider {data-background="../Backgrounds/group.jpg" class="default"}

> Where an we make use of NLP in business?

- We can use it for call centers
- We can make products out of it (like <a href="https://youtu.be/D5VN56jQMWM?t=68">Google Duplex</a> and other tech firms)
- Where else?

## Brainstorming... {data-background="../Backgrounds/rain.jpg" class="default present-only"}

<iframe allowfullscreen frameborder="0" height="600px" mozallowfullscreen src="https://app.wooclap.com/events/OYLTZU/questions/5f72eef3489b2e495a379376" width="100%"></iframe>

# Working with 1 text file {data-background="../Backgrounds/section.jpg" class="header"}

## Before we begin: Special characters {data-background="../Backgrounds/reference.jpg" class="default"}

- Some characters in R have special meanings for string functions
    - `\ | ( ) [ { } ^ $ * + ? . !`
- To type a special character, we need to precede it with a `\`
    - Since `\` is a special character, we'll need to put `\` before `\`...
        - To type `$`, we would use `\\$`
- Also, some spacing characters have special symbols:
    - `\t` is tab
    - `\r` is newline (files from Macs)
    - `\r\n` is newline (files from Windows)
    - `\n` is newline (files from Unix, Linux, etc.)
    - You'll need to write `\\` to get the backslashes though

## Loading in text data from files {data-background="../Backgrounds/reference.jpg" class="default"}

- Use `read_file()` from `package:tidyverse`'s `package:readr` package to read in text data
- We'll use <a href="https://www.sec.gov/Archives/edgar/data/831001/000110465914015152/0001104659-14-015152-index.htm">Citigroup's annual report from 2014</a>
    - Note that there is a full text link at the bottom which is a .txt file
    - I will instead use a cleaner version derived from the linked file
        - The cleaner version can be made using the same techniques we will discuss today

```{r}
# Read text from a .txt file using read_file()
doc <- read_file("../../Data/0001104659-14-015152.txt")
# str_wrap is from stringr from tidyverse
cat(str_wrap(substring(doc,1,500), 80))
```

## Loading from other file types {data-background="../Backgrounds/default.jpg" class="default"}

- Ideally you have a .txt file already -- such files are generally just the text of the documents
- Other common file types:
    - HTML files (particularly common from web data)
        - You can load it as a text file -- just note that there are html tags embedded in it
            - Things like `<a>`, `<table>`, `<img>`, etc.
        - You can load from a URL using `package:httr` or `package:RCurl`
        - In R, you can use `package:XML` or `package:rvest` to parse out specific pieces of html files
        - If you use python, use lxml or BeautifulSoup 4 (bs4) to quickly turn these into structured documents
        - In R, you can process JSON data using `package:jsonlite`

## Loading from other file types {data-background="../Backgrounds/default.jpg" class="default"}

- Ideally you have a .txt file already -- such files are generally just the text of the documents
- Other common file types:
    - PDF files
        - Use `package:pdftools` to extract text into a vector of pages of text
        - Use `package:tabulizer` to extract tables straight from PDF files!
            - This is very painful to code by hand without this package
            - The package itself is a bit difficult to install, requiring Java and `package:rJava`, though

## Example using html {data-background="../Backgrounds/default.jpg" class="default"}

```{r, message=F, warning=F}
library(httr)
library(XML)

httpResponse <- GET('https://coinmarketcap.com/currencies/ethereum/')
html = httr::content(httpResponse, "text")
paste0('...', str_wrap(substring(html, 135418, 135480), 80), '...')
xpath <- '//*[@id="__next"]/div/div[1]/div[2]/div/div[1]/div[2]/div/div[2]/div[1]/div/text()'
hdoc = htmlParse(html, asText=TRUE)  # from XML
price <- xpathSApply(hdoc, xpath, xmlValue)
print(paste0("Ethereum was priced at $", price,
             " when these slides were compiled"))
```

## Automating crypto pricing in a document {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
# The actual version I use (with caching to avoid repeated lookups) is in the appendix
cryptoMC <- function(name) {
  httpResponse <- GET(paste('https://coinmarketcap.com/currencies/',name,'/',sep=''))
  html = httr::content(httpResponse, "text")
  xpath <- '//*[@id="__next"]/div/div[1]/div[2]/div/div[1]/div[2]/div/div[2]/div[1]/div/text()'
  hdoc = htmlParse(html, asText=TRUE)
  plain.text <- xpathSApply(hdoc, xpath, xmlValue)
  plain.text
}
```

```{r}
paste("Ethereum was priced at", cryptoMC("ethereum"))
paste("Litecoin was priced at", cryptoMC("litecoin"))
```

## Basic text functions in R {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
- Subsetting text
- Transformation
    - Changing case
    - Adding or combining text
    - Replacing text
    - Breaking text apart
- Finding text
</div>
<div class="right">
<img src="../Figures/stringr.png" height="200px">
</div>
</div>

> We will cover these using `package:stringr` as opposed to base R -- `package:stringr`'s commands are much more consistent

- Every function in `package:stringr` can take a vector of strings for the first argument, which is *tidy*

## Subsetting text {data-background="../Backgrounds/reference.jpg" class="default"}

- Base R: Use `substr()` or `substring()`
- `package:stringr`: use `str_sub()`
    - First argument is a vector of strings
    - Second argument is the starting position (inclusive)
    - Third argument is that ending position (inclusive)

```{r}
cat(str_wrap(str_sub(doc, 9896, 9929), 80))
cat(str_wrap(str_sub(doc, 28900,29052), 80))
```

## Transforming text {data-background="../Backgrounds/reference.jpg" class="default"}

- Commonly used functions:
    - `tolower()` or `str_to_lower()`: make the text lowercase
    - `toupper()` or `str_to_upper()`: MAKE THE TEXT UPPERCASE
    - `str_to_title()`: Make the Text Titlecase
- `paste()` to combine text
    - It puts spaces between by default
        - You can change this with the `sep=` option
    - If everything to combine is in 1 vector, use `collapse=` with the desired separator
    - `paste0()` is paste with `sep=""`

## Examples: Case {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
sentence <- str_sub(doc, 9896, 9929)
str_to_lower(sentence)
str_to_upper(sentence)
str_to_title(sentence)
```

- The `str_` prefixed functions support non-English languages as well

```{r, eval=F}
# You can run this in an R terminal! (It doesn't work in Rmarkdown though)
str_to_upper("Citis net income was $13.5 billion", locale='tr')  # Turkish
```

## Examples: `paste` {data-background="../Backgrounds/default.jpg" class="default"}

```{r, include=F}
board <- c("Michael L. Corbat", "Michael E. O’Neill", "Anthony M. Santomero",
           "William S. Thompson, Jr.", "Duncan P. Hennes", "Gary M. Reiner",
           "Joan E. Spero", "James S. Turley", "Franz B. Humer",
           "Judith Rodin", "Robert L. Ryan", "Diana L. Taylor",
           "Ernesto Zedillo Ponce de Leon", "Robert L. Joss")
titles <- c("CEO", "Chairman", "Former president, Fed (Philidelphia)",
            "CEO, Retired, PIMCO", "Co-Founder/Partner, Atrevida Partners",
            "Operating Partner, General Atlantic",
            "Senior Research Scholar, Columbia University",
            "Former Chairman & CEO, E&Y", "Chairman, Roche",
            "President, Rockefeller Foundation", "CFO, Retired, Medtronic",
            "MD, Wolfensohn Fund Management", "Professor, Yale University",
            "Professor/Dean Emeritus, Stanford GSB")
```
```{r}
# board is a list of director names
# titles is a list of the director's titles
paste(board, titles, sep=", ")
cat(str_wrap(paste0("Citi's board consists of: ",
                    paste(board[1:length(board)-1], collapse=", "),
                    ", and ", board[length(board)], "."), 80))
```

## Transforming text {data-background="../Backgrounds/reference.jpg" class="default"}

- Replace text with `str_replace_all()`
    - First argument is text data
    - Second argument is what you want to remove
    - Third argument is the replacement
- If you only want to replace the first occurrence, use `str_replace()` instead

```{r}
sentence
str_replace_all(sentence, "\\$13.5", "over $10")
```

## Transforming text {data-background="../Backgrounds/reference.jpg" class="default"}

- Split text using `str_split()`
    - This function returns a list of vectors!
        - This is because it will turn every string passed to it into a vector, and R can't have a vector of vectors
    - `[[1]]` can extract the first vector
- You can also limit the number of splits using `n=`
    - A bit more elegant solution is using `str_split_fixed()` with `n=`
        - Returns a character matrix (nicer than a list)

## Example: Splitting text {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
paragraphs <- str_split(doc, '\n')[[1]]

# number of paragraphs
length(paragraphs)

# Last paragraph
cat(str_wrap(paragraphs[206], 80))
```

## Finding phrases in text {data-background="../Backgrounds/default.jpg" class="default"}

- How did I find the previous examples?

```{r}
str_locate_all(str_to_lower(doc), "net income")
```

## Finding phrases in text {data-background="../Backgrounds/reference.jpg" class="default"}

- 4 primary functions:
    1. `str_detect()`: Reports `TRUE` or `FALSE` for the presence of a string in the text
    2. `str_count()`: Reports the number of times a string is in the text
    3. `str_locate()`: Reports the first location of a string in the text
        - `str_locate_all()`: Reports every location as a list of matrices
    4. `str_extract()`: Reports the matched phrases
- All take a character vector as the first argument, and something to match for the second argument

## Example: Finding phrases {data-background="../Backgrounds/default.jpg" class="default"}

- How many paragraphs mention net income in any case?

```{r}
x <- str_detect(str_to_lower(paragraphs), "net income")
x[1:10]
sum(x)
```

- What is the most net income is mentioned in any paragraph

```{r}
x <- str_count(str_to_lower(paragraphs), "net income")
x[1:10]
max(x)
```

## Example: Finding phrases {data-background="../Backgrounds/default.jpg" class="default"}

- Where is net income first mentioned in the document?

```{r}
str_locate(str_to_lower(doc), "net income")
```

- First mention of net income
    - This function may look useless now, but it'll be on of the most useful later

```{r}
str_extract(str_to_lower(doc), "net income")
```

## R Practice {data-background="../Backgrounds/coding.jpg" class="default"}

- Text data is already loaded, as if it was loaded using `read_file()`
- Try:
    - Subsetting the text data
    - Transforming the text data
        - To all upper case
        - Replacing a phrase
    - Finding specific text in the document
- Do exercises 1 through 3 in today's practice file
    - <a href="http://rmc.link/Slides/acct420v4/Session_7/Session_7_R.html">R Practice</a>
    - Shortlink: <a href="http://rmc.link/Slides/acct420v4/Session_7/Session_7_R.html">rmc.link/420r7</a>

# Pattern matching {data-background="../Backgrounds/section.jpg" class="header"}

## Finding *patterns* in the text (regex) {data-background="../Backgrounds/default.jpg" class="default"}

- Regular expressions, aka regex or regexp, are ways of finding patterns in text
- This means that instead of looking for a specific phrase, we can match a set of phrases
- Most of the functions we discussed accept regexes for matching
    - `str_replace()`, `str_split()`, `str_detect()`, `str_count()`, `str_locate()`, and `str_extract()`, plus their variants
- This is why `str_extract()` is so great -- we can extract anything from a document with it!

## Regex example {data-background="../Backgrounds/default.jpg" class="default"}

- Breaking down an email
    1. A local name
    2. An `@` sign
    3. A domain, which will have a `.` in it
- Local names can have many different characters in them
    - Match it with `[:graph:]+`
- The domain is pretty restrictive, generally just alphanumeric and `.`
    - There can be multiple `.` though
    - Match it with `[:alnum:]+\\.[.[:alnum:]]+`

```{r}
# Extract all emails from the annual report
str_extract_all(doc,'[:graph:]+@[:alnum:]+\\.[.[:alnum:]]+')
```

## Breaking down the example {data-background="../Backgrounds/default.jpg" class="default"}

- `@` was itself -- it isn't a special character in strings in R
- `\\.` is just a period -- we need to escape `.` because it is special
- Anything in brackets with colons, `[: :]`, is a set of characters
    - `[:graph:]` means any letter, number, or punctuation
    - `[:alnum:]` means any letter or number
- `+` is used to indicate that we want 1 or more of the preceding element -- as many as it can match
    - `[:graph:]+` meant "Give us every letter, number, and punctuation you can, but make sure there is at least 1."
- Brackets with no colons, `[ ]`, ask for anything inside
    - `[.[:alnum:]]+` meant "Give us every letter, number, and `.` you can, but make sure there is at least 1."

## Breaking down the example {data-background="../Backgrounds/default.jpg" class="default"}

- Let's examine the output `shareholder@computershare.com`
- Our regex was `[:graph:]+@[:alnum:]+\\.[.[:alnum:]]+`
- Matching regex components to output:
    - `[:graph:]+` $\Rightarrow$ `shareholder`
    - `@` $\Rightarrow$ `@`
    - `[:alnum:]+` $\Rightarrow$ `computershare`
    - `\\.` $\Rightarrow$ `.`
    - `[.[:alnum:]]+` $\Rightarrow$ `com`

## Useful regex components: Content {data-background="../Backgrounds/reference.jpg" class="default"}

- There's a <a href="https://github.com/rstudio/cheatsheets/raw/master/strings.pdf">nice cheat sheet here</a>
    - <a href="https://cran.r-project.org/web/packages/stringr/vignettes/regular-expressions.html">More detailed documentation here</a>
- Matching collections of characters
    - `.` matches everything
    - `[:alpha:]` matches all letters
    - `[:lower:]` matches all lowercase letters
    - `[:upper:]` matches all UPPERCASE letters
    - `[:digit:]` matches all numbers 0 through 9
    - `[:alnum:]` matches all letters and numbers
    - `[:punct:]` matches all punctuation
    - `[:graph:]` matches all letters, numbers, and punctuation
    - `[:space:]` or `\s` match ANY whitespace
        - `\S` is the exact opposite
    - `[:blank:]` matches whitespace except newlines

## Example: Regex content {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
text <- c("abcde", 'ABCDE', '12345', '!?!?.', 'ABC123?', "With space", "New\nline")
html_df(data.frame(
  text=text,
  alpha=str_detect(text,'[:alpha:]'),
  lower=str_detect(text,'[:lower:]'),
  upper=str_detect(text,'[:upper:]'),
  digit=str_detect(text,'[:digit:]'),
  alnum=str_detect(text,'[:alnum:]')
))

```

## Example: Regex content {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
text <- c("abcde", 'ABCDE', '12345', '!?!?.', 'ABC123?', "With space", "New\nline")
html_df(data.frame(
  text=text,
  punct=str_detect(text,'[:punct:]'),
  graph=str_detect(text,'[:graph:]'),
  space=str_detect(text,'[:space:]'),
  blank=str_detect(text,'[:blank:]'),
  period=str_detect(text,'.')
))

```

## Useful regex components: Form {data-background="../Backgrounds/reference.jpg" class="default"}

- `[ ]` can be used to create a class of characters to look for
    - `[abc]` matches anything that is `a`, `b`, `c`
- `[^ ]` can be used to create a class of everything else
    - `[^abc]` matches anything that isn't `a`, `b`, or `c`
- Quantity, where `x` is some element
    - `x?` looks for 0 or 1 of `x`
    - `x*` looks for 0 or more of `x`
    - `x+` looks for 1 or more of `x`
    - `x{n}` looks for `n` (a number) of `x`
    - `x{n, }` looks for at least `n` of `x`
    - `x{n,m}` looks for at least `n` and at most `m` of `x`
- Lazy operators
    - Regexes always prefer the longest match by default
    - Append `?` to any quantity operator to make it prefer the shortest match possible

## Useful regex components: Form {data-background="../Backgrounds/reference.jpg" class="default"}

- Position
    - `^` indicates the start of the string
    - `$` indicates the end of the string
- Grouping
    - `( )` can be used to group components
    - `|` can be used within groups as a logical *or*
    - Groups can be referenced later using the position of the group within the regex
        - `\\1` refers to the first group
        - `\\2` refers to the second group
        - ...

## Example: Regexes on real estate firm names {data-background="../Backgrounds/default.jpg" class="default"}

```{r, include=FALSE}
df_RE <- read.csv("../../Data/Session_2-1.csv")
RE_names <- unique(df_RE$conm)
```

```{r}
# Real estate firm names with 3 vowels in a row
str_subset(RE_names, '[AEIOU]{3}')
# Real estate firm names with no vowels
str_subset(RE_names, '^[^AEIOU]+$')
# Real estate firm names with at least 12 vowels
str_subset(RE_names, '([^AEIOU]*[AEIOU]){11,}')
# Real estate firm names with a repeated 4 letter pattern
str_subset(RE_names, '([:upper:]{4}).*\\1')
```

## Why is regex so important? {data-background="../Backgrounds/default.jpg" class="default"}

- Regex can be used to match anything in text
    - Simple things like phone numbers
    - More complex things like addresses
- It can be used to parse through large markup documents
    - HTML, XML, LaTeX, etc.
- Very good for validating the format of text
    - For birthday in the format YYYYMMDD, you could validate with:
        - YYYY: `[12][90][:digit:][:digit:]`
        - MM: `[01][:digit:]`
        - DD: `[0123][:digit:]`

> Cavaet: Regexes are generally slow.  If you can code something to avoid them, that is often better.  But often that may be infeasible.

## Some extras {data-background="../Backgrounds/default.jpg" class="default"}

- While the `str_*()` functions use regex by default, they actually have four modes
    1. You can specify a regex normally
        - Or you can use `regex()` to construct more customized ones, such as regexes that operate by line in a string
    2. You can specify an exact string to match using `fixed()` -- fast but fragile
    3. You can specify an exact string to match using `coll()` -- slow but robust; recognizes characters that are equivalent
        - Important when dealing with non-English words, since certain characters can be encoded in multiple ways
    4. You can ask for boundaries with `boundary()` such as words, using `boundary("word")`


## Expanding usage {data-background="../Backgrounds/default.jpg" class="default"}

- Anything covered so far can be used for text in data
    - Ex.: Firm names or addresses in Compustat

```{r}
# Compustat firm names example
df_RE_names <- df_RE %>%
  group_by(isin) %>%
  slice(1) %>%
  mutate(SG_in_name = str_detect(conm, "(SG|SINGAPORE)"),
         name_length = str_length(conm),
         SG_firm = ifelse(fic=="SGP",1,0)) %>%
  ungroup()

df_RE_names %>%
  group_by(SG_firm) %>%
  mutate(pct_SG = mean(SG_in_name) * 100) %>%
  slice(1) %>%
  ungroup() %>%
  select(SG_firm, pct_SG)
```

## Expanding usage {data-background="../Backgrounds/default.jpg" class="default"}

```{r, warning=F}
library(DT)
df_RE_names %>%
  group_by(fic) %>%
  mutate(avg_name_length = mean(name_length)) %>%
  slice(1) %>%
  ungroup() %>%
  select(fic, avg_name_length) %>%
  arrange(desc(avg_name_length), fic) %>%
  datatable(options = list(pageLength = 5))
```

## R Practice 2 {data-background="../Backgrounds/coding.jpg" class="default"}

- This practice explores the previously used practice data using regular expressions for various purposes
- Do exercises 4 and 5 in today's practice file
    - <a href="http://rmc.link/Slides/acct420v4/Session_7/Session_7_R.html">R Practice</a>
    - Shortlink: <a href="http://rmc.link/Slides/acct420v4/Session_7/Session_7_R.html">rmc.link/420r7</a>


# Readability and Sentiment {data-background="../Backgrounds/section.jpg" class="header"}

## Readability {data-background="../Backgrounds/default.jpg" class="default"}

- Thanks to the `package:quanteda` package, readability is very easy to calculate in R
    - Use the `textstat_readability()` function
- There are many readability measures, however
    - Flesch Kinkaid grade level: A measure of readability developed for the U.S. Navy to ensure manuals were written at a level any 15 year old should be able to understand
    - Fog: A grade level index that was commonly used in business and publishing
    - Coleman-Liau: An index with a unique calculation method, relying only on character counts

## Readability: Flesch Kincaid {data-background="../Backgrounds/default.jpg" class="default"}

$$
0.39 \left(\frac{\#~words}{\#~sentences}\right) + 11.8\left(\frac{\#~syllables}{\#~words}\right) - 15.59
$$

- An approximate grade level required for reading a document
    - *Lower is more readable*
    - A JC or poly graduate should read at a level of 12
        - New York Times articles are usually around 13
    - A Bachelor's degree could be necessary for anything 16 or above

```{r, message=F}
library(quanteda)
library(quanteda.textstats)
textstat_readability(doc, "Flesch.Kincaid")
```

## Readability: Fog {data-background="../Backgrounds/default.jpg" class="default"}

$$
\begin{aligned}
  & \left[ Mean(Words~per~sentence) +\right.\\
  &\left.(\%~of~words~>3~syllables)\right] \times 0.4
\end{aligned}
$$

- An approximate grade level required for reading a document
    - *Lower is more readable*

```{r}
textstat_readability(doc, "FOG")
```

## Readability: Coleman-Liau {data-background="../Backgrounds/default.jpg" class="default"}

$$
5.88\left(\frac{\#~letters}{\#~words}\right)-29.6\left(\frac{\#~sentences}{\#~words}\right)-15.8
$$

- An approximate grade level required for reading a document
    - *Lower is more readable*

```{r}
textstat_readability(doc, "Coleman.Liau.short")
```


## Converting text to words {data-background="../Backgrounds/default.jpg" class="default"}

- Tidy text is when you have one *token* per document per row, in a data frame
- *Token* is the unit of text you are interested in
    - Words: "New"
    - Phrases: "New York Times"
    - Sentences: "The New York Times is a publication."
    - etc.
- The `package:tidytext` package can handle this conversion for us!
    - Use the `unnest_tokens()` function
    - Note: it also converts to lowercase.  Use the option `to_lower=FALSE` to avoid this if needed

```{r, eval=T, warning=F}
# Example of "tokenizing"
library(tidytext)
df_doc <- data.frame(ID=c("0001104659-14-015152"), text=c(doc)) %>%
  unnest_tokens(word, text)
# word is the name for the new column
# text is the name of the string column in the input data
```

## The details {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
html_df(head(df_doc))
```

## The details {data-background="../Backgrounds/default.jpg" class="default"}

- `package:tidytext` uses the `package:tokenizers` package in the backend to do the conversion
    - You can call that package directly instead if you want to
- Available tokenizers include: (specify with `token=`)
    - "word": The default, individual words
    - "ngram": Collections of words (default of 2, specify with `n=`)
    - A few other less commonly used tokenizers

## Word case {data-background="../Backgrounds/spooky.jpg" class="default"}

- Why convert to lowercase?
- How much of a difference is there between "The" and "the"?
    - "Singapore" and "singapore" -- still not much difference
    - Only words like "new" versus "New" matter
        - "New York" versus "new yorkshire terrier"
- Benefit: We get rid of a bunch of distinct words!
    - Helps with *the curse of dimensionality*

## The Curse of dimensionality {data-background="../Backgrounds/spooky.jpg" class="default"}

- There are a lot of words
- A LOT OF WORDS
- At least 171,476 according to <a href="https://en.oxforddictionaries.com/explore/how-many-words-are-there-in-the-english-language/">Oxford Dictionary</a>
- What happens if we make a matrix of words per document?

<div class="fragment exclude">
For right now, not much
</div>

<div class="fragment exclude">
- If we have every publicly available government filed press release in the US?
    - 1,479,068 files through July 2018...
        - ~2TB if we include all English words
        - ~70GB if we restrict just to the 5,884 words in the Citigroup annual report...
</div>


## Stopwords {data-background="../Backgrounds/default.jpg" class="default"}

- Stopwords -- words we remove because they have little content
    - the, a, an, and, ...
- Also helps with our curse a bit -- removes the words entirely
- We'll use the `stopword` package to remove stopwords

```{r, warning=F}
# get a list of stopwords
stop_en <- stopwords::stopwords("english")  # Snowball English
paste0(length(stop_en), " words: ", paste(stop_en[1:5], collapse=", "))
stop_SMART <- stopwords::stopwords(source="smart")  # SMART English
paste0(length(stop_SMART), " words: ", paste(stop_SMART[1:5], collapse=", "))
stop_fr <- stopwords::stopwords("french")  # Snowball French
paste0(length(stop_fr), " words: ", paste(stop_fr[1:5], collapse=", "))
```

## Applying stopwords to a corpus {data-background="../Backgrounds/default.jpg" class="default"}

- When we have a tidy set of text, we can just use `package:dplyr` for this!
    - `package:dplyr`'s `anti_join()` function is like a merge, but where all matches are deleted
    

```{r}
df_doc_stop <- df_doc %>%
  anti_join(data.frame(word=stop_SMART))
nrow(df_doc)
nrow(df_doc_stop)
```

## Converting to term frequency {data-background="../Backgrounds/default.jpg" class="default"}

```{r, warning=F, message=F}
terms <- df_doc_stop %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)
tf
```

```{r, echo=F, warning=F, error=F, message=F}
terms_ns <- df_doc %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms_ns <- terms_ns %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf_no_stop <- left_join(terms_ns, total_terms_ns) %>% mutate(tf=n/total)
```

## Sentiment {data-background="../Backgrounds/default.jpg" class="default"}

- Sentiment works similarly to stopwords, except we are identifying words with specific, useful meanings
    - We can grab off-the-shelf sentiment measures using `get_sentiments()` from `package:tidytext`

<div class="clearfix">
<div class="left">
```{r}
# Need to install the `textdata` package for the below to work
get_sentiments("afinn") %>%
  group_by(value) %>%
  slice(1) %>%
  ungroup()
```
</div>
<div class="right">
```{r}
get_sentiments("bing") %>%
  group_by(sentiment) %>%
  slice(1) %>%
  ungroup()
```
</div>
</div>

## Sentiment {data-background="../Backgrounds/default.jpg" class="default"}

<div class="clearfix">
<div class="left">
```{r}
get_sentiments("nrc") %>%
  group_by(sentiment) %>%
  slice(1) %>%
  ungroup()
```
</div>
<div class="right">
Loughran & McDonald dictionary -- finance specific, targeted at annual reports
```{r}
get_sentiments("loughran") %>%
  group_by(sentiment) %>%
  slice(1) %>%
  ungroup()
```
</div>
</div>

## Merging in sentiment data {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
tf_sent <- tf %>% left_join(get_sentiments("loughran"))
tf_sent[1:5,]
tf_sent[!is.na(tf_sent$sentiment),][1:5,]
```

## Summarizing document sentiment {data-background="../Backgrounds/default.jpg" class="default"}

```{r}
tf_sent %>%
  spread(sentiment, tf, fill=0) %>%
  select(constraining, litigious, negative, positive, superfluous, uncertainty) %>%
  colSums()
```

## visualizing sentiment {data-background="../Backgrounds/default.jpg" class="default"}

```{r, warning=F, message=F, echo=F}
#library(plotly)
tf_sent %>%
  filter(!is.na(sentiment)) %>%
  group_by(sentiment) %>%
  arrange(desc(n)) %>%
  mutate(row = row_number()) %>%
  filter(row < 10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(y=n, x=word)) + geom_col() + theme(axis.text.x = element_text(angle=90, hjust=1)) + 
  facet_wrap(~sentiment, ncol=3, scales="free_x")
#ggplotly(p)
```


## Visualizing a document as a word cloud {data-background="../Backgrounds/default.jpg" class="default"}

- `package:quanteda` provides `textplot_wordcloud()`
- `cast_dfm()` converts tidy term frequencies to Quanteda
- There are also the `package:wordcloud` and `package:wordcloud2` packages for this

```{r, message=F, warning=F, fig.height=4.5}
library(quanteda.textplots)
corp <- cast_dfm(tf, ID, word, n)
textplot_wordcloud(dfm(corp), color = RColorBrewer::brewer.pal(9, "Set1"))
```

## Another reason to use stopwords {data-background="../Backgrounds/default.jpg" class="default"}

- Without removing stopwords, the word cloud shows almost nothing useful

```{r, message=F, warning=F, fig.height=4.5}
corp_no_stop <- cast_dfm(tf_no_stop, ID, word, n)
textplot_wordcloud(dfm(corp_no_stop), color = RColorBrewer::brewer.pal(9, "Set1"))
```

## R Practice 3 {data-background="../Backgrounds/coding.jpg" class="default"}

- Using the same data as before, we will explore
    - Readability
    - Sentiment
    - Word clouds
- Note: Due to missing packages, you will need to run the code in RStudio, not in the DataCamp light console
- Do exercises 6 through 8 in today's practice file
    - <a href="http://rmc.link/Slides/acct420v4/Session_7/Session_7_R.html">R Practice</a>
    - Shortlink: <a href="http://rmc.link/Slides/acct420v4/Session_7/Session_7_R.html">rmc.link/420r7</a>

## What's next {data-background="../Backgrounds/default.jpg" class="default exclude"}

- Armed with an understanding of how to process unstructured data, all of the sudden the amount of data available to us is expanding rapidly
- To an extent, anything in the world can be viewed as data, which can get overwhelming pretty fast
- We'll require some better and newer tools to deal with this
- Revisiting last session...
    - Last session we used *topics* in our analysis
    - Topics were derived from the 10-Ks we have been working with

# End matter {data-background="../Backgrounds/ending.jpg" class="header"}

## For next week {data-background="../Backgrounds/ending.jpg" class="default"}

- For next week:
    - Finish the third assignment
        - Submit on eLearn
    - Datacamp
        - Do the assigned chapter on text analysis
    - Start on the group project
- Survey on the class session at <a href="https://docs.google.com/forms/d/e/1FAIpQLScM5KvNYSG5Z5EJQS8F9WcpvjNTfTDxSdYltxRfxDRjeuYOhQ/viewform?usp=sf_link">rmc.link/420survey7</a>

## Packages used for these slides {data-background="../Backgrounds/code.jpg" class="default"}

- `package:httr`
- `package:kableExtra`
- `package:knitr`
- `package:magrittr`
- `package:quanteda`
- `package:RColorBrewer`
- `package:readtext`
- `package:revealjs`
- `package:tidytext`
- `package:tidyverse`
    - `package:dplyr`, `package:readr`, `package:stringr`
- `package:XML`

## References {data-background="../Backgrounds/books-old.jpg" class="default stack never"}

## Custom code {data-background="../Backgrounds/code.jpg" class="default"}

<small style="width:100%">

```{r,eval=F}
library(knitr)
library(kableExtra)
html_df <- function(text, cols=NULL, col1=FALSE, full=F) {
  if(!length(cols)) {
    cols=colnames(text)
  }
  if(!col1) {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full)
  } else {
    kable(text,"html", col.names = cols, align = c("l",rep('c',length(cols)-1))) %>%
      kable_styling(bootstrap_options = c("striped","hover"), full_width=full) %>%
      column_spec(1,bold=T)
  }
}
```

```{r,eval=F}
cryptoMC <- function(name) {
  if (exists(name)) {
    get(name)
  } else{
  html <- getURL(paste('https://coinmarketcap.com/currencies/',name,'/',sep=''))
  xpath <- '//*[@id="__next"]/div[1]/div[2]/div[1]/div[2]/div[1]/div/div[2]/span[1]/span[1]/text()'
  doc = htmlParse(html, asText=TRUE)
  plain.text <- xpathSApply(doc, xpath, xmlValue)
  assign(name, gsub("\n","",gsub(" ", "", paste(plain.text, collapse = ""), fixed = TRUE), fixed = TRUE),envir = .GlobalEnv)
  get(name)
  }
}
```

```{r, eval=F}
# Create a plot of the top words by sentiment
tf_sent %>%
  filter(!is.na(sentiment)) %>%
  group_by(sentiment) %>%
  arrange(desc(n)) %>%
  mutate(row = row_number()) %>%
  filter(row < 10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(y=n, x=word)) + geom_col() + theme(axis.text.x = element_text(angle=90, hjust=1)) + 
  facet_wrap(~sentiment, ncol=3, scales="free_x")
```
</small>

## NOTES {class="never"}

- Update Gartner AI diagram (perpetual)
    - Update seems to be on a 1 year lag
- Run all `get_sentiments()` code interactively first to accept licenses and install data (perpetual)

- Pretty long; data processing to percentage and the curse of dimensionality stuff could be tighter/more time efficient.

- Add in an example of CSR discussion