---
title: "Session 7 R practice (offline)"
output: html_notebook
author: "Dr. Richard M. Crowley"
date: "ACCT 420 Session 7"
---

## Calculating quantities with text

Note: Due to missing packages in DataCamp light, namely `quanteda`, `textdata`, and `tidytext`, I have provided sample code that you can run on your own computer in RStudio.  Make sure to run `install.packages("quanteda")`, `install.packages("textdata")`, and `install.packages("tidytext")` to install those packages if you don't have them.

Each of the three exercises below can be run as standalone scripts, as they contain all needed imports within their code blocks

### Exercise 6: Readability with Quanteda

How does the readability of JPMorgan's annual report compare to the Citigroup annual report from class?

```{r}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in quanteda
library(quanteda)

# Calculate the three readability measures
textstat_readability(doc, "Flesch.Kincaid")
textstat_readability(doc, "FOG")
textstat_readability(doc, "Coleman.Liau")

#END
```

### Exercise 7: Readability with Quanteda

How does the sentiment of JPMorgan's annual report compare to the Citigroup annual report from class?

```{r}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in tidytext
library(tidytext)

# Load some components of tidyverse
library(dplyr)  # for the usual commands
library(tidyr)  # for spread

# convert document to tidy format
df_doc <- data.frame(ID=c("0000019617-14-000289"), text=c(doc),
                     stringsAsFactors = F) %>%
  unnest_tokens(word, text)

# Calculate term frequency
terms <- df_doc %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)

# Get the Loughran McDonald sentiment dictionary
sentiment <- get_sentiments("loughran")

# Merge in sentiment
tf_sent <- tf %>% left_join(sentiment)

# Calculate the three readability measures
tf_sent %>%
  spread(sentiment, tf, fill=0) %>%
  select(constraining, litigious, negative, positive, superfluous, uncertainty) %>%
  colSums()

#END
```

### Exercise 8: Make a word cloud after removing stopwords

```{r}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in quanteda and tidytext
library(quanteda)
library(tidytext)

# Load in some of tidyverse
library(dplyr)

# convert document to tidy format
df_doc <- data.frame(ID=c("0000019617-14-000289"), text=c(doc),
                     stringsAsFactors = F) %>%
  unnest_tokens(word, text)

# Pull a list of stopwords
stopwords <- stopwords::stopwords(source="smart")

# Remove stopwords
df_doc_stop <- df_doc %>%
  anti_join(data.frame(word=stopwords, stringsAsFactors=F))

# Calculate term frequency
terms <- df_doc_stop %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)

# Build a corpus object for quanteda
corp <- cast_dfm(tf, ID, word, n)

# Plot a word cloud -- If you don't have RColorBrewer installed, you can
# remove the `color=` option.
textplot_wordcloud(dfm(corp), color = RColorBrewer::brewer.pal(9, "Set1"))

#END
```