---
title: "Session 7 R practice (offline)"
output: html_notebook
author: "Dr. Richard M. Crowley"
date: "ACCT 420 Session 7"
---

## Calculating quantities with text

Note: Due to missing packages in DataCamp light, namely `quanteda`, `quanteda.textstats`, `quanteda.textplots`, `textdata`, and `tidytext`, I have provided sample code that you can run on your own computer in RStudio.  Make sure to run `install.packages("quanteda")`, `install.packages("textdata")`, and `install.packages("tidytext")` to install those packages if you don't have them.

Each of the three exercises below can be run as standalone scripts, as they contain all needed imports within their code blocks

Code for this section can be <a href="./Session_7_R_offline.Rmd">downloaded as an Rmd file here</a>.  The output of this code can be viewed in <a href="./Session_7_R_offline.nb.html">this Rmarkdown notebook</a>.

### Exercise 6: Readability with Quanteda

How does the readability of JPMorgan's annual report compare to the Citigroup annual report from class?

```{r ex="create_g", type="sample-code"}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v4/Session_7/0000019617-14-000289.txt")

# Load in quanteda
library(quanteda)
library(quanteda.textstats)

# Calculate the three readability measures
textstat_readability(doc, "Flesch.Kincaid")
textstat_readability(doc, "FOG")
textstat_readability(doc, "Coleman.Liau")

#END
```

### Exercise 7: Sentiment with Quanteda

How does the sentiment of JPMorgan's annual report compare to the Citigroup annual report from class?

```{r ex="create_h", type="sample-code"}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in tidytext
library(tidytext)

# Load some components of tidyverse
library(dplyr)  # for the usual commands
library(tidyr)  # for spread

# convert document to tidy format
df_doc <- data.frame(ID=c("0000019617-14-000289"), text=c(doc),
                     stringsAsFactors = F) %>%
  unnest_tokens(word, text)

# Calculate term frequency
terms <- df_doc %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)

# Get the Loughran McDonald sentiment dictionary
sentiment <- get_sentiments("loughran")

# Merge in sentiment
tf_sent <- tf %>% left_join(sentiment)

# Calculate the three readability measures
tf_sent %>%
  spread(sentiment, tf, fill=0) %>%
  select(constraining, litigious, negative, positive, superfluous, uncertainty) %>%
  colSums()

#END
```

### Exercise 8: Make a word cloud after removing stopwords

```{r ex="create_i", type="sample-code"}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in quanteda and tidytext
library(quanteda)
library(quanteda.textplots)
library(tidytext)

# Load in some of tidyverse
library(dplyr)

# convert document to tidy format
df_doc <- data.frame(ID=c("0000019617-14-000289"), text=c(doc),
                     stringsAsFactors = F) %>%
  unnest_tokens(word, text)

# Pull a list of stopwords
stopwords <- stopwords::stopwords(source="smart")

# Remove stopwords
df_doc_stop <- df_doc %>%
  anti_join(data.frame(word=stopwords, stringsAsFactors=F))

# Calculate term frequency
terms <- df_doc_stop %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)

# Build a corpus object for quanteda
corp <- cast_dfm(tf, ID, word, n)

# Plot a word cloud -- If you don't have RColorBrewer installed, you can
# remove the `color=` option.
textplot_wordcloud(dfm(corp), color = RColorBrewer::brewer.pal(9, "Set1"))

#END
```