---
title: "Session 7 R practice"
author: "Dr. Richard M. Crowley"
date: "ACCT 420, Session 7"
output:
  html_document:
    theme: spacelab
    highlight: default
    toc: true
    toc_float:
      collapsed: false
    self_contained: false
---

```{r, include=FALSE}
tutorial::go_interactive()
```

## Working with text data

To run your code, click *run*.  It will let you know if your code is correct or not, and will offer hints on what to fix if needed.  If you are stuck, you can see the solution by clicking *Solution*.

### Exercise 1: Subsetting data

```{r ex="create_b", type="pre-exercise-code"}
library(dplyr)
doc <- "On June 5, 2013, the SEC approved the publication of proposed structural reforms of money market funds. The proposal considered two reform alternatives that could be adopted either alone or in combination: (i) requiring prime and tax-exempt institutional money market funds to float their net asset values or (ii) requiring all non-governmental money market funds to impose liquidity fees of up to 2% and to have the option to temporarily suspend redemptions (or gate the money market fund) upon the occurrence of specified events indicating that the fund may be under stress. It is currently anticipated that the SEC will adopt final structural reforms in 2014. The Financial Stability Board (the FSB) has endorsed and published for public consultation 15 policy recommendations proposed by the International Organization of Securities Commissions, including requiring money market funds to adopt a floating net asset value. In addition, in September 2013 the European Commission (the EC) released a proposal for a new regulation on money market funds in the EU. The EC proposed two options for stable net asset value money market funds: either (i) maintain a capital buffer of at least three percent of assets under management or (ii) float the net asset value of the money market fund. The EC proposal is currently being reviewed by the European Parliament and the Council of Member States as co-legislators, and is expected to be approved in 2014. For further information on international regulatory initiatives, see Significant international regulatory initiatives on pages 89 . bank subsidiaries pay annually to the FDIC. For more information, see Deposit insurance on page 6 . "
```

```{r ex="create_b", type="sample-code"}
# The variable below contains a paragraph from JPM's 2014 annual report
cat(doc)

# Load in stringr for easy string functions
library(stringr)

# Extract the first 100 characters and store them in a variable name first
first <- 

# Extract the middle 100 characters (793 to 892, inclusive) and store in a
# Variable named middle
middle <- 

# Extract the last 100 characters and store them in a variable named last
last <- 

# print out the subsets
print(paste(first, middle, last, sep=' --- '))
#END
```

```{r ex="create_b", type="solution"}
# The variable below contains a paragraph from JPM's 2014 annual report
cat(doc)

# Load in stringr for easy string functions
library(stringr)

# Extract the first 100 characters and store them in a variable name first
first <- str_sub(doc, 1, 100)

# Extract the middle 100 characters (793 to 892, inclusive) and store in a
# Variable named middle
middle <- str_sub(doc, 793, 892)

# Extract the last 100 characters and store them in a variable named last
last <- str_sub(doc, str_length(doc)-99, str_length(doc))

# print out the subsets
print(paste(first, middle, last, sep=' --- '))
#END
```

```{r ex="create_b", type="sct"}
# Template based on https://www.rdocumentation.org/packages/testwhat/versions/4.1.1
# Check if something is explicitly typed

test_expression_output("str_length(first)", incorrect_msg="It appears your `first` isn't the right length.")
test_expression_output("first", incorrect_msg="It appears your `first` isn't quite right.")
test_expression_output("str_length(middle)", incorrect_msg="It appears your `middle` isn't the right length.")
test_expression_output("middle", incorrect_msg="It appears your `middle` isn't quite right.")
test_expression_output("str_length(last)", incorrect_msg="It appears your `last` isn't the right length.")
test_expression_output("last", incorrect_msg="It appears your `last` isn't quite right.")

# test_student_typed('x <- 2', not_typed_msg='')

# Check if function was used in input code
# test_function('c',incorrect_msg='')  

# Requires an object `x` to have the same value as the solution
# test_object("x",incorrect_msg = "",undefined_msg = "")  

# Requires an onject with the same value of `x` in the solution
# test_an_object("x",undefined_msg="")

# Checks if output of student's code contains given evaluated expression
# test_output_contains("x",incorrect_msg = "")

# Check if a vector of predefined objects are unchanged
# test_predefined_objects(c('x','y'),incorrect_msg="Don't onverwrite the predefined variables")

# Checks for a regex pattern in trhe output
# test_output_regex(pattern,fixed=F, times=1, incorrect_msg='')

# Can check an arbitrary expression across both solution and student code
#test_expression_output("typeof(company_name)", incorrect_msg="Did you store textual data in `company_name`?")

test_error()
success_msg("Awesome!")

# Other functions to note:
#     - test_or(a,b) -- checks if either test a or test b pass
#     - test_ggplot() -- can check if plots are correct
#     - test_function() -- can also check included parameters
#     - test_loop() -- checking for and while loops
#     - test_library_function('package', not_called_msg='',incorrect_msg='')
#     - test_if_else() -- checking if statements
#     - test_expression_error() -- can check if functions are properly defined
#     - test_operator('operator',), not_called_msg='',incorrect_msg='')
#     - test_function_definition() -- rigorously check defined function
#     - test_data_frame() -- check if dataframe [columns] are equivalent
#     - test_function_result, test_expression_result
```

### Exercise 2: Changing case

```{r ex="create_c", type="pre-exercise-code"}
library(dplyr)
library(stringr)
doc <- "On June 5, 2013, the SEC approved the publication of proposed structural reforms of money market funds. The proposal considered two reform alternatives that could be adopted either alone or in combination: (i) requiring prime and tax-exempt institutional money market funds to float their net asset values or (ii) requiring all non-governmental money market funds to impose liquidity fees of up to 2% and to have the option to temporarily suspend redemptions (or gate the money market fund) upon the occurrence of specified events indicating that the fund may be under stress. It is currently anticipated that the SEC will adopt final structural reforms in 2014. The Financial Stability Board (the FSB) has endorsed and published for public consultation 15 policy recommendations proposed by the International Organization of Securities Commissions, including requiring money market funds to adopt a floating net asset value. In addition, in September 2013 the European Commission (the EC) released a proposal for a new regulation on money market funds in the EU. The EC proposed two options for stable net asset value money market funds: either (i) maintain a capital buffer of at least three percent of assets under management or (ii) float the net asset value of the money market fund. The EC proposal is currently being reviewed by the European Parliament and the Council of Member States as co-legislators, and is expected to be approved in 2014. For further information on international regulatory initiatives, see Significant international regulatory initiatives on pages 89 . bank subsidiaries pay annually to the FDIC. For more information, see Deposit insurance on page 6 . "
first <- str_sub(doc, 1, 100)
```

```{r ex="create_c", type="sample-code"}
# The variable doc contains a paragraph from JPM's 2014 annual report
# The variable first from the previous exercise is loaded
cat(first)

# Create a version of first that is all lowercase
lower <- 

# Create a version of first that is all UPPERCASE
upper <- 

# Create a version of first that is all Titlecase
title <- 

# print out the different versions
print(paste(lower, upper, title, sep=' --- '))
  
#END
```

```{r ex="create_c", type="solution"}
# The variable doc contains a paragraph from JPM's 2014 annual report
# The variable first from the previous exercise is loaded
cat(first)

# Create a version of first that is all lowercase
lower <- str_to_lower(first)

# Create a version of first that is all UPPERCASE
upper <- str_to_upper(first)

# Create a version of first that is all Titlecase
title <- str_to_title(first)

# print out the different versions
print(paste(lower, upper, title, sep=' --- '))
  
#END
```

```{r ex="create_c", type="sct"}
# Template based on https://www.rdocumentation.org/packages/testwhat/versions/4.1.1
# Check if something is explicitly typed

test_expression_output("lower", incorrect_msg="It appears your `lower` isn't quite right.")
test_expression_output("upper", incorrect_msg="It appears your `upper` isn't quite right.")
test_expression_output("title", incorrect_msg="It appears your `title` isn't quite right.")

# test_student_typed('x <- 2', not_typed_msg='')

# Check if function was used in input code
# test_function('c',incorrect_msg='')  

# Requires an object `x` to have the same value as the solution
# test_object("x",incorrect_msg = "",undefined_msg = "")  

# Requires an onject with the same value of `x` in the solution
# test_an_object("x",undefined_msg="")

# Checks if output of student's code contains given evaluated expression
# test_output_contains("x",incorrect_msg = "")

# Check if a vector of predefined objects are unchanged
# test_predefined_objects(c('x','y'),incorrect_msg="Don't onverwrite the predefined variables")

# Checks for a regex pattern in trhe output
# test_output_regex(pattern,fixed=F, times=1, incorrect_msg='')

# Can check an arbitrary expression across both solution and student code
#test_expression_output("typeof(company_name)", incorrect_msg="Did you store textual data in `company_name`?")

test_error()
success_msg("Awesome!")

# Other functions to note:
#     - test_or(a,b) -- checks if either test a or test b pass
#     - test_ggplot() -- can check if plots are correct
#     - test_function() -- can also check included parameters
#     - test_loop() -- checking for and while loops
#     - test_library_function('package', not_called_msg='',incorrect_msg='')
#     - test_if_else() -- checking if statements
#     - test_expression_error() -- can check if functions are properly defined
#     - test_operator('operator',), not_called_msg='',incorrect_msg='')
#     - test_function_definition() -- rigorously check defined function
#     - test_data_frame() -- check if dataframe [columns] are equivalent
#     - test_function_result, test_expression_result
```

### Exercise 3: Searching for phrases

```{r ex="create_d", type="pre-exercise-code"}
library(dplyr)
library(stringr)
doc <- "On June 5, 2013, the SEC approved the publication of proposed structural reforms of money market funds. The proposal considered two reform alternatives that could be adopted either alone or in combination: (i) requiring prime and tax-exempt institutional money market funds to float their net asset values or (ii) requiring all non-governmental money market funds to impose liquidity fees of up to 2% and to have the option to temporarily suspend redemptions (or gate the money market fund) upon the occurrence of specified events indicating that the fund may be under stress. It is currently anticipated that the SEC will adopt final structural reforms in 2014. The Financial Stability Board (the FSB) has endorsed and published for public consultation 15 policy recommendations proposed by the International Organization of Securities Commissions, including requiring money market funds to adopt a floating net asset value. In addition, in September 2013 the European Commission (the EC) released a proposal for a new regulation on money market funds in the EU. The EC proposed two options for stable net asset value money market funds: either (i) maintain a capital buffer of at least three percent of assets under management or (ii) float the net asset value of the money market fund. The EC proposal is currently being reviewed by the European Parliament and the Council of Member States as co-legislators, and is expected to be approved in 2014. For further information on international regulatory initiatives, see Significant international regulatory initiatives on pages 89 . bank subsidiaries pay annually to the FDIC. For more information, see Deposit insurance on page 6 . "
first <- str_sub(doc, 1, 100)
```

```{r ex="create_d", type="sample-code"}
# The variable doc contains a paragraph from JPM's 2014 annual report
cat(doc)

# How many times does the paragraph mention the "SEC"?
SEC_mentions <- 

# Where is the SEC mentioned?  (All locations)
SEC_location <- 

# print out the results
SEC_mentions
SEC_location

#END
```

```{r ex="create_d", type="solution"}
# The variable doc contains a paragraph from JPM's 2014 annual report
cat(doc)

# How many times does the paragraph mention the "SEC"?
SEC_mentions <- str_count(doc, "SEC")

# Where is the SEC mentioned?  (All locations)
SEC_location <- str_locate_all(doc, "SEC")

# print out the results
SEC_mentions
SEC_location

#END
```

```{r ex="create_d", type="sct"}
# Template based on https://www.rdocumentation.org/packages/testwhat/versions/4.1.1
# Check if something is explicitly typed

test_expression_output("SEC_mentions", incorrect_msg="It appears your `SEC_mentions` isn't quite right.")
test_student_typed("str_locate_all", not_typed_msg="Did you remember to use `str_locate_all()` so as to get ALL mentions' locations?")
test_expression_output("SEC_location", incorrect_msg="It appears your `SEC_location` isn't quite right.")

# test_student_typed('x <- 2', not_typed_msg='')

# Check if function was used in input code
# test_function('c',incorrect_msg='')  

# Requires an object `x` to have the same value as the solution
# test_object("x",incorrect_msg = "",undefined_msg = "")  

# Requires an onject with the same value of `x` in the solution
# test_an_object("x",undefined_msg="")

# Checks if output of student's code contains given evaluated expression
# test_output_contains("x",incorrect_msg = "")

# Check if a vector of predefined objects are unchanged
# test_predefined_objects(c('x','y'),incorrect_msg="Don't onverwrite the predefined variables")

# Checks for a regex pattern in trhe output
# test_output_regex(pattern,fixed=F, times=1, incorrect_msg='')

# Can check an arbitrary expression across both solution and student code
#test_expression_output("typeof(company_name)", incorrect_msg="Did you store textual data in `company_name`?")

test_error()
success_msg("Awesome!")

# Other functions to note:
#     - test_or(a,b) -- checks if either test a or test b pass
#     - test_ggplot() -- can check if plots are correct
#     - test_function() -- can also check included parameters
#     - test_loop() -- checking for and while loops
#     - test_library_function('package', not_called_msg='',incorrect_msg='')
#     - test_if_else() -- checking if statements
#     - test_expression_error() -- can check if functions are properly defined
#     - test_operator('operator',), not_called_msg='',incorrect_msg='')
#     - test_function_definition() -- rigorously check defined function
#     - test_data_frame() -- check if dataframe [columns] are equivalent
#     - test_function_result, test_expression_result
```

## Regular expressions

### Exercise 4: Finding mentions by pattern

```{r ex="create_e", type="pre-exercise-code"}
library(dplyr)
library(stringr)
doc <- "On June 5, 2013, the SEC approved the publication of proposed structural reforms of money market funds. The proposal considered two reform alternatives that could be adopted either alone or in combination: (i) requiring prime and tax-exempt institutional money market funds to float their net asset values or (ii) requiring all non-governmental money market funds to impose liquidity fees of up to 2% and to have the option to temporarily suspend redemptions (or gate the money market fund) upon the occurrence of specified events indicating that the fund may be under stress. It is currently anticipated that the SEC will adopt final structural reforms in 2014. The Financial Stability Board (the FSB) has endorsed and published for public consultation 15 policy recommendations proposed by the International Organization of Securities Commissions, including requiring money market funds to adopt a floating net asset value. In addition, in September 2013 the European Commission (the EC) released a proposal for a new regulation on money market funds in the EU. The EC proposed two options for stable net asset value money market funds: either (i) maintain a capital buffer of at least three percent of assets under management or (ii) float the net asset value of the money market fund. The EC proposal is currently being reviewed by the European Parliament and the Council of Member States as co-legislators, and is expected to be approved in 2014. For further information on international regulatory initiatives, see Significant international regulatory initiatives on pages 89 . bank subsidiaries pay annually to the FDIC. For more information, see Deposit insurance on page 6 . "
first <- str_sub(doc, 1, 100)
```

```{r ex="create_e", type="sample-code"}
# The variable doc contains a paragraph from JPM's 2014 annual report
cat(doc)

# what are the page references in the text?  These are of the form "page #" or
# "pages #".
page_references <- str_extract_all()

# How was the SEC mentioned? Extract the text from two words before "SEC" is
# mentioned until two words after the SEC is mentioned
SEC_references <- str_extract_all()

# print out the results
page_references
SEC_references

#END
```

```{r ex="create_e", type="solution"}
# The variable doc contains a paragraph from JPM's 2014 annual report
cat(doc)

# what are the page references in the text?  These are of the form "page #" or
# "pages #".
page_references <- str_extract_all(doc, "pages?[:blank:][:digit:]+")

# How was the SEC mentioned? Extract the text from two words before "SEC" is
# mentioned until two words after the SEC is mentioned
SEC_references <- str_extract_all(doc, "[:graph:]+[:blank:][:graph:]+[:blank:]SEC[:blank:][:graph:]+[:blank:][:graph:]+")

# print out the results
page_references
SEC_references

#END
```

```{r ex="create_e", type="sct"}
# Template based on https://www.rdocumentation.org/packages/testwhat/versions/4.1.1
# Check if something is explicitly typed

test_expression_output("length(page_references[[1]])", incorrect_msg="`page_references` should end up with 2 matches.")
test_expression_output("page_references", incorrect_msg="It appears your `SEC_mentions` isn't quite right, but it has the right number of matches.")
test_expression_output("length(SEC_references[[1]])", incorrect_msg="`SEC_references` should end up with 2 matches.")
test_expression_output("SEC_references", incorrect_msg="It appears your `SEC_location` isn't quite right, but it has the right number of matches.")

# test_student_typed('x <- 2', not_typed_msg='')

# Check if function was used in input code
# test_function('c',incorrect_msg='')  

# Requires an object `x` to have the same value as the solution
# test_object("x",incorrect_msg = "",undefined_msg = "")  

# Requires an onject with the same value of `x` in the solution
# test_an_object("x",undefined_msg="")

# Checks if output of student's code contains given evaluated expression
# test_output_contains("x",incorrect_msg = "")

# Check if a vector of predefined objects are unchanged
# test_predefined_objects(c('x','y'),incorrect_msg="Don't onverwrite the predefined variables")

# Checks for a regex pattern in trhe output
# test_output_regex(pattern,fixed=F, times=1, incorrect_msg='')

# Can check an arbitrary expression across both solution and student code
#test_expression_output("typeof(company_name)", incorrect_msg="Did you store textual data in `company_name`?")

test_error()
success_msg("Awesome!")

# Other functions to note:
#     - test_or(a,b) -- checks if either test a or test b pass
#     - test_ggplot() -- can check if plots are correct
#     - test_function() -- can also check included parameters
#     - test_loop() -- checking for and while loops
#     - test_library_function('package', not_called_msg='',incorrect_msg='')
#     - test_if_else() -- checking if statements
#     - test_expression_error() -- can check if functions are properly defined
#     - test_operator('operator',), not_called_msg='',incorrect_msg='')
#     - test_function_definition() -- rigorously check defined function
#     - test_data_frame() -- check if dataframe [columns] are equivalent
#     - test_function_result, test_expression_result
```

### Exercise 5: Further regex practice

```{r ex="create_f", type="pre-exercise-code"}
library(dplyr)
library(stringr)
doc <- "On June 5, 2013, the SEC approved the publication of proposed structural reforms of money market funds. The proposal considered two reform alternatives that could be adopted either alone or in combination: (i) requiring prime and tax-exempt institutional money market funds to float their net asset values or (ii) requiring all non-governmental money market funds to impose liquidity fees of up to 2% and to have the option to temporarily suspend redemptions (or gate the money market fund) upon the occurrence of specified events indicating that the fund may be under stress. It is currently anticipated that the SEC will adopt final structural reforms in 2014. The Financial Stability Board (the FSB) has endorsed and published for public consultation 15 policy recommendations proposed by the International Organization of Securities Commissions, including requiring money market funds to adopt a floating net asset value. In addition, in September 2013 the European Commission (the EC) released a proposal for a new regulation on money market funds in the EU. The EC proposed two options for stable net asset value money market funds: either (i) maintain a capital buffer of at least three percent of assets under management or (ii) float the net asset value of the money market fund. The EC proposal is currently being reviewed by the European Parliament and the Council of Member States as co-legislators, and is expected to be approved in 2014. For further information on international regulatory initiatives, see Significant international regulatory initiatives on pages 89 . bank subsidiaries pay annually to the FDIC. For more information, see Deposit insurance on page 6 . "
first <- str_sub(doc, 1, 100)
```

```{r ex="create_f", type="sample-code"}
# The variable doc contains a paragraph from JPM's 2014 annual report
cat(doc)

# Find all numbers (just the number)
regex1 <- str_extract_all()

# Find all phrases in parentheses, including the parentheses themselves
regex2 <- str_extract_all()

# Find all words that include a doubled letter, like "ss" or "oo"
# "pages #".
regex3 <- str_extract_all()

# print out the results
regex1
regex2
regex3

#END
```

```{r ex="create_f", type="solution"}
# The variable doc contains a paragraph from JPM's 2014 annual report
cat(doc)

# Find all numbers (just the number)
regex1 <- str_extract_all(doc, "[:digit:]+")

# Find all phrases in parentheses, including the parentheses themselves
regex2 <- str_extract_all(doc, "\\(.*?\\)")

# Find all words that include a doubled letter, like "ss" or "oo"
# "pages #".
regex3 <- str_extract_all(doc, "[:alpha:]*([:alpha:])\\1[:alpha:]*")

# print out the results
regex1
regex2
regex3

#END
```

```{r ex="create_f", type="sct"}
# Template based on https://www.rdocumentation.org/packages/testwhat/versions/4.1.1
# Check if something is explicitly typed

test_expression_output("length(regex1[[1]])", incorrect_msg="`regex3` has an incorrect number of matches")
test_expression_output("regex1", incorrect_msg="It appears your `regex1` isn't quite right, but it has the right number of matches.")
test_expression_output("length(regex2[[1]])", incorrect_msg="`regex2` has an incorrect number of matches")
test_expression_output("regex2", incorrect_msg="It appears your `regex2` isn't quite right, but it has the right number of matches.")
test_expression_output("length(regex3[[1]])", incorrect_msg="`regex3` has an incorrect number of matches.  Did you make sure to keep the text short?")
test_expression_output("regex3", incorrect_msg="It appears your `regex3` isn't quite right, but it has the right number of matches.")

# test_student_typed('x <- 2', not_typed_msg='')

# Check if function was used in input code
# test_function('c',incorrect_msg='')  

# Requires an object `x` to have the same value as the solution
# test_object("x",incorrect_msg = "",undefined_msg = "")  

# Requires an onject with the same value of `x` in the solution
# test_an_object("x",undefined_msg="")

# Checks if output of student's code contains given evaluated expression
# test_output_contains("x",incorrect_msg = "")

# Check if a vector of predefined objects are unchanged
# test_predefined_objects(c('x','y'),incorrect_msg="Don't onverwrite the predefined variables")

# Checks for a regex pattern in trhe output
# test_output_regex(pattern,fixed=F, times=1, incorrect_msg='')

# Can check an arbitrary expression across both solution and student code
#test_expression_output("typeof(company_name)", incorrect_msg="Did you store textual data in `company_name`?")

test_error()
success_msg("Awesome!")

# Other functions to note:
#     - test_or(a,b) -- checks if either test a or test b pass
#     - test_ggplot() -- can check if plots are correct
#     - test_function() -- can also check included parameters
#     - test_loop() -- checking for and while loops
#     - test_library_function('package', not_called_msg='',incorrect_msg='')
#     - test_if_else() -- checking if statements
#     - test_expression_error() -- can check if functions are properly defined
#     - test_operator('operator',), not_called_msg='',incorrect_msg='')
#     - test_function_definition() -- rigorously check defined function
#     - test_data_frame() -- check if dataframe [columns] are equivalent
#     - test_function_result, test_expression_result
```

## Calculating quantities with text

Note: Due to missing packages in DataCamp light, namely `quanteda`, `quanteda.textstats`, `quanteda.textplots`, `textdata`, and `tidytext`, I have provided sample code that you can run on your own computer in RStudio.  Make sure to run `install.packages("quanteda")`, `install.packages("textdata")`, and `install.packages("tidytext")` to install those packages if you don't have them.

Each of the three exercises below can be run as standalone scripts, as they contain all needed imports within their code blocks

Code for this section can be <a href="./Session_7_R_offline.Rmd">downloaded as an Rmd file here</a>.  The output of this code can be viewed in <a href="./Session_7_R_offline.nb.html">this Rmarkdown notebook</a>.

### Exercise 6: Readability with Quanteda

How does the readability of JPMorgan's annual report compare to the Citigroup annual report from class?

```{r ex="create_g", type="sample-code"}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v4/Session_7/0000019617-14-000289.txt")

# Load in quanteda
library(quanteda)
library(quanteda.textstats)

# Calculate the three readability measures
textstat_readability(doc, "Flesch.Kincaid")
textstat_readability(doc, "FOG")
textstat_readability(doc, "Coleman.Liau")

#END
```

### Exercise 7: Sentiment with Quanteda

How does the sentiment of JPMorgan's annual report compare to the Citigroup annual report from class?

```{r ex="create_h", type="sample-code"}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in tidytext
library(tidytext)

# Load some components of tidyverse
library(dplyr)  # for the usual commands
library(tidyr)  # for spread

# convert document to tidy format
df_doc <- data.frame(ID=c("0000019617-14-000289"), text=c(doc),
                     stringsAsFactors = F) %>%
  unnest_tokens(word, text)

# Calculate term frequency
terms <- df_doc %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)

# Get the Loughran McDonald sentiment dictionary
sentiment <- get_sentiments("loughran")

# Merge in sentiment
tf_sent <- tf %>% left_join(sentiment)

# Calculate the three readability measures
tf_sent %>%
  spread(sentiment, tf, fill=0) %>%
  select(constraining, litigious, negative, positive, superfluous, uncertainty) %>%
  colSums()

#END
```

### Exercise 8: Make a word cloud after removing stopwords

```{r ex="create_i", type="sample-code"}
# load in readr (or tidyverse) to get read_file() function
library(readr)

# Load in all of JPM's 2014 annual report
doc <- read_file("https://rmc.link/Slides/acct420v3/Session_7/0000019617-14-000289.txt")

# Load in quanteda and tidytext
library(quanteda)
library(quanteda.textplots)
library(tidytext)

# Load in some of tidyverse
library(dplyr)

# convert document to tidy format
df_doc <- data.frame(ID=c("0000019617-14-000289"), text=c(doc),
                     stringsAsFactors = F) %>%
  unnest_tokens(word, text)

# Pull a list of stopwords
stopwords <- stopwords::stopwords(source="smart")

# Remove stopwords
df_doc_stop <- df_doc %>%
  anti_join(data.frame(word=stopwords, stringsAsFactors=F))

# Calculate term frequency
terms <- df_doc_stop %>%
  count(ID, word, sort=TRUE) %>%
  ungroup()
total_terms <- terms %>% 
  group_by(ID) %>% 
  summarize(total = sum(n))
tf <- left_join(terms, total_terms) %>% mutate(tf=n/total)

# Build a corpus object for quanteda
corp <- cast_dfm(tf, ID, word, n)

# Plot a word cloud -- If you don't have RColorBrewer installed, you can
# remove the `color=` option.
textplot_wordcloud(dfm(corp), color = RColorBrewer::brewer.pal(9, "Set1"))

#END
```