---
title: "ACCT 420: Course Logistics + R Refresh"
author: 
  - name: Dr. Richard M. Crowley
    email: rcrowley@smu.edu.sg
    affiliation: https://rmc.link/
title-slide-attributes:
  data-background-video: "../Backgrounds/title.mp4"
  data-background-video-loop: "true"
  data-background-size: stretch
format:
  revealjs:
    theme: default
    css: quarto_white.css
    navigation-mode: vertical
    controls: true
    transition: slide
    background-transition: fade
    transition-speed: default
    width: 1536
    height: 864
    logo: ../Logos/logo2.png
    code-link: true
    code-line-numbers: false
    chalkboard:
      theme: whiteboard
    auto-stretch: false
revealjs-plugins:
  - attribution
execute:
  echo: true
  warning: false
  message: false
editor: source
---

# Front Matter {data-background-video="../Backgrounds/section.mp4"}

<!--# Note that the downlit and xml2 pacakges are required to compile -->
<!--# Installed extensions include `attribution`, `qrcode`, and `material-icons` -->

```{r helpers, echo=FALSE, warning=FALSE, message=F}
library(knitr)
library(kableExtra)

#' Pretty formatted HTML tables
#' 
#' @param df The dataframe to format as a table
#' @param names Names to use for columns if not using the df column names. Vector
#' @param highlight_cols Columns to highlight. Default is c(1) for highlighting the first row. Use c() to remove.
#' @param highlight_rows Rows to highlight. Default is c() to not highlight any rows.
#' @param color_rows Rows to highlight in color. Default is c() to not highlight any rows.
#' @param highlight_end_rows Rows to highlight. Uses a reverse indexing where 1 is the last row, 2 is second to last, etc.
#' @param full Use to span the full width of a slide. FALSE by default.
#' @param fixed_header Whether the first row should be treated as a header row. TRUE by default.
html_df <- function(df, names=NULL, highlight_cols=c(1), highlight_rows=c(), color_rows=c(), highlight_end_rows=c(), full=F, fixed_header=T) {
  if(!length(names)) {
    names=colnames(df)
  }
  kbl(df,"html", col.names = names, align = c("l",rep('c',length(cols)-1))) %>%
    kable_paper(c("striped","hover"), full_width=full, fixed_thead=F, html_font = "\"Source Sans Pro\", Helvetica, sans-serif") %>%
    {if(length(highlight_cols)) column_spec(., highlight_cols,bold=T, background = "#ffffff99") else .} %>%
    {if(length(highlight_rows)) row_spec(., highlight_rows,bold=T, background = "#ffffff99") else .} %>%
    {if(length(color_rows)) row_spec(., color_rows,bold=T, background = "#aaaaff99") else .} %>%
    {if(length(highlight_end_rows)) row_spec(., nrow(df) + 1 - highlight_end_rows,bold=T, background = "#ffffff99") else .}
}

library(tidyverse)
```

<!--# List pacakges here that are not being correctly highlighted -->
<!--# The usual cause is calling the library from a echo=FALSE cell -->

```{r}
#| include: false
options(downlit.attached = c("knitr", "kableExtra", "tidyverse"))
```

## Learning objectives {data-background="../Backgrounds/default.jpg"}

::: columns
::: {.column width="50%"}

![](../Course_Diagrams/Map-1.png){fig-alt="Roadmap for the course"}

:::

::: {.column width="50%"}
-   <b>Theory:</b>
    -   What is analytics?
-   <b>Application:</b>
    -   Who uses analytics? (and why?)
-   <b>Methodology:</b>
    -   Review of **R**
-   Almost every class will touch on each of these three aspects
:::
:::

# About Me {data-background-video="../Backgrounds/section.mp4"}

## {data-background="../Photos/snow.jpg" data-background-size="contain" class="photo present-only"}

::: {.attribution}

Champaign, IL, USA

:::

## {data-background="../Photos/MAME.jpg" data-background-size="contain" class="photo present-only"}

::: {.attribution}

Coffee from MAME, Zurich, Switzerland

:::

## {data-background="../Photos/FOODS.png" data-background-size="contain" class="photo present-only"}

::: {.attribution}

Soufi's (Toronto, CA [closed]), Kamat Bugle Rock (Bangalore, IN), and Tsukiji Market and Fuunji (Tokyo, JP)

:::

## {data-background="../Photos/INGREDIENTS.png" data-background-size="contain" class="photo present-only"}

::: {.attribution}

From markets in: Helsinki, Finland; Lindau, Germany; Feldkirch, Austria; Vienna, Austria

:::

## {data-background="../Photos/COOKING.png" data-background-size="contain" class="photo present-only"}

::: {.attribution}

From scratch: Apple pie, Thai green curry, Whole goose in the style of peking duck

:::

## {data-background="../Photos/The_Coffee_Line.jpg" data-background-size="contain" class="photo present-only"}

::: {.attribution}

John Sloan, *The Coffee Line*, 1905

:::

## {data-background="../Photos/hiking_alps.jpg" data-background-size="contain" class="photo present-only"}

::: {.attribution}

Taken in Southern Tyrol, Italy

:::

## {data-background="../Photos/Potato.jpg" data-background-size="contain" class="photo present-only"}

## {data-background="../Photos/Garden.jpg" data-background-size="contain" class="photo present-only"}

## {class="present-only"}

:::: {.columns}

::: {.column width="50%"}
<iframe src="https://giphy.com/embed/JRgLEZCsivjHoRWpb2" width="480" height="270" frameBorder="0" class="giphy-embed" allowFullScreen></iframe><p><a href="https://giphy.com/gifs/JRgLEZCsivjHoRWpb2">via GIPHY</a></p>
:::

::: {.column width="50%"}
![](../Photos/vr_spaces.png){width="600px" fig-align="center"}
:::

::::

## {data-background="../Photos/server.jpg" data-background-size="contain" class="photo present-only"}

## Teaching {data-background="../Backgrounds/blackboard.jpg"}

- At SMU since 2016
- Teaching:
    - IDIS 700, Machine Learning for Social Science
        - A country-wide PhD course on *empirical* machine learning
    - ACCT 703, Analytical Methods in Accounting
        - Theory of accounting through the lenses of economics, history, and philosophy
    - ACCT 420, Forecasting and Forensic Analytics
        - Your are here!
    - ACCT 101, Financial Accounting

- Before SMU: Completed my PhD at University of Illinois Urbana-Champaign

![](../Figures/logo2.png){height="180px" fig-align="center"} ![](../Figures/UIUC.png){height="180px" fig-align="center"}

## Research {data-background="../Backgrounds/math_idea.jpg"}

- Accounting disclosure: What companies say, and why it matters
    - Focus on social media and regulatory filings
- Approach this using AI/ML techniques

<img style="max-width:25%" src="../Figures/R017_vector_space.png"/><img style="max-width:25%" src="../Figures/POFMA.png"/><img style="max-width:25%" src="../Figures/twitter.png"/><img style="max-width:25%" src="../Figures/Enron.png"/>

## {background-iframe="https://quotes.wsj.com/index/SG/STI/advanced-chart" background-interactive="true"}

## Research highlights {data-background="../Backgrounds/neural-net.jpg"}

1. An advanced model for detecting financial misreporting using annual report text.
2. Multiple projects on Twitter showcasing:
    1. How companies disclose when financials are not normal or expected
    2. That executives' disclosures are as important on Twitter as their firms' disclosures
    3. That ESG disclosure on Twitter is not credible
3. Newer work on
    - Misinformation laws (e.g., POFMA)
    - Foundational problems of text-based sentiment in accounting and finance
    - COVID-19 reactions worldwide
    

::: {.callout-note}
## What is the common thread?

All of the above use text-based data paired with AI/ML algorithms. A secondary thread is the importance of *content*, while some papers also push for better *causality* in research.

:::

# About this course {data-background-video="../Backgrounds/section.mp4"}

## What will this course cover? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
![](../Course_Diagrams/Map.png){height="500px" fig-align="center"}
:::

::: {.column width="50%"}
1. Foundations (*today*)
    - Thinking about analytics
    - In class: Setting a foundation for the course
    - Outside: Practice and refining skills on Datacamp
        - Pick any R course, any level, and try it out!
2. Financial forecasting
    - Predict financial outcomes
    - Linear models
:::

::::

> Getting familiar with forecasting using **real** data and R

## What will this course cover? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
![](../Course_Diagrams/Map.png){height="500px" fig-align="center"}
:::

::: {.column width="50%"}
3. Binary classification
    - Event prediction
    - Classification & detection
4. Advanced methods
    - Non-numeric data (text)
    - Clustering
    - AI/Machine learning (AI/ML)
        - Ethics of AI
        - Current developments, including LLMs (e.g., ChatGPT)
:::

::::

> Higher level financial forecasting, detection, and AI/ML

## Datacamp {data-background="../Backgrounds/coding.jpg"}

- Datacamp is providing *free* access to their *full* library of analytics and coding online tutorials
    - You will have free access for 6 months (Usually \$25 USD/mo)
- Online tutorials include short exercises and videos to help you learn R
- I have suggested some limited materials via a Datacamp class
    - These are entirely optional
    - If you are struggling with coding, do make use of this resource!
        - Sign up through the link on eLearn
        - Make sure you sign up in the *first week* of the class!
- You are encouraged to go beyond the assigned materials


> Datacamp's tutorials teach R from the ground up, and are mandatory unless you can already code in R.

## Textbook {data-background="../Backgrounds/reference.jpg"}

:::: {.columns}

::: {.column width="50%"}

- There is no required textbook
    - Datacamp is taking the place of the textbook
- If you prefer having a textbook...
    - <a href="https://www.jaredlander.com/r-for-everyone/">R for Everyone</a> by Jared Lander is a good one on R

:::

::: {.column width="50%"}
![](../Figures/R_for_Everyone.jpg){height="255px" fig-align="center"}
:::

::::

<br>

- Other course materials (slides and articles) are available at:
    - eLearn
    - [https://rmc.link/acct420](https://rmc.link/acct420)
        - This includes html versions of the slides with interactive content
- Announcements will be only on eLearn

## Teaching philosphy {data-background="../Backgrounds/books.jpg"}

1. Analytics is best learned by doing it
    - Less lecture, more thinking
2. Working with others greatly extends learning
    - If you are ahead:
        - The best sign that you've mastered a topic is if you can explain it to others
    - If you are lost:
        - Gives you a chance to get help the help you need

## Grading {data-background="../Backgrounds/road.jpg"}

- Standard SMU grading policy
- Participation @ 10%
- Individual work @ 20%
- Group project @ 30%
- Final exam @ 40%

## Participation {data-background="../Backgrounds/people.jpg"}

- Come to class
    - If you have a conflict, email me
        - Excused classes do not impact your participation grade
- Ask questions to *extend* **or** *clarify*
- Answer questions and explain answers
    - Give it your best shot!
- Help those in your group to understand concepts
- Present your work to the class
    - Come up with effective and creative solutions to the problems!

## Outside of class {data-background="../Backgrounds/library.jpg"}

- Verify your understanding of the material
- Apply to other real world data
    - Techniques and code will be useful after graduation
- Answers are expected to be your own work, unless otherwise stated
    - No sharing answers (unless otherwise stated)
- Submit on eLearn
- I will provide snippets of code to help you with trickier parts

## Group project {data-background="../Backgrounds/presentation.jpg"}

- Data science competition format, hosted on <a href="https://www.kaggle.com/competitions">Kaggle</a>
    - Multiple options for the project will be available
- The project will start on session 7
- The project will finish on session 12 with group presentations

![](../Figures/Kaggle_logo.png){height="200px" fig-align="center"}



## Final exam {data-background="../Backgrounds/studying.jpg"}

- Why?
    - Ex post indicator of attainment
- How?
    - 2 hours long
    - Most of it is lLong format: problem solving oriented
    - A small amount of MCQ focused on techniques

## Expectations {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
**In class**

- Participate
    - Ask questions
        - Clarify
        - Add to the discussion
    - Answer questions
    - Work with classmates
:::

::: {.column width="50%"}
**Out of class**

- Check eLearn for course announcements
- Do the assigned tutorials on Datacamp
    - This will make the course much easier!
- Do individual work on your own (unless otherwise stated)
    - Submit on eLearn
- Office hours are there to help!
    - Short questions can be emailed instead
:::

::::


## Tech use {data-background="../Backgrounds/devices.jpg"}

- Laptops and other tech are OK!
    - Use them for learning, not messaging
    - Furthermore, you will *need* a computer for this class
        - If you do not have access to one, I can provide you a laptop loan
- Examples of good tech use:
    - Taking notes
    - Viewing slides
    - Working out problems
    - Group work
- Avoid during class:
    - Messaging your friends on Telegram
    - Working on homework for the class in a few hours
    - Watching livestreams of pandas or Hearthstone

## Office hours {data-background="../Backgrounds/reference.jpg"}

- Prof office hours:
    - Bookable at <a href="https://rmc.link/420OH">rmc.link/420OH</a>
- **Short** questions can be emailed
    - I try to respond within 24 hours

## Lastly... {data-background="../Backgrounds/default.jpg"}

- I don't expect everything to run 100% smoothly on either side, and there will be leniency to account for this as needed
    - If you will miss an in-person session, let me know the reason in advance, and then work with your peers to catch up
    - If you will miss a Zoom session, please let me know the reason in advance, and then work through the recording on your own
- I always provide a survey at the end of each class session that allows you to anonymously voice anything you liked or didn't like about a session.  Do use this channel if you encounter any difficulties.  Common agreed-upon problems will be addressed within 1-2 class sessions.
    - The survey link is on eLearn (under the session's folder) and will be on the last slide I present each week.

## Likely schedule adjustment {data-background="../Backgrounds/calendar.png"}

**Expect one class session to be cancelled in mid-to-late October**

- Some class session may be moved to Zoom as well, as needed

![](../Figures/chick_hatch.png){width="500px" fig-align="center"}

# About this course: Online version addendum {data-background-video="../Backgrounds/section.mp4"}

## General Zoom etiquette {data-background="../Backgrounds/Zoom_Faces_Mariia_Sennikova.jpg"}

- Keep your mic muted when you are not speaking
    - 20+ mics all on at once creates **a lot** of background noise
- You are welcome to leave your video on -- seeing your reactions helps me to gauge your learning of the course content
    - If you are uncomfortable doing so, please have a profile photo of yourself
        - To do this, click yourself in the participants window, click "more" or "..." and then "Edit Profile Picture"
- Feel free to use Zoom's built in functionality for backgrounds
    - Just be mindful that this is considered a professional environment and that the class sessions are recorded
    
> All sessions will be recorded to provide flexibility for anyone missing class to still see the material.  It also allows you to easily review the class material.

::: {.attribution}

Background: Mariia Sennikova, Zoom Faces, 2021-2022.

:::
        
## Asking questions {data-background="../Backgrounds/Zoom_Faces_Mariia_Sennikova.jpg"}

- If you have a question, use the *Raise Hand* function
    - Where to find it:
        - Desktop: Click *Reactions* and then *Raise hand*
        - Mobile: Under *More* in the toolbar
    - When called on:
        1. Unmute yourself.
        2. Turn on your video if you are comfortable with it
        3. Ask your question.
        4. You are always welcome to ask follow up questions or clarifications in succession
        5. After your question is answered, **mute your mic**.

::: {.attribution}

Background: Mariia Sennikova, Zoom Faces, 2021-2022.

:::

## Group work on Zoom {data-background="../Backgrounds/Zoom_Faces_Mariia_Sennikova.jpg"}

- I will make use of the Breakout room functionality
- Your group can use the "Share screen function" to emulate crowding around one laptop
- If your group is stuck or needs clarification, you can use the *Ask for help* function to get my attention
- I will drop by each group from time to time to check in and see how you are doing with the problem
- I may also ask your group to present something to the class after a breakout session is finished.

> Before project groups are settled, groups will be randomized each class session to encourage you to meet each other.  Once group project groups are set, breakout sessions will be with your group project group.

::: {.attribution}

Background: Mariia Sennikova, Zoom Faces, 2021-2022.

:::

# About you {data-background-video="../Backgrounds/section.mp4"}

## About you {data-background="../Backgrounds/default.jpg"}

- Survey at <a href="https://docs.google.com/forms/d/e/1FAIpQLSdgQ5AVSQIV8qj82mZnUc-QDxVc3ncQCDofAruKDw95t3lo7w/viewform?usp=sf_link">rmc.link/aboutyou</a>
- Results are anonymous
- We will go over the survey next week at the start of class

{{< qrcode rmc.link/aboutyou qrabout width=300 height=300 >}}

# Analytics {data-background-video="../Backgrounds/section.mp4"}

## Learning objectives {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
![](../Course_Diagrams/Map-1.png){width="600px" fig-align="center"}
:::

::: {.column width="50%"}
- <b>Theory:</b>
    - What is analytics?
- <b>Application:</b>
    - Who uses analytics? (and why?)
- <b>Methodology:</b>
    - Review of **R**

*Almost every class will touch on each of these three aspects
:::

::::


## What is analytics? {data-background="../Backgrounds/question.jpg"}



## What is analytics? {data-background="../Backgrounds/reference.jpg"}

> Oxford: The systematic computational analysis of data or statistics

> Webster: The method of logical analysis

> Gartner: catch-all term for a variety of different business intelligence [...] and application-related initiatives <!--https://www.gartner.com/it-glossary/analytics pulled June 7, 2018-->

## What is analytics? {data-background="../Backgrounds/reference.jpg"}

> Simply put: Answering questions using data

- Additional layers we can add to the definition:
    - Answering questions using *a lot of* data
    - Answering questions using data *and statistics*
    - Answering questions using data *and computers*

```{r}
#| eval: true
#| echo: false
#| fig-width: 8
#| fig-height: 2.8
#| fig-align: center

# Google ngrams for Data
library(ngramr)
library(ggplot2)
ngd <- c("(Analytics + analytics)", "(Machine learning + machine learning)")
ggram(ngd, year_start=1960, geom = "area", google_theme=F, smoothing = 3) + theme(legend.position="bottom", legend.direction="horizontal")
```

::: aside

Made using `{ngramr}`

:::



## Analytics vs AI/machine learning {data-background="../Backgrounds/case.jpg"}

:::: {.columns}

::: {.column width="50%"}
![](../Figures/analytics_ml.png){width="450px" fig-align="center"}
:::

::: {.column width="50%"}
- In class reading:
    - <a href="https://blog.datarobot.com/ai-will-enhance-us-not-replace-us-1">AI Will Enhance Us, Not Replace Us</a>
    - By DataRobot's VP of AI Strategy
    - Short link: *rmc.link/420class1*

{{< qrcode rmc.link/420class1 qrarticle1 width=200 height=200 >}}

:::

::::

<div class="fragment fade-up"><blockquote style="text-align:left"><p>How will Analytics/AI/ML change society and the accounting profession?</p></blockquote></div>

## What are forecasting analytics? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Forecasting is about making an educated guess of events to come in the future
    - Who will win the next soccer game?
    - What stock will have the best (risk-adjusted) performance?
    - What will Singtel's earnings be next quarter?
- Leverage *past* information
    - Implicitly assumes that the past and the future predictably related
:::

::: {.column width="50%"}
![](../Figures/crystal_ball.png){height="466px" fig-align="center"}
:::

::::

## Past and future examples {data-background="../Backgrounds/default.jpg"}

```{r graph1, echo=FALSE, warning=FALSE, message=FALSE, fig.height = 5, fig.width = 4.3}
library(readr)
df <- read_csv("../../Data/Session_1-1.csv")
df <- df[df$indfmt != "FS",]  # remove FS codes
df <- df[order(df$gvkey,df$datadate),]
df$ni_lag <- c(NA, df$ni[1:nrow(df)-1])
df <- df[floor(df$datadate/10000) == 2017,]
df <- df[!(is.na(df$ni) | is.na(df$ni_lag)),]
library(ggplot2)
suppressPackageStartupMessages(library(plotly))
cor <- cor(x=df$ni, y=df$ni_lag, method="pearson")
set.seed(11)
df_s <- df[sample(nrow(df), 400),]
```

:::: {.columns}

::: {.column width="50%"}
- Past company earnings predicts future company earnings
    - Some earnings are stable over time (Ohlsson model)
    - Correlation: `r cor`
:::

::: {.column width="50%"}
```{r}
#| eval: true
#| echo: false
#| fig-width: 6
#| fig-height: 6
#| fig-align: center

plot <- ggplot(df_s, aes(x=ni_lag, y=ni)) + 
  geom_point(shape=1, aes(text=sprintf("Ticker: %s", tic))) + 
  geom_smooth(aes(x=ni_lag, y=ni), method=lm, se=T) + 
  labs(x="2016 Net Income ($M USD)",
       y="2017 Net Income ($M USD)")
ggplotly(plot, tooltip="text")
```

:::

::::


## Past and future examples {data-background="../Backgrounds/default.jpg"}

```{r graph2, echo=FALSE, warning=FALSE, message=FALSE}
library(readr)
# import GDP data
dfg <- read_csv("../../Data/per-capita-gni-and-per-capita-gdp-at-current-market-prices-annual-2019.08.csv")
dfg <- dfg[dfg$level_1 == "Per Capita GDP",]
dfg <- dfg[order(dfg$year),]

# import unemployment data
dfu <- read_csv("../../Data/overall-unemployment-rate-annual-2019.08.csv")
dfu <- dfu[order(dfu$year),]

# merge into one dataset
df <- data.frame(year=1993:2016,
                 growth=dfg[dfg$year>1992,]$value / dfg[dfg$year > 1991 & dfg$year < 2016,]$value - 1,
                 unemploy=dfu[dfu$year>1992,]$unemployment_rate,
                 unemploy_lag=dfu[dfu$year<2016,]$unemployment_rate)

library(ggplot2)
suppressPackageStartupMessages(library(plotly))
cor <- cor(x=df$growth, y=df$unemploy, method="pearson")


```

:::: {.columns}

::: {.column width="50%"}
- Job reports predicts GDP growth in Singapore
    - Economic relationship
    - More unemployment in a year is related to lower GDP growth
        - Correlation of `r cor`
:::

::: {.column width="50%"}
```{r}
#| eval: true
#| echo: false
#| fig-width: 6
#| fig-height: 6
#| fig-align: center

plot <- ggplot(df, aes(x=unemploy, y=growth)) + 
  geom_point(shape=1, aes(text=sprintf("Year: %i", year))) + 
  geom_smooth(method=lm, se=T) + 
  labs(x="Unemployment rate",
       y="GDP Growth")
ggplotly(plot, tooltip="text")
```
:::

::::


## Past and future examples {data-background="../Backgrounds/default.jpg" class="default print-not"}

```{r graph3, eval=TRUE, include=FALSE, warning=FALSE, message=FALSE}
drown <- c(109,102,102,98,85,95,96,98,123,94,102) # US CDC data, 1999 to 2009 (falling in)
drown2 <- c(421,465,494,538,430,530,511,600,582,605,603)
ic <- c(16.1, 15.8, 16.2, 15.9 ,14.6 ,15.1 ,15.3, 14.8, 14.2 ,13.9 ,14 ,13.2 ,13.2 ,13 ,12.5 ,12.9 ,13.1) # Statista, 2000 to 2016
icprice <- c(126.858,128.467,130.858,131.733,134.808,137.508,144.592,150.575,155.500,161.742,164.375,173.408,179.033,175.467,178.250,177.575,179.233,183.373,192.768,196.637,195.055,209.161,215.524,215.489,215.412,219.201,219.452,219.903,219.593) # CPI data for ice cream and related products, 1990-2018
cancerthursday <- c(80262,80994,81321,81988,81390,82682,82109,82516,82663,83166,84978,84742) # CDC (tylervigen.com)
# https://futurism.com/correlation-vs-causation-2/
df <- data.frame(drown=drown2[2:length(drown2)],
                 icecream=ic[1 : length(drown2)-1],
                 icecreamprice=icprice[11:(length(drown2)+9)],
                 year=2000:2009)
df$drown <- df$drown / mean(df$drown)
df$icecream <- df$icecream / mean(df$icecream)
df$icecream2 <- df$icecream*df$icecreamprice / mean(df$icecream*df$icecreamprice)
df$icecream3 <- df$icecreamprice / mean(df$icecreamprice)
cor_unit <- cor(x=df$icecream, y=df$drown)
cor_rev <- cor(x=df$icecream2, y=df$drown)
cor_price <- cor(x=df$icecreamprice, y=df$drown)
library(ggplot2)

library(reshape2)
dm <- melt(df[c("drown", "icecream2", "year")],id.var=3)
plot1 <- ggplot(dm, aes(x=year, y=value, colour=variable)) + 
  geom_point(shape=1) +
  geom_smooth(method=loess, se=F) +
  labs(x="Year",
       y="% of average year") + 
  scale_x_continuous(breaks=c(2000,2003,2006,2009)) + 
  scale_colour_discrete(name="Statistic",
                        breaks=c("icecream2","drown"),
                        labels=c("Ice cream revenue","Drowning in pools"))
dm <- melt(df[c("drown", "icecream3", "year")],id.var=3)
plot2 <- ggplot(dm, aes(x=year, y=value, colour=variable)) + 
  geom_point(shape=1) +
  geom_smooth(method=loess, se=F) +
  labs(x="Year",
       y="% of average year") + 
  scale_x_continuous(breaks=c(2000,2003,2006,2009)) + 
  scale_colour_discrete(name="Statistic",
                        breaks=c("icecream3","drown"),
                        labels=c("Ice cream volume sold","Drowning in pools"))
dm <- melt(df[c("drown", "icecream", "year")],id.var=3)
plot3 <- ggplot(dm, aes(x=year, y=value, colour=variable)) + 
  geom_point(shape=1) +
  geom_smooth(method=loess, se=F) +
  labs(x="Year",
       y="% of average year") + 
  scale_x_continuous(breaks=c(2000,2003,2006,2009)) + 
  scale_colour_discrete(name="Statistic",
                        breaks=c("icecream","drown"),
                        labels=c("Ice cream price","Drowning in pools"))
```

::::: {.columns}

:::: {.column width="50%"}

- Ice cream revenue predicts pool drownings in the US
    - ???
    - Correlation is... <span class="fragment" data-fragment-index="1"> only `r cor_rev`</span>

::: {.fragment .fade-up fragment-index=2}

- What about units sold?</li>
    - Correlation is negative!!!
    - `r cor_unit`

:::

::: {.fragment .fade-up fragment-index=3}

- What about price?
    - Correlation is `r cor_price`

:::

::::

:::: {.column width="50%"}


::: {.fragment fragment-index=1}

```{r}
#| eval: true
#| echo: false
#| fig-width: 5
#| fig-height: 2
#| fig-align: center

ggplotly(plot1)
```

:::

::: {.fragment fragment-index=2}

```{r}
#| eval: true
#| echo: false
#| fig-width: 5
#| fig-height: 2
#| fig-align: center

ggplotly(plot3)
```

:::

::: {.fragment fragment-index=3}

```{r}
#| eval: true
#| echo: false
#| fig-width: 5
#| fig-height: 2
#| fig-align: center

ggplotly(plot2)
```

:::

::::

:::::

> This is where the "educated" comes in


## Forecasting analytics in this class {data-background="../Backgrounds/default.jpg"}

- Revenue/sales
- Shipping delays
- Bankruptcy
- Machine learning applications

![](../Course_Diagrams/Forecasting.png){width="1200px" fig-align="center"}

## What are forensic analytics? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Forensic analytics focus on *detection*
    - Detecting crime such as bribery
    - Detecting fraud within companies
    - <a href="https://www.kaggle.com/c/dog-breed-identification">Looking at a lot of dog pictures</a> to identify features unique to each breed 
:::

::: {.column width="50%"}
![](../Figures/magnifying_glass.svg){width="400px" fig-align="center"}
:::

::::



## Forensic analytics in this class {data-background="../Backgrounds/default.jpg"}

- Fraud detection
- Working with textual data
- Detecting changes
- Machine learning applications

![](../Course_Diagrams/Forensics.png){width="1200px" fig-align="center"}

## Forecasting vs forensic analytics {data-background="../Backgrounds/default.jpg"}

- Forecasting analytics requires a time dimension
    - Predicting *future* events
- Forensic analytics is about understanding or detecting something
    - Doesn't need a time dimension, but it can help


> These are not mutually exclusive.  Forensic analytics can be used for forecasting!

# Who uses analytics? {data-background-video="../Backgrounds/section.mp4"}

## In general {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Companies
    - Finance
    - Manufacturing
    - Transportation
    - Computing
    - ...
:::

::: {.column width="50%"}
- Governments
    - AI.Singapore
    - Big data office
    - "Smart" initiatives
- Academics
- Individuals!
:::

::::

> 53% of companies were using big data in a <a href="https://www.forbes.com/sites/louiscolumbus/2017/12/24/53-of-companies-are-adopting-big-data-analytics/#299f766239a1">2017 survey</a>!

## What do companies use analytics for? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Customer service
    - <a href="https://www.forbes.com/sites/tomgroenfeldt/2018/05/03/rbs-uses-analytics-to-make-customer-service-more-than-just-a-slogan/#3d02f7d42108">Royal Bank of Scotland</a>
        - Understanding customer complaints
- Improving products
    - Siemens' <a href="https://www.forbes.com/sites/bernardmarr/2017/05/30/how-siemens-is-using-big-data-and-iot-to-build-the-internet-of-trains/#3b22cfd372b8">Internet of Trains</a>
        - Improving train reliability
- Their business
    - <a href="https://www.gartner.com/newsroom/id/3612617">\$18.3B USD market in 2017</a>
        - Just a small portion of overall IT spending (<a href="https://www.businesswire.com/news/home/20180116005566/en/Gartner-Global-Spending-Reach-3.7-Trillion-2018">\$3.7T USD</a>)
:::

::: {.column width="50%"}
<a href="https://www.forbes.com/sites/tomgroenfeldt/2018/05/03/rbs-uses-analytics-to-make-customer-service-more-than-just-a-slogan/#3d02f7d42108"><img src="../Figures/RBS.png" width="120px"/></a>
<a href="https://www.forbes.com/sites/bernardmarr/2017/05/30/how-siemens-is-using-big-data-and-iot-to-build-the-internet-of-trains/#3b22cfd372b8"><img src="../Figures/Siemens-logo.png" width="400px"/></a>
<a href="https://www.gartner.com/newsroom/id/3612617"><img src="../Figures/Gartner_logo.svg" width="400px"></a>
:::

::::


## What do governments use analytics for? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- [Govtech](https://www.tech.gov.sg/)
- Open data
    - [Data.gov.sg](https://data.gov.sg/)
    - [City of New York](https://opendata.cityofnewyork.us/)
- [AI Singapore](https://www.aisingapore.org/)
    - Talent matching
        - [100 Experiments](https://www.aisingapore.org/100e/)
    - [Grand Challenges](https://aisingapore.org/technology/grand-challenges/)
:::

::: {.column width="50%"}
<a href="https://www.tech.gov.sg/"><img src="../Figures/govtech.gif" width="300px"/></a>

<a href="https://data.gov.sg/"><img src="../Figures/dataSG.svg" width="300px"/></a>

<a href="https://www.aisingapore.org/"><img src="../Figures/AIsingapore-logo.png" width="300px"/></a>

:::

::::


## What do academics use analytics for? {data-background="../Backgrounds/internet.jpg"}

:::: {.columns}

::: {.column width="50%"}
![](../Figures/tweets_by_day_5min.png){width="600px" fig-align="center"}
:::

::: {.column width="50%"}
- Tweeting frequency by S\&P 1500 companies (<a href="https://ssrn.com/abstract=3105847">paper</a>)
- Aggregates every tweet from 2012 to 2016
- Shows frequency in 5 minute chunks
    - Note the spikes every hour!
- The white part is the time the NYSE is open
:::

::::

## What do academics use analytics for? {data-background="../Backgrounds/fraud.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Annual report content that predicts fraud (<a href="https://ssrn.com/abstract=2803733">paper</a>)
- For instance, discussing income is useful
    - first row is decreases, second is increases
    - But if it's good or bad depends on the year
    - For instance, in 1999 it is a red flag
        - And one that Enron is flagged for
:::

::: {.column width="50%"}
![](../Figures/Topic_Fraud_over_time_AAER.png){width="500px" fig-align="center"}
:::

::::


## What do academics use analytics for? {data-background="../Backgrounds/COVID.jpg"}

:::: {.columns}

::: {.column width="50%"}
- COVID-19 fear on Twitter, 2020 Mar to Oct
- Helps us to understand socieconomic reactions to events

> What do the spikes indicate?

<!--# Passing 100k deaths in US and Trump getting COVID -->

:::

::: {.column width="50%"}

<center><video src="../Figures/Fear_USA-v2.webm" height="400px" autoplay muted loop></video></center>

:::

::::



> To learn more, [check out these slides](https://rmc.link/articles/smuphd2022.html)

## What do individuals use analytics for? {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
![](../Figures/wearables.svg){width="400px" fig-align="center"}
:::

::: {.column width="50%"}
- Consulting
    - <a href="https://radimrehurek.com/about/">Radim Rehurek</a>: Maintainer of <a href="https://radimrehurek.com/gensim/" class="ref">gensim</a>, freelance consultant
- Investing
    - <a href="https://quantnet.com/forum/">Quantnet discussions</a>
- Health
    - Smart watches and other wearables
:::

::::


## Why should you learn analytics? {data-background="../Backgrounds/default.jpg"}

- Important skill for understanding the world
    - <a href="https://www.datacamp.com/community/blog/hidden-revolution-data-science">Good timing to learn it, too!</a>
- Gives you an edge over many others
    - Particularly useful for your career

- Jobs for "Management analysts" are expected to expand by 14% from 2016 to 2026
    - Accountants and auditors: 10%
    - Financial analysts: 11%
    - Average industry: 7%
    - All figures from US Bureau of Labor Statistics

# Review of R {data-background-video="../Backgrounds/section.mp4"}

## What is R? {data-background="../Backgrounds/default.jpg"}

- R is a "statistical programming language"
    - Focused on data handling, calculation, data analysis, and visualization
- We will use R for all work in this course

<a href="https://www.r-project.org/about.html"><img src="../Figures/R.svg" width="400px"></a>

## Why do we need R? {data-background="../Backgrounds/default.jpg"}

- Analytics deals with more data than we can process by hand
    - We need to ask a computer to do the work!
- R is one of the de facto standards for analytics work
    - Third most popular language for data analytics and machine learning (<a href="https://www.kdnuggets.com/2017/01/most-popular-language-machine-learning-data-science.html">source</a>)
    - Fastest growing of all mainstream languages
    - Free and open source, so you can use it anywhere
    - It can do most any analytics
    - Not a general programming language

> Programming in R provides a way of talking with the computer to make it do what you want it to do

## Alternatives to R {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
<a href="https://www.python.org"><img src="../Figures/python-logo.png" height="100px" style="margin:0px"></a>

- Extremely popular
- Free and open source
- Very strong AI/ML support

<a href="https://www.tensorflow.org/"><img src="../Figures/tensorflow_logo.png" height="100px"></a><br>
<a href="https://www.microsoft.com/en-us/cognitive-toolkit/"><img src="../Figures/cntk.png" height="100px"></a>
<a href="https://pytorch.org/"><img src="../Figures/pytorch.svg" height="100px"></a>
<a href="https://www.h2o.ai/"><img src="../Figures/h2o_logo.svg" height="100px"></a>
:::

::: {.column width="50%"}
<a href="https://julialang.org"><img src="../Figures/julia-logo.png" height="100px" style="margin:0px"></a>

- Fast and free
- Mathematics oriented
- Still young though

<a href="https://www.scala-lang.org"><img src="../Figures/Scala-logo.png" height="100px" style="margin:0px"></a>

- Fast and free
- Focused on scalability, basis of Apache Spark
:::

::::


# Setup for R {data-background-video="../Backgrounds/section.mp4"}

## Setup {data-background="../Backgrounds/default.jpg"}

- For this class, I will assume you are using RStudio with the default R installation
    - <a href="https://www.rstudio.com/products/rstudio/download/#download">RStudio downloads</a>
    - <a href="https://cloud.r-project.org/bin/windows/base/">R for Windows</a>
    - <a href="https://cloud.r-project.org/bin/macosx/">R for (Max) OS X</a> (Download R-4.1.1.pkg)
    - <a href="https://cloud.r-project.org/bin/linux/">R for Linux</a>
- For the homeworks and Code files, also install [Quarto](https://quarto.org/)
    - Quarto is the next-gen version of R markdown
- For the most part, everything will work the same across all computer types
- Everything in these slides was tested on R 4.1 and 4.3 on Windows and Linux

## How to use R Studio {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
1. R markdown or Quarto file
    - You can write out reports with embedded analytics
2. Console
    - Useful for testing code and exploring your data
    - Enter your code one line at a time
3. R Markdown console
    - Shows if there are any errors when preparing your report
:::

::: {.column width="50%"}
![](../Figures/RStudio_1-annotated.png){width="350px" fig-align="center"}
:::

::::

## How to use R Studio {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
4. Environment
    - Shows all the values you have stored
5. Help
    - Can search documentation for instructions on how to use a function
6. Viewer
    - Shows any output you have at the moment.
7. Files
    - Shows files on your computer
:::

::: {.column width="50%"}
![](../Figures/RStudio_2-annotated.png){width="350px" fig-align="center"}
:::

::::


# Basic R commands {data-background-video="../Backgrounds/section.mp4"}

## Arithmetic {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Anything in boxes like those on the right in my slides are R code
- The slides themselves are made in R, so you could copy and paste any code in the slides right into R to use it yourself
- Blue boxes: Code
    - Lines starting with `#` are comments
        - They only explain what the code does
- White boxes: Output
:::

::: {.column width="50%"}
```{r, eval=TRUE}
# Addition uses '+'
1 + 1

# Subtraction uses '-'
2 - 1

# Multiplication uses '*'
3 * 3

# Division uses '/'
4 / 2
```
:::

::::


## Arithmetic {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Exponentiation
    - Write $x^y$ as `x ^ y`
- Modulus
    - The remainder after division
    - Ex.: $46\text{ mod }6 = 4$
        1. $6 \times 7 = 42$
        2. $46 - 42 = 4$
        3. $4 < 6$, so 4 is the remainder
- Integer division (not used often)
    - Like division, but it drops any decimal
:::

::: {.column width="50%"}
```{r, eval=TRUE}
# Exponentiation uses '^'
5 ^ 5

# Modulus (aka the remainder) uses '%%'
46 %% 6

# Integer division uses '%/%'
46 %/% 6

```
:::

::::


## Variable assignment {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Variable assignment lets you give something a name
    - This lets you easily reuse it
- In R, we can name almost anything that we create
    - Values
    - Data
    - Functions
    - etc...
- We will name things using the `<-` operator
:::

::: {.column width="50%"}
```{r eval=TRUE}
# Store 2 in 'x'
x <- 2

# Check the value of x
x

# Store arithmetic in y
y <- x * 2

# Check the value of y
y
```
:::

::::


## Variable assignment {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Note that values are calculated at the time of assignment
- We previously set `y <- 2 * x`
- If we change the values of `x` and `y` remain unchanged!
:::

::: {.column width="50%"}
```{r, eval=TRUE}
# Previous value of x and y
cat(x, y)

# Change x, then recheck the value
# of x and y
x <- 200

cat(x, y)
```
:::

::::


## Application: Singtel's earnings growth {data-background="../Backgrounds/case.jpg"}

> Set a variable `growth` to the amount of Singtel's earnings growth percent in 2018

```{r Singtel, eval=TRUE}
# Data from Singtel's earnings reports, in Millions of SGD
singtel_2017 <- 3831.0
singtel_2018 <- 5430.3

# Compute growth
growth <- singtel_2018 / singtel_2017 - 1

# Check the value of growth
growth
```

## Recap {data-background="../Backgrounds/default.jpg"}

- So far, we are using R as a glorified calculator
- The key to using R is that we can scale this up with little effort
    - Calculating *every* public companies' earnings growth isn't much harder than calculating Singtel's!

> Scaling this up will give use a lot more value

- How to scale up:
    1. Use data structures to hold collections of data
        - Could calculate growth for **all** companies instead of just Singtel, using the same basic structure
    2. Leverage **functions** to automate more complex operations
        - There are many functions built in, and **many** more freely available

# Data structures {data-background-video="../Backgrounds/section.mp4"}

## Data types {data-background="../Backgrounds/default.jpg"}

:::: {.columns}

::: {.column width="50%"}
- Numeric: Any number
    - Positive or negative
    - With or without decimals
- Boolean: `TRUE` or `FALSE`
    - Capitalization matters!
    - Shorthand is `T` and `F`
- Character: "text in quotes"
    - More difficult to work with
    - You can use either single or double quotes
- Factor: Converts text into numeric data
    - Categorical data
:::

::: {.column width="50%"}
```{r, eval=TRUE}
company_name <- "Google"  # character
company_name

company_name <- 'Google'  # character
company_name

tech_firm <- TRUE  # boolean
tech_firm

earnings <- 12662  # numeric, $M USD
earnings
```
:::

::::


## Scaling up... {data-background="../Backgrounds/default.jpg"}

- We already have some data entered, but it's only a small amount
- We can scale this up using ...
    - Vectors using `c()` -- holds only 1 type
    - Matrices using `matrix()` -- holds only 1 type
    - Lists using `list()` -- holds anything (including other structures)
    - Data frames using `data.frame()` -- holds different types by column

## Vectors: What are they? {data-background="../Backgrounds/default.jpg"}

- Remember back to linear algebra...

Examples:

$$
\begin{matrix}
\left(\begin{matrix}
1 \\
2 \\
3 \\
4
\end{matrix}\right) & \text{or} & 
\left(\begin{matrix}
1 & 2 & 3 & 4
\end{matrix}\right)
\end{matrix}
$$

> A row (or column) of data

## Vector example: Profit margin for tech firms {data-background="../Backgrounds/case.jpg"}

```{r, include=FALSE}
library(tidyverse)
df <- read.csv("../../Data/Session_1-2.csv", stringsAsFactors=FALSE)
# Filter to firms with at least $1M USD revenue, known net income, and fiscal year of 2017
clean_df <- df %>% filter(fyear==2017, !is.na(revt), !is.na(ni), revt > 1)
tech_df <- clean_df %>%
  filter(gsector==45) %>%
  mutate(revenue = revt,
         earnings = ni,
         margin = ni/revt)
earnings_2017 <- tech_df$ni
revenue_2017 <- tech_df$revt
names_2017 <- tech_df$conm
names(earnings_2017) <- names_2017
names(revenue_2017) <- names_2017
```

```{r, eval=TRUE}
# Calculating profit margin for all public US tech firms
# 715 tech firms in Compustat with >1M sales in 2017

# Data:
#    earnings_2017: vector of earnings, $M USD
#    revenue_2017: vector of revenue, $M USD
#    names_2017: a vector of tickers (strings)

# Naming the vectors
names(earnings_2017) <- names_2017
names(revenue_2017) <- names_2017

earnings_2017[1:6]
revenue_2017[1:6]
```

## Vector example: Profit margin for tech firms {data-background="../Backgrounds/case.jpg"}

```{r, eval=TRUE}
# Summarizing vectors
summary(earnings_2017)
```
<div class="fragment">
```{r, eval=TRUE}
summary(revenue_2017)
```
</div>
<div class="fragment">
```{r, eval=TRUE}
# Calculating profit margin
margin <- earnings_2017 / revenue_2017
summary(margin)
```
</div>
<div class="fragment">
```{r, eval=TRUE}
# Worst, midpoint, and best profit margin firms in 2017. Our names carried over :)
margin[order(margin)][c(1, length(margin) / 2, length(margin))]
```
</div>

## Matrices: What are they? {data-background="../Backgrounds/default.jpg"}

- Remember back to linear algebra...

Example:

$$
\left(\begin{matrix}
1 & 2 & 3 & 4\\
5 & 6 & 7 & 8\\
9 & 10 & 11 & 12
\end{matrix}\right)
$$

> A rows *and* columns of data

## Selecting from matrices {data-background="../Backgrounds/default.jpg"}

- Select using 2 indexes instead of 1:
    - `matrix_name[rows,columns]`
    - To select all rows or columns, leave that index blanks

:::: {.columns}

::: {.column width="50%"}
```{r}
columns <- c("Google", "Microsoft", "Goldman")
rows <- c("Earnings","Revenue")

firm_data <- matrix(data=c(12662, 21204, 4286,
  110855, 89950, 42254), nrow=2)
# Equivalent:
# matrix(data=c(12662, 21204, 4286,
#   110855, 89950, 42254), ncol=3)

# Apply names
rownames(firm_data) <- rows
colnames(firm_data) <- columns

# Print the matrix
firm_data
```
:::

::: {.column width="50%"}
```{r}
firm_data[2, 3]
firm_data[, c("Google", "Microsoft")]
firm_data[1,]
firm_data["Revenue", "Goldman"]
```
:::

::::


## Combining matrices {data-background="../Backgrounds/default.jpg"}

- Matrices are combined top to bottom as rows with `rbind()`
- Matrices are combined side-by-side as columns with `cbind()`

```{r,include=FALSE}
indcode <- c(45,45,40)
jpdata <- c(17370, 115475)
# Preloaded: industry codes as indcode (vector)
```

```{r}
# Preloaded: industry codes as indcode (vector)
#     - GICS codes: 40=Financials, 45=Information Technology
#     - See: https://en.wikipedia.org/wiki/Global_Industry_Classification_Standard
# Preloaded: JPMorgan data as jpdata (vector)

mat <- rbind(firm_data,indcode)  # Add a row
rownames(mat)[3] <- "Industry"  # Name the new row
mat
mat <- cbind(firm_data,jpdata)  # Add a column
colnames(mat)[4] <- "JPMorgan"  # Name the new column
mat
```

## Lists: What are they? {data-background="../Backgrounds/default.jpg"}

- Like vectors, but with mixed types
- Generally not something we will create
- Often returned by analysis functions in R

```{r}
# Ignore this code for now...
model <- summary(lm(earnings ~ revenue, data=tech_df))
#Note that this function is hiding something...
model
```

## Looking into lists {data-background="../Backgrounds/default.jpg"}

- Lists generally use double square brackets, `[[index]]`
    - Used for pulling individual elements out of a list
- `[[c()]]` will drill through lists, as opposed to pulling multiple values
- Single square brackets pull out elements as is
- Double square brackets extract just the element
- For 1 level, we can also use `$`

:::: {.columns}

::: {.column width="50%"}
```{r}
model["r.squared"]
model[["r.squared"]]
model$r.squared
```
:::

::: {.column width="50%"}
```{r}
earnings <- c(12662, 21204, 4286)
company <- c("Google", "Microsoft", "Goldman")
names(earnings) <- company
earnings["Google"]
earnings[["Google"]]
#Can't use $ with vectors
```
:::

::::


## Structure of a list {data-background="../Backgrounds/default.jpg"} 
- `str()` will tell us what's in this list

```{r}
str(model)
```

## What are data frames? {data-background="../Backgrounds/default.jpg"}

- Data frames are like a hybrid between lists and matrices

:::: {.columns}

::: {.column width="50%"}
Like a matrix:

- 2 dimensional like matrices
- Can access data with `[]`
- All elements in a column must be the same data type
:::

::: {.column width="50%"}
Like a list:

- Can have different data types for different columns
- Can access data with `$`
:::

::::

> Columns $\approx$ variables, e.g., earnings

> Rows $\approx$ observations, e.g., Google in 2017

## Three schools of thought on data handling {data-background="../Backgrounds/default.jpg"}

1. Use *Base R* functions (i.e., what's built in)
    - Tends to be tedious
2. Use *tidy* methods (from `{tidyverse}`)
    - Almost always cleaner and more readable
    - Usually faster, sometimes slower
    - This creates a structure called a `tibble`
3. Use *data.table* (from `{data.table}`)
    - Very structured syntax, but difficult to read
    - Almost always fastest and memory efficient -- use when speed is needed
    - This creates a structure called a `data.table`

::: {.callout-tip}
## title.optional

You can case `tibble` and `data.table` objects to a `data.frame` using `as.data.frame()`

:::

## Data in Base R {data-background="../Backgrounds/default.jpg"}

> Note: Base R methods are explained in the <a href="https://rmc.link/Slides/acct420v5/Session_1_Sup/Session_1_Sup.html">R Supplement</a>

```{r}
library(tidyverse)  # Imports most tidy packages
# Base R data import -- stringsAsFactors is important here
df <- read.csv("../../Data/Session_1-2.csv", stringsAsFactors=FALSE)
df <- subset(df, fyear == 2017 & !is.na(revt) & !is.na(ni) &
             revt > 1 & gsector == 45)
df$margin = df$ni / df$revt
summary(df)
```

## Data the tidy way {data-background="../Backgrounds/default.jpg"}
```{r, message=F}
# Tidy import
df <- read_csv("../../Data/Session_1-2.csv") %>%
  filter(fyear == 2017,      # fiscal year
         !is.na(revt),       # revenue not missing
         !is.na(ni),         # net income not missing
         revt > 1,           # at least 1M USD in revenue
         gsector == 45) %>%  # tech firm
  mutate(margin = ni/revt)   # profit margin
summary(df)
```

## Other important tidy methods {data-background="../Backgrounds/default.jpg"}

- Sorting: use `arrange()`
- Grouping for calculations:
    - Group using `group_by()`
    - Ungroup using `ungroup()` once you are done
- Keep only a subset of variables using `select()`
- We'll see many more along the way!

## A note on syntax: Piping {data-background="../Backgrounds/default.jpg"}

> Pipe notation is never necessary and not built in to R

- Piping comes from `{magrittr}`
    - The `%>%` pipe is loaded with `{tidyverse}`
- Pipe notation is done using `%>%`
    - `Left %>% Right(arg2, ...)` is the same as `Right(Left, arg2, ...)`

::: {.callout-note}
## Why Pipes?

Piping can drastically improve code readability. Piped code tells you the list of operations applied to a single starting object, such as a data frame.

:::

- `{magrittr}` has other interesting pipes, such as `%<>%`
    - `Left %<>% Right(arg2, ...)` is `Left <- Right(Left, arg2, ...)`
    - To load these other pipes, you need to explicitly load `{magrittr}`

## Tidy example without piping {data-background="../Backgrounds/default.jpg"}

```{r, message=F}
df <- mutate(
        filter(
          read_csv("../../Data/Session_1-2.csv"),
          fyear == 2017,      # fiscal year
          !is.na(revt),       # revenue not missing
          !is.na(ni),         # net income not missing
          revt > 1,           # at least 1M USD in revenue
          gsector == 45),  # tech firm
        margin = ni/revt)   # profit margin
summary(df)
```


## Practice: Data types and structures {data-background="../Backgrounds/coding.jpg"}

- This practice is to make sure you understand data types
- Do exercises 1 through 3 on today's R practice file:
    - [R Practice](https://rmc.link/420r1)
    - Short link: [rmc.link/420r1](https://rmc.link/420r1)

# Useful functions {data-background-video="../Backgrounds/section.mp4"}

## Reference {data-background="../Backgrounds/default.jpg"}

> Many useful functions are highlighted in the <a href="https://rmc.link/Slides/acct420v4/Session_1_Sup/Session_1_Sup_post.html">R Supplement</a>

1. Installing and loading packages

```{r, eval=FALSE}
# Install the tidyverse package from inside R
install.packages("tidyverse")

# Load the package
library(tidyverse)
```

2. Help functions

```{r, eval=FALSE}
# To see a help page for a function (such as data.frame()) run either of:
help(data.frame)
?data.frame
```

```{r}
# To see the arguments a function takes, run:
args(data.frame)
```

## Making your own functions! {data-background="../Backgrounds/default.jpg"}

- Use the `function()` function!
    - `my_func <- function(agruments) {code}`

> Simple function: Add 2 to a number

```{r}
add_two <- function(n) {
  n + 2
}

add_two(500)
```

## Slightly more complex function example {data-background="../Backgrounds/default.jpg"}

```{r}
mult_together <- function(n1, n2=0, square=FALSE) {
  if (!square) {
    n1 * n2
  } else {
    n1 * n1
  }
}

mult_together(5,6)
mult_together(5,6,square=TRUE)
mult_together(5,square=TRUE)
```

## Example: Currency conversion function {data-background="../Backgrounds/case_digital.jpg"}

```{r, warning=F, message=F}
FXRate <- function(from="USD", to="SGD", dt=Sys.Date()) {
  options("getSymbols.warning4.0"=FALSE)
  if (!require(quantmod)) return("Install the quantmod package for this function to work")
  data <- getSymbols(paste0(from, "/", to), from=dt-3, to=dt, src="oanda", auto.assign=F)
  return(data[[1]])
}
date()
FXRate(from="USD", to="SGD")  # Today's SGD to USD rate
FXRate(from="SGD", to="CNY")  # Today's SGD to CNY rate
FXRate(from="USD", to="SGD", dt=Sys.Date()-90)  # Last quarter's SGD to USD rate
```

## Practice: Functions {data-background="../Backgrounds/coding.jpg"}

- This practice is to make sure you understand functions and their construction
- Do exercises 4 and 5 on today's R practice file:
    - [R Practice](https://rmc.link/420r1)
    - Short link: [rmc.link/420r1](https://rmc.link/420r1)


# End Matter {data-background-video="../Backgrounds/section.mp4"}

## Wrap up {data-background="../Backgrounds/ending.jpg"}

- For next week:
    - Take a look at Datacamp!
        - Be sure to complete the assignment there
        - A complete list of assigned modules over the course is on eLearn
    - We'll start in on some light analytics next week
- Survey on the class session at this QR code:

{{< qrcode https://docs.google.com/forms/d/e/1FAIpQLSc9rHKLMH-ZfZwI-LzhmV-2sVnW5Md6muW2h1BFLxCwixpvDQ/viewform?usp=sf_link qrsurvey width=200 height=200 >}}



## Packages used for these slides {data-background="../Backgrounds/code.jpg" visibility="uncounted"}

- `{DT}`
- `{downlit}`
- `{kableExtra}`
- `{knitr}`
- `{ngramr}`
- `{plotly}`
- `{quantmod}`
- `{quarto}`
- `{revealjs}`
- `RColorBrewer`
- `{tidyverse}`


# Appendix: Getting data from WRDS {data-background-video="../Backgrounds/section.mp4" visibility="uncounted"}

## Data Sources {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

- WRDS
    - WRDS is a provider of business data for academic purposes
    - Through your class account, you can access vast amounts of data
    - We will be particularly interested in:
        - Compustat (accounting statement data since 1950)
        - CRSP (stock price data, daily since 1926)
- We will use other public data from time to time
    - Singapore's big data repository
    - US Government data
    - Other public data collected by the Prof

<img src="../Figures/WRDS.png" height="50px"> <img src="../Figures/dataSG.svg" height="50px"> <img src="../Figures/dataUS.png" height="50px">

## How to download from WRDS {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

1. Log in using a class account (posted on eLearn)
2. Pick the data provider that has your needed data
3. Select the data set you would like (some data sets only)
4. Apply any needed conditional restrictions (years, etc.)
    - These can help keep data sizes manageable
        - CRSP without any restrictions is >10 GB
5. Select the specific variables you would like export
6. Export as a csv file, zipped csv file (or other format)

> A picture walkthrough is on the next set of slides

## Go to <a href="https://wrds-web.wharton.upenn.edu/">WRDS</a> and sign in {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS1.PNG){width="1000px" fig-align="center"}

## Pick a data provider, e.g. "Compustat - Capital IQ" {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS2.PNG){width="1000px" fig-align="center"}

## Pick a data set, e.g. "North America - Daily" {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS3.PNG){width="1000px" fig-align="center"}

## Pick a data set, e.g. "Fundamentals Annual" {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS4.PNG){width="1000px" fig-align="center"}

## Selecting data: Time range {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS5.PNG){width="1000px" fig-align="center"}

## Selecting data: Companies and data format {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS6.PNG){width="1000px" fig-align="center"}

## Selecting data fields {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS7.PNG){width="1000px" fig-align="center"}

## Select output formats {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS8.PNG){width="1000px" fig-align="center"}

## Wait for the data to be prepared {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS9.PNG){width="1000px" fig-align="center"}

## Download the data! {data-background="../Backgrounds/default.jpg" visibility="uncounted"}

![](../Figures/WRDS10.PNG){width="1000px" fig-align="center"}
