Univariate Visualization & Summaries

Notes and in-class exercises

You can download the .qmd file for this activity here and open in R-studio. The rendered version is posted in the course website (Activities tab). I often experiment with the class activities (and see it in live!) and make updates, but I always post the final version before class starts. To be sure you have the most up-to-date copy, please download it once you’ve settled in before class begins.

Notes

Learning goals

By the end of this activity, you should be able to:

What summarizations and visualizations are appropriate for categorical or quantitative variables
Write R code to read in data and to summarize and visualize a single variable at a time.
Interpret key features of barplots, boxplots, histograms, and density plots
Describe information about the distribution of a quantitative variable using the concepts of shape, center, spread, and outliers
Relate summary statistics of data to the concepts of shape, center, spread, and outliers

Readings and videos

You should have gone through the followings before the class and finished checkpoint quizzes!

Reading: Sections 2.1-2.4, 2.6 in the STAT 155 Notes
Videos:
- Univariate summaries (slides)
  - Part 1
  - Part 2
- R Code for Categorical Visualization and Summarization
- R Code for Quantitative Visualization and Summarization
- Quarto docs

File organization: Save this file in the “Activities” subfolder of your “STAT155” folder.

Exercises

Guiding question: What anxieties have been on Americans’ minds over the decades?

Context: Dear Abby is America’s longest running advice column. Started in 1956 by Pauline Phillips under the pseudonym Abigail van Buren, the column continues to this day under the stewardship of her daughter Jeanne. Each column features one or more letters to Abby from anonymous individuals, all signed with a pseudonym. Abby’s response follows each letter.

In 2018, the data journalism site The Pudding published a visual article called 30 Years of American Anxieties in which the authors explored themes in Dear Abby letters from 1985 to 2017. (We only have the questions, not Abby’s responses.) The codebook is available here.

Exercise 1: Importing and getting to know the data

First, in the Console pane of RStudio, run the following command to install some necessary packages (you will need to do this any time you are installing a new package):

install.packages("tidyverse") # We have already installed it!

Now, in the Quarto pane, run the following code chunk to load the package and load a dataset (you can either click the green arrow in the top right of the code chunk, put your cursor in the code chunk and hit Ctrl+Alt+C [on Windows/Linux] or Command+Option+C [on Mac]).

# Load package
library(tidyverse)

# Read in the Dear Abby data
abby <- read_csv("https://mac-stat.github.io/data/dear_abby.csv")

Throughout this activity, we’ll work only with the most recent year of data, from 2017. Run the following chunk:

# Wrangle the Dear Abby data
# Ignore this code for now!
abby <- abby %>% 
  filter(year == 2017) %>% 
  mutate(month = month(month, label = TRUE)) %>%
  mutate(
    parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"),
    marriage = str_detect(question_only, "marriage|marry|married"),
    money = str_detect(question_only, "money|finance")
  ) %>%
  rowwise() %>%
  mutate(
    themes = c(
      if (parents) "parents",
      if (marriage) "marriage",
      if (money) "money"
    ) %>% paste(collapse = ", "),
    themes = ifelse(themes == "", "other", themes)
  ) %>%
  ungroup() %>% 
  select(year, month, day, question_only, bing_pos, afinn_overall, afinn_pos, afinn_neg, themes)

Click on the Environment tab (generally in the upper right hand pane in RStudio). Then click the abby line. The abby data will pop up as a separate pane (like viewing a spreadsheet) – check it out.
In this tidy dataset, what is the unit of observation? That is, what is represented in each row of the dataset?
What term do we use for the columns of the dataset?
Try out each function below. Identify what each function tells you about the abby data and note this in the ???:

# ??? [what do both numbers mean?]
dim(abby)
## [1] 514   9

# ???
nrow(abby)
## [1] 514

# ???
ncol(abby)
## [1] 9

# ???
head(abby)
## # A tibble: 6 × 9
##    year month day   question_only     bing_pos afinn_overall afinn_pos afinn_neg
##   <dbl> <ord> <chr> <chr>                <dbl>         <dbl>     <dbl>     <dbl>
## 1  2017 Aug   30    "i moved to the …    0.75             14        16         2
## 2  2017 Aug   30    "under what circ…   NA                NA        NA        NA
## 3  2017 Aug   28    "i'm not a dog p…    0.333             5         5         0
## 4  2017 Aug   28    "my 62-year-old …    0.143           -11         8        19
## 5  2017 Aug   27    "i have a friend…    0.222             0         7         7
## 6  2017 Aug   27    "i have been sel…    0.333            -5         2         7
## # ℹ 1 more variable: themes <chr>

# ???
names(abby)
## [1] "year"          "month"         "day"           "question_only"
## [5] "bing_pos"      "afinn_overall" "afinn_pos"     "afinn_neg"    
## [9] "themes"

[OPTIONAL] If you’re not sure how exactly to use a function, you can pull up a built-in help page with information about the arguments a function takes (i.e., what goes inside the parentheses), and the output it produces. To do this, click inside the Console pane, and enter ?function_name. For example, to pull up a help page for the dim() function, we can type ?dim and hit Enter. Try pulling up the help page for the read_csv() function we used to load the dataset.

Exercise 2: Preparing to summarize and visualize the data

In the next exercises, we will be exploring themes in the Dear Abby questions and the overall “mood” or sentiment of the questions. Before continuing, read the codebook for this dataset for some context about sentiment analysis, which gives us a measure of the mood/sentiment of a text.

What sentiment variables do we have in the dataset? Are they quantitative or categorical?
Check out the theme variable. Is this quantitative or categorical?
What visualizations are appropriate for looking at the distribution of a single quantitative variable? What about a single categorical variable?

Exercise 3: Exploring themes in the letters

The code below makes a barplot of the themes variable using the ggplot2 visualization package. Before making the plot, make note of what you expect the plot might look like. (This might be hard–just do your best!) Then compare to what you observe when you run the code chunk to make the plot. (Clearly defining your expectations first is good scientific practice to avoid confirmation bias.)

# Load package
# install.packages("ggplot2") # Run in R Console first time to install (copy without '#')
library(ggplot2)

# barplot
ggplot(abby, aes(x = themes)) +
    geom_bar() +
    theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1))

We can follow up on the barplot with a simple numerical summary. Whereas the ggplot2 package is great for visualizations, dplyr is great for numerical summaries. The code below constructs a table of the number of questions with each theme. Make sure that these numerical summaries match up with what you saw in the barplot.

# install.packages("dplyr")
# Construct a table of counts
abby %>% 
    count(themes)
## # A tibble: 8 × 2
##   themes                       n
##   <chr>                    <int>
## 1 marriage                    75
## 2 marriage, money              5
## 3 money                       21
## 4 other                      234
## 5 parents                    127
## 6 parents, marriage           33
## 7 parents, marriage, money     4
## 8 parents, money              15

Before proceeding, let’s break down the plotting code above. Run each chunk to see how the two lines of code above build up the plot in “layers”. Add comments (on the lines starting with #) to document what you notice.

# ???
ggplot(abby, aes(x = themes)) #sets up the "canvas" of the plot with axis labels

# ???
ggplot(abby, aes(x = themes)) +
    geom_bar() # Adds the bars- Any Problem you notice?

# ???
ggplot(abby, aes(x = themes)) +
    geom_bar() +
    theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) #Rotates the x axis labels

# ???
ggplot(abby, aes(x = themes)) +
    geom_bar() +
    theme_classic() +
    theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) # Changes the visual theme of the plot with a white background and removes gridlines

Exercise 4: Exploring sentiment

We’ll look at the distribution of the bing_pos sentiment variable and associated summary statistics.

The code below creates a boxplot of this variable. In the comment, make note of how this code is similar to the code for the barplot above. As in the previous exercise, before running the code chunk to create the plot, make note of what you expect the boxplot to look like.

# ???
ggplot(abby, aes(x = bing_pos)) +
    geom_boxplot()

Challenge: Using the code for the barplot and boxplot as a guide, try to make a histogram and a density plot of the overall average ratings.
- What information is given by the tallest bar of the histogram?
- How would you describe the shape of the distribution?

# Histogram

# Density plot

We can compute summary statistics (numerical summaries) for a quantitative variable using the summary() function or with the summarize() function from the dplyr package. (1st Qu. and 3rd Qu. stand for first and third quartile.) After inspecting these summaries, look back to your boxplot, histogram, and density plot. Which plots show which summaries most clearly?

# Summary statistics
# Using summary() - convenient for computing many summaries in one command
# Does not show the standard deviation
abby %>% 
    select(bing_pos) %>% 
    summary()
##     bing_pos     
##  Min.   :0.0000  
##  1st Qu.:0.1667  
##  Median :0.3333  
##  Mean   :0.3650  
##  3rd Qu.:0.5000  
##  Max.   :1.0000  
##  NA's   :19

# Using summarize() from dplyr
# Note that we use %>% to pipe the data into the summarize() function
# We need to use na.rm = TRUE because there are missing values (NAs)
abby %>% 
    summarize(mean(bing_pos, na.rm = TRUE), median(bing_pos, na.rm = TRUE), sd(bing_pos, na.rm = TRUE))
## # A tibble: 1 × 3
##   `mean(bing_pos, na.rm = TRUE)` median(bing_pos, na.rm…¹ sd(bing_pos, na.rm =…²
##                            <dbl>                    <dbl>                  <dbl>
## 1                          0.365                    0.333                  0.279
## # ℹ abbreviated names: ¹`median(bing_pos, na.rm = TRUE)`,
## #   ²`sd(bing_pos, na.rm = TRUE)`

Write a good paragraph describing the information in the histogram (or density plot) by discussing shape, center, spread, and outliers. Incorporate the numerical summaries from part c.

Pause: Math box

Below is an example of a “math box” which summarizes the formulas for some of the numerical summaries above. You are not required to memorize, nor will you be assessed on, any formulas presented in this or any future math box. They serve 3 purposes:

To emphasize that there’s “math” / a formal structure behind what we’re doing.
To provide students that plan to continue studying Statistics a glimpse into the formal statistical theory they’ll explore in later courses.
To make happy the students that are simply interested in math!

MATH BOX: Univariate numerical summaries

Let \((y_1, y_2, ..., y_n)\) be a sample of \(n\) data points.

mean: \[\overline{y} = \frac{y_1 + y_2 + \cdots + y_n}{n} = \frac{\sum_{i=1}^n y_i}{n}\]

variance: \[\text{var}(y) = \frac{(y_1 - \overline{y})^2 + (y_2 - \overline{y})^2 + \cdots + (y_n - \overline{y})^2}{n - 1} = \frac{\sum_{i=1}^n (y_i - \overline{y})^2}{n - 1}\]

standard deviation: \[\text{sd}(y) = \sqrt{\text{var}(y)}\]

Exercise 5: Box plots vs. histograms vs. density plots

We took 3 different approaches to plotting the quantitative average course variable above. They all have pros and cons.

What is one pro about the boxplot in comparison to the histogram and density plot?
What is one con about the boxplot in comparison to the histogram and density plots?
In this example, which plot do you prefer and why?

Exercise 6: Returning to our context, looking ahead

In this activity, we explored data on Dear Abby question, with a focus on exploring a single variable at a time.

In big picture terms, what have we learned about Dear Abby questions?
What further curiosities do you have about the data?

Exercise 7: Different ways to think about data visualization

In working with and visualizing data, it’s important to keep in mind what a data point represents. It can reflect the experience of a real person. It might reflect the sentiment in a piece of art. It might reflect history. We’ve taken one very narrow and technical approach to data visualization. Check out the following examples, and write some notes about anything you find interesting.

Exercise 8: Rendering your work

Save this file, and then click the “Render” button in the menu bar for this pane (blue arrow pointing right). This will create an HTML file containing all of the directions, code, and responses from this activity. A preview of the HTML will appear in the browser.

Scroll through and inspect the document to see how your work was translated into this HTML format. Neat!
Close the browser tab.
Go to the “Background Jobs” pane in RStudio and click the Stop button to end the rendering process.
Navigate to your “Activities” subfolder within your “STAT155” folder and locate the HTML file. You can open it again in your browser to double check.

Reflection

Go to the top of this file and review the learning objectives for this lesson. Which objectives do you have a good handle on, are at least familiar with, or are struggling with? What feels challenging right now? What are some wins from the day?

Response: Put your response here.

Advice: make an R code “cheat sheet”!

You will continue to pick up new R code and ideas. You’re highly encouraged to start tracking this in a cheat sheet (eg: in a Google doc). The cheat sheet will be a handy reference for you, and the act of making it will help deepen your understanding and retention.

Additional Practice

If you have time and want additional practice, try out the following exercises.

Exercise 9: Read in and get to know the weather data

Daily weather data are available for 3 locations in Perth, Australia.

View the codebook here.
Complete the code below to read in the data.

# Replace the ??? with your own name for the weather data
# Replace the ___ with the correct function
??? <- ___("https://mac-stat.github.io/data/weather_3_locations.csv")
## Error in parse(text = input): <text>:3:5: unexpected assignment
## 2: # Replace the ___ with the correct function
## 3: ??? <-
##        ^

Exercise 10: Exploring the data structure

Check out the basic features of the weather data.

# Examine the first six cases

# Find the dimensions of the data

What does a case represent in this data?

Exercise 11: Exploring rainfall

The raintoday variable contains information about rainfall.

Is this variable quantitative or categorical?
Create an appropriate visualization, and compute appropriate numerical summaries.
What do you learn about rainfall in Perth?

# Visualization

# Numerical summaries

Exercise 12: Exploring temperature

The maxtemp variable contains information on the daily high temperature.

Is this variable quantitative or categorical?
Create an appropriate visualization, and compute appropriate numerical summaries.
What do you learn about high temperatures in Perth?

# Visualization

# Numerical summaries

Exercise 13: Customizing! (CHALLENGE)

Though you will naturally absorb some RStudio code throughout the semester, being an effective statistical thinker and “programmer” does not require that we memorize all code. That would be impossible! In contrast, using the foundation you built today, do some digging online to learn how to customize your visualizations.

For the histogram below, add a title and more meaningful axis labels. Specifically, title the plot “Distribution of max temperatures in Perth”, change the x-axis label to “Maximum temperature” and y-axis label to “Number of days”. HINT: Do a Google search for something like “add axis labels ggplot”.

# Add a title and axis labels
ggplot(weather, aes(x = maxtemp)) + 
    geom_histogram()
## Error: object 'weather' not found

Adjust the code below in order to color the bars green. NOTE: Color can be an effective tool, but here it is simply gratuitous.

# Make the bars green
ggplot(weather, aes(x = raintoday)) + 
    geom_bar()
## Error: object 'weather' not found

Check out the ggplot2 cheat sheet. Try making some of the other kinds of univariate plots outlined there.
What else would you like to change about your plot? Try it!

Exercise 14: Optional challenge

At the top of this activity, we searched for words related to some topics of interest (parents, marriage, money) and combined them into a single theme variable. It looked something like this:

abby_new <- abby %>% 
  mutate(
    parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"),
    marriage = str_detect(question_only, "marriage|marry|married"),
    money = str_detect(question_only, "money|finance")
  ) %>%
  rowwise() %>%
  mutate(
    themes = c(
      if (parents) "parents",
      if (marriage) "marriage",
      if (money) "money"
    ) %>% paste(collapse = ", "),
    themes = ifelse(themes == "", "other", themes)
  ) %>%
  ungroup()

Check it out:

head(abby_new)
## # A tibble: 6 × 12
##    year month day   question_only     bing_pos afinn_overall afinn_pos afinn_neg
##   <dbl> <ord> <chr> <chr>                <dbl>         <dbl>     <dbl>     <dbl>
## 1  2017 Aug   30    "i moved to the …    0.75             14        16         2
## 2  2017 Aug   30    "under what circ…   NA                NA        NA        NA
## 3  2017 Aug   28    "i'm not a dog p…    0.333             5         5         0
## 4  2017 Aug   28    "my 62-year-old …    0.143           -11         8        19
## 5  2017 Aug   27    "i have a friend…    0.222             0         7         7
## 6  2017 Aug   27    "i have been sel…    0.333            -5         2         7
## # ℹ 4 more variables: themes <chr>, parents <lgl>, marriage <lgl>, money <lgl>

Understand the code!
- Inside mutate() the line parents = str_detect(question_only, "mother|mama|mom|father|papa|dad") created a new variable called parents. This variable takes on TRUE or FALSE. Explain what TRUE and FALSE mean here.
- The themes variable combines the information from the parents, marriage, and money variables. Check out the themes for the first 3 rows / data points. Convince yourself that you understand how it corresponds to the parents, marriage, and money variables.
Beyond parents, marriage, and money, what are some other topics that might pop up in the Dear Abby letters (and that you’re interested in exploring)? Modify the code below to explore those topics! Update the themes variable accordingly.

abby_new <- abby %>% 
  mutate(
    parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"),
    marriage = str_detect(question_only, "marriage|marry|married"),
    money = str_detect(question_only, "money|finance")
  ) %>%
  rowwise() %>%
  mutate(
    themes = c(
      if (parents) "parents",
      if (marriage) "marriage",
      if (money) "money"
    ) %>% paste(collapse = ", "),
    themes = ifelse(themes == "", "other", themes)
  ) %>%
  ungroup()

# Check out the raw data
head(abby_new)
## # A tibble: 6 × 12
##    year month day   question_only     bing_pos afinn_overall afinn_pos afinn_neg
##   <dbl> <ord> <chr> <chr>                <dbl>         <dbl>     <dbl>     <dbl>
## 1  2017 Aug   30    "i moved to the …    0.75             14        16         2
## 2  2017 Aug   30    "under what circ…   NA                NA        NA        NA
## 3  2017 Aug   28    "i'm not a dog p…    0.333             5         5         0
## 4  2017 Aug   28    "my 62-year-old …    0.143           -11         8        19
## 5  2017 Aug   27    "i have a friend…    0.222             0         7         7
## 6  2017 Aug   27    "i have been sel…    0.333            -5         2         7
## # ℹ 4 more variables: themes <chr>, parents <lgl>, marriage <lgl>, money <lgl>

# Check out the number of letters belonging to each theme
abby_new %>% 
  count(themes)
## # A tibble: 8 × 2
##   themes                       n
##   <chr>                    <int>
## 1 marriage                    75
## 2 marriage, money              5
## 3 money                       21
## 4 other                      234
## 5 parents                    127
## 6 parents, marriage           33
## 7 parents, marriage, money     4
## 8 parents, money              15

Solutions