--- title: "Univariate Visualization & Summaries" subtitle: "Notes and in-class exercises" format: html: embed-resources: true toc: true --- ```{r setup} #| include: false knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, error = TRUE, fig.height = 2.75, fig.width = 4.25, fig.env = 'figure', fig.pos = 'h', fig.align = 'center') ``` You can download the .qmd file for this activity [here](../activity_templates/02-foundations-univariate.qmd) and open in R-studio. The rendered version is posted in the [course website](https://mutasim221b.github.io/Mac-STAT-155-Fall-25/) (Activities tab). I often experiment with the class activities (and see it in live!) and make updates, but I always post the final version before class starts. To be sure you have the most up-to-date copy, please download it once you’ve settled in before class begins. # Notes ## Learning goals By the end of this activity, you should be able to: - What summarizations and visualizations are appropriate for **categorical** or **quantitative** variables - Write R code to read in data and to summarize and visualize a single variable at a time. - Interpret key features of barplots, boxplots, histograms, and density plots - Describe information about the distribution of a quantitative variable using the concepts of shape, center, spread, and outliers - Relate summary statistics of data to the concepts of shape, center, spread, and outliers ## Readings and videos You should have gone through the followings **before** the class and finished checkpoint quizzes! - Reading: Sections 2.1-2.4, 2.6 in the [STAT 155 Notes](https://mac-stat.github.io/Stat155Notes/) - Videos: - Univariate summaries ([slides](https://drive.google.com/file/d/1xnN-IWM7BUEXxlvIFYpkOvstRtOj1Hti/view?usp=sharing)) - [Part 1](https://www.youtube.com/watch?v=rEjOZgk2Z0Q) - [Part 2](https://www.youtube.com/watch?v=gQqoiMi4FFA) - [R Code for Categorical Visualization and Summarization](https://voicethread.com/share/14968914/) - [R Code for Quantitative Visualization and Summarization](https://voicethread.com/share/14923042/) - [Quarto docs](https://www.youtube.com/watch?v=8ykEm0_9sYI) **File organization:** Save this file in the "Activities" subfolder of your "STAT155" folder. # Exercises {-} \ > **Guiding question:** What anxieties have been on Americans' minds over the decades? **Context:** *Dear Abby* is America's longest running advice column. Started in 1956 by Pauline Phillips under the pseudonym Abigail van Buren, the column continues to this day under the stewardship of her daughter Jeanne. Each column features one or more letters to Abby from anonymous individuals, all signed with a pseudonym. Abby's response follows each letter. In 2018, the data journalism site [The Pudding](https://pudding.cool/) published a visual article called [30 Years of American Anxieties](https://pudding.cool/2018/11/dearabby/) in which the authors explored themes in Dear Abby letters from 1985 to 2017. (We only have the questions, not Abby's responses.) The codebook is available [here](https://github.com/Mac-STAT/data/blob/main/dear_abby_codebook.md). ## Exercise 1: Importing and getting to know the data First, in the **Console pane** of RStudio, run the following command to install some necessary packages (you will need to do this any time you are installing a new package): ``` install.packages("tidyverse") # We have already installed it! ``` Now, in the **Quarto pane**, run the following code chunk to load the package and load a dataset (you can either click the green arrow in the top right of the code chunk, put your cursor in the code chunk and hit Ctrl+Alt+C \[on Windows/Linux\] or Command+Option+C \[on Mac\]). ```{r} # Load package library(tidyverse) # Read in the Dear Abby data abby <- read_csv("https://mac-stat.github.io/data/dear_abby.csv") ``` Throughout this activity, **we'll work only with the most recent year of data, from 2017**. Run the following chunk: ```{r} # Wrangle the Dear Abby data # Ignore this code for now! abby <- abby %>% filter(year == 2017) %>% mutate(month = month(month, label = TRUE)) %>% mutate( parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"), marriage = str_detect(question_only, "marriage|marry|married"), money = str_detect(question_only, "money|finance") ) %>% rowwise() %>% mutate( themes = c( if (parents) "parents", if (marriage) "marriage", if (money) "money" ) %>% paste(collapse = ", "), themes = ifelse(themes == "", "other", themes) ) %>% ungroup() %>% select(year, month, day, question_only, bing_pos, afinn_overall, afinn_pos, afinn_neg, themes) ``` a. Click on the **Environment tab** (generally in the upper right hand pane in RStudio). Then click the `abby` line. The `abby` data will pop up as a separate pane (like viewing a spreadsheet) -- check it out. b. In this **tidy** dataset, what is the unit of observation? That is, what is represented in each row of the dataset? c. What term do we use for the columns of the dataset? d. Try out each function below. Identify what each function tells you about the `abby` data and note this in the `???`: ```{r explore} # ??? [what do both numbers mean?] dim(abby) ``` ```{r} # ??? nrow(abby) ``` ```{r} # ??? ncol(abby) ``` ```{r} # ??? head(abby) ``` ```{r} # ??? names(abby) ``` e. **\[OPTIONAL\]** If you're not sure how exactly to use a function, you can pull up a built-in help page with information about the arguments a function takes (i.e., what goes inside the parentheses), and the output it produces. To do this, click inside the Console pane, and enter `?function_name`. For example, to pull up a help page for the `dim()` function, we can type `?dim` and hit Enter. Try pulling up the help page for the `read_csv()` function we used to load the dataset. ## Exercise 2: Preparing to summarize and visualize the data In the next exercises, we will be exploring themes in the Dear Abby questions and the overall "mood" or sentiment of the questions. Before continuing, read the [codebook](https://github.com/Mac-STAT/data/blob/main/dear_abby_codebook.md) for this dataset for some context about sentiment analysis, which gives us a measure of the mood/sentiment of a text. a. What sentiment variables do we have in the dataset? Are they **quantitative** or **categorical**? b. Check out the `theme` variable. Is this **quantitative** or **categorical**? c. What visualizations are appropriate for looking at the distribution of a single quantitative variable? What about a single categorical variable? ## Exercise 3: Exploring themes in the letters a. The code below makes a **barplot** of the `themes` variable using the `ggplot2` visualization package. *Before making the plot*, make note of what you expect the plot might look like. (This might be hard--just do your best!) *Then* compare to what you observe when you run the code chunk to make the plot. (Clearly defining your expectations first is good scientific practice to avoid confirmation bias.) ```{r} # Load package # install.packages("ggplot2") # Run in R Console first time to install (copy without '#') library(ggplot2) # barplot ggplot(abby, aes(x = themes)) + geom_bar() + theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) ``` b. We can follow up on the barplot with a simple numerical summary. Whereas the `ggplot2` package is great for visualizations, `dplyr` is great for numerical summaries. The code below constructs a table of the number of questions with each theme. Make sure that these numerical summaries match up with what you saw in the barplot. ```{r} # install.packages("dplyr") # Construct a table of counts abby %>% count(themes) ``` c. Before proceeding, let's break down the plotting code above. Run each chunk to see how the two lines of code above build up the plot in "layers". Add comments (on the lines starting with `#`) to document what you notice. ```{r} # ??? ggplot(abby, aes(x = themes)) #sets up the "canvas" of the plot with axis labels ``` ```{r} # ??? ggplot(abby, aes(x = themes)) + geom_bar() # Adds the bars- Any Problem you notice? ``` ```{r} # ??? ggplot(abby, aes(x = themes)) + geom_bar() + theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) #Rotates the x axis labels ``` ```{r} # ??? ggplot(abby, aes(x = themes)) + geom_bar() + theme_classic() + theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) # Changes the visual theme of the plot with a white background and removes gridlines ``` ## Exercise 4: Exploring sentiment We'll look at the distribution of the `bing_pos` sentiment variable and associated summary statistics. \ \ a. The code below creates a **boxplot** of this variable. In the comment, make note of how this code is similar to the code for the barplot above. As in the previous exercise, *before* running the code chunk to create the plot, make note of what you expect the boxplot to look like. ```{r} # ??? ggplot(abby, aes(x = bing_pos)) + geom_boxplot() ``` b. **Challenge:** Using the code for the barplot and boxplot as a guide, try to make a **histogram** and a **density** plot of the overall average ratings. - What information is given by the tallest bar of the histogram? - How would you describe the shape of the distribution? ```{r} # Histogram # Density plot ``` c. We can compute summary statistics (numerical summaries) for a quantitative variable using the `summary()` function or with the `summarize()` function from the `dplyr` package. (`1st Qu.` and `3rd Qu.` stand for first and third quartile.) After inspecting these summaries, look back to your boxplot, histogram, and density plot. Which plots show which summaries most clearly? ```{r} # Summary statistics # Using summary() - convenient for computing many summaries in one command # Does not show the standard deviation abby %>% select(bing_pos) %>% summary() # Using summarize() from dplyr # Note that we use %>% to pipe the data into the summarize() function # We need to use na.rm = TRUE because there are missing values (NAs) abby %>% summarize(mean(bing_pos, na.rm = TRUE), median(bing_pos, na.rm = TRUE), sd(bing_pos, na.rm = TRUE)) ``` d. Write a good paragraph describing the information in the histogram (or density plot) by discussing shape, center, spread, and outliers. Incorporate the numerical summaries from part c. ## Pause: Math box Below is an example of a "math box" which summarizes the formulas for some of the numerical summaries above. You are *not* required to memorize, nor will you be assessed on, any formulas presented in this or any future math box. They serve 3 purposes: 1. To emphasize that there's "math" / a formal structure behind what we're doing. 2. To provide students that plan to *continue* studying Statistics a glimpse into the formal statistical theory they'll explore in later courses. 3. To make happy the students that are simply interested in math! \ \ ::: {.callout-note title = "MATH BOX: Univariate numerical summaries"} Let $(y_1, y_2, ..., y_n)$ be a sample of $n$ data points. mean: $$\overline{y} = \frac{y_1 + y_2 + \cdots + y_n}{n} = \frac{\sum_{i=1}^n y_i}{n}$$ variance: $$\text{var}(y) = \frac{(y_1 - \overline{y})^2 + (y_2 - \overline{y})^2 + \cdots + (y_n - \overline{y})^2}{n - 1} = \frac{\sum_{i=1}^n (y_i - \overline{y})^2}{n - 1}$$ standard deviation: $$\text{sd}(y) = \sqrt{\text{var}(y)}$$ ::: ## Exercise 5: Box plots vs. histograms vs. density plots We took 3 different approaches to plotting the quantitative average course variable above. They all have pros and cons. a. What is one pro about the boxplot in comparison to the histogram and density plot? b. What is one con about the boxplot in comparison to the histogram and density plots? c. In this example, which plot do you prefer and why? ## Exercise 6: Returning to our context, looking ahead In this activity, we explored data on Dear Abby question, with a focus on exploring a single variable at a time. - In big picture terms, what have we learned about Dear Abby questions? - What further curiosities do you have about the data? ## Exercise 7: Different ways to think about data visualization In working with and visualizing data, it's important to keep in mind what a data point *represents*. It can reflect the experience of a real person. It might reflect the sentiment in a piece of art. It might reflect history. We've taken one very narrow and technical approach to data visualization. Check out the following examples, and write some notes about anything you find interesting. - [Dear Data](http://www.dear-data.com/by-week) - [W.E.B. DuBois](https://hyperallergic.com/476334/how-w-e-b-du-bois-meticulously-visualized-20th-century-black-america/) - [Decolonizing Data Viz](https://stephanieevergreen.com/decolonizing-data-viz/) ## Exercise 8: Rendering your work Save this file, and then click the "Render" button in the menu bar for this pane (blue arrow pointing right). This will create an HTML file containing all of the directions, code, and responses from this activity. A preview of the HTML will appear in the browser. - Scroll through and inspect the document to see how your work was translated into this HTML format. Neat! - Close the browser tab. - Go to the "Background Jobs" pane in RStudio and click the Stop button to end the rendering process. - Navigate to your "Activities" subfolder within your "STAT155" folder and locate the HTML file. You can open it again in your browser to double check. ## Reflection Go to the top of this file and review the learning objectives for this lesson. Which objectives do you have a good handle on, are at least familiar with, or are struggling with? What feels challenging right now? What are some wins from the day? > **Response:** Put your response here. ## Advice: make an R code "cheat sheet"! You will continue to pick up new R code and ideas. You're highly encouraged to start tracking this in a cheat sheet (eg: in a Google doc). The cheat sheet will be a handy reference for you, and the act of making it will help deepen your understanding and retention. # Additional Practice If you have time and want additional practice, try out the following exercises. ## Exercise 9: Read in and get to know the weather data Daily weather data are available for 3 locations in Perth, Australia. - View the codebook [here](https://github.com/Mac-STAT/data/blob/main/weather_3_locations_codebook.md). - Complete the code below to read in the data. ```{r} # Replace the ??? with your own name for the weather data # Replace the ___ with the correct function ??? <- ___("https://mac-stat.github.io/data/weather_3_locations.csv") ``` ## Exercise 10: Exploring the data structure Check out the basic features of the weather data. ```{r} # Examine the first six cases # Find the dimensions of the data ``` What does a case represent in this data? ## Exercise 11: Exploring rainfall The `raintoday` variable contains information about rainfall. - Is this variable quantitative or categorical? - Create an appropriate visualization, and compute appropriate numerical summaries. - What do you learn about rainfall in Perth? ```{r} # Visualization # Numerical summaries ``` ## Exercise 12: Exploring temperature The `maxtemp` variable contains information on the daily high temperature. - Is this variable quantitative or categorical? - Create an appropriate visualization, and compute appropriate numerical summaries. - What do you learn about high temperatures in Perth? ```{r} # Visualization # Numerical summaries ``` ## Exercise 13: Customizing! (CHALLENGE) Though you will naturally absorb some RStudio code throughout the semester, being an effective statistical thinker and "programmer" does not require that we memorize *all* code. That would be impossible! In contrast, using the *foundation* you built today, do some digging online to learn how to customize your visualizations. a. For the histogram below, add a title and more meaningful axis labels. Specifically, title the plot "Distribution of max temperatures in Perth", change the x-axis label to "Maximum temperature" and y-axis label to "Number of days". HINT: Do a Google search for something like "add axis labels ggplot". ```{r} # Add a title and axis labels ggplot(weather, aes(x = maxtemp)) + geom_histogram() ``` b. Adjust the code below in order to color the bars green. NOTE: Color can be an effective tool, but here it is simply gratuitous. ```{r} # Make the bars green ggplot(weather, aes(x = raintoday)) + geom_bar() ``` c. Check out the `ggplot2` [cheat sheet](https://rstudio.github.io/cheatsheets/data-visualization.pdf). Try making some of the other kinds of univariate plots outlined there. d. What else would you like to change about your plot? Try it! ## Exercise 14: Optional challenge At the top of this activity, we searched for words related to some topics of interest (`parents`, `marriage`, `money`) and combined them into a single `theme` variable. It looked something like this: ```{r} abby_new <- abby %>% mutate( parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"), marriage = str_detect(question_only, "marriage|marry|married"), money = str_detect(question_only, "money|finance") ) %>% rowwise() %>% mutate( themes = c( if (parents) "parents", if (marriage) "marriage", if (money) "money" ) %>% paste(collapse = ", "), themes = ifelse(themes == "", "other", themes) ) %>% ungroup() ``` Check it out: ```{r} head(abby_new) ``` a. Understand the code! - Inside `mutate()` the line `parents = str_detect(question_only, "mother|mama|mom|father|papa|dad")` created a new variable called `parents`. This variable takes on `TRUE` or `FALSE`. Explain what `TRUE` and `FALSE` mean here. - The `themes` variable combines the information from the `parents`, `marriage`, and `money` variables. Check out the `themes` for the first 3 rows / data points. Convince yourself that you understand how it corresponds to the `parents`, `marriage`, and `money` variables. b. Beyond `parents`, `marriage`, and `money`, what are some other topics that might pop up in the Dear Abby letters (and that you're interested in exploring)? Modify the code below to explore those topics! Update the `themes` variable accordingly. ```{r} abby_new <- abby %>% mutate( parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"), marriage = str_detect(question_only, "marriage|marry|married"), money = str_detect(question_only, "money|finance") ) %>% rowwise() %>% mutate( themes = c( if (parents) "parents", if (marriage) "marriage", if (money) "money" ) %>% paste(collapse = ", "), themes = ifelse(themes == "", "other", themes) ) %>% ungroup() # Check out the raw data head(abby_new) # Check out the number of letters belonging to each theme abby_new %>% count(themes) ``` \ \ \ # Solutions ```{r eval = TRUE, echo = FALSE, message = FALSE, warning = FALSE} # Load package library(tidyverse) # Read in the Dear Abby data abby <- read_csv("https://mac-stat.github.io/data/dear_abby.csv") # Wrangle the data abby <- abby %>% filter(year == 2017) %>% mutate(month = month(month, label = TRUE)) %>% mutate( parents = str_detect(question_only, "mother|mama|mom|father|papa|dad"), marriage = str_detect(question_only, "marriage|marry|married"), money = str_detect(question_only, "money|finance") ) %>% rowwise() %>% mutate( themes = c( if (parents) "parents", if (marriage) "marriage", if (money) "money" ) %>% paste(collapse = ", "), themes = ifelse(themes == "", "other", themes) ) %>% ungroup() %>% select(year, month, day, question_only, bing_pos, themes) ``` ## Exercise 1: Importing and getting to know the data a. Note how clicking the `abby` data causes both a popup pane and the command `View(abby)` to appear in the Console. In fact, the `View()` function is the underlying command that opens a dataset pane. (`View()` should always be entered in the Console and NOT your Quarto document.) b. Each row / case corresponds to a single question. c. Columns = variables d. Try out each function below. Identify what each function tells you about the `abby` data and note this in the `???`: ```{r eval = TRUE} # First number = number of rows / cases # Second number = number of columns / variables dim(abby) # Number of rows (cases) nrow(abby) # Number of columns (variables) ncol(abby) # View first few rows of the dataset (6 rows, by default) head(abby) # Get all column (variable) names names(abby) ``` e. We can display the first 10 rows with `head(abby, n = 10)`. ## Exercise 2: Preparing to summarize and visualize the data a. The sentiment variables are `afinn_overall`, `afinn_pos`, `afinn_neg`, and `bing_pos`, and they are **quantitative**. The `afinn` variables don't have units but we can still get a sense of the scale by remembering that each word gets a score between -5 and 5. The `bing_pos` variable doesn't have units because it's a fraction, but we know that it ranges from 0 to 1. b. **categorical** c. Appropriate visualizations: - single quantitative variable: boxplot, histogram, density plot - single categorical variable: barplot ## Exercise 3: Exploring themes in the letters a. Expectations about the plot will vary ```{r eval = TRUE} ggplot(abby, aes(x = themes)) + geom_bar() + theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) ``` b. Counts in the table below match the barplot ```{r eval = TRUE} # Construct a table of counts abby %>% count(themes) ``` c. What do the plot layers do? ```{r eval = TRUE} # Just sets up the "canvas" of the plot with axis labels ggplot(abby, aes(x = themes)) ``` ```{r eval = TRUE} # Adds the bars ggplot(abby, aes(x = themes)) + geom_bar() ``` ```{r eval = TRUE} # Rotates the x axis labels ggplot(abby, aes(x = themes)) + geom_bar() + theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) ``` ```{r eval = TRUE} # Changes the visual theme of the plot with a white background and removes gridlines ggplot(abby, aes(x = themes)) + geom_bar() + theme_classic() + theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1)) ``` ## Exercise 4: Exploring sentiment a. ```{r eval = TRUE} ggplot(abby, aes(x = bing_pos)) + geom_boxplot() ``` b. We replace `geom_boxplot()` with `geom_histogram()` and `geom_density()`. ```{r eval = TRUE} # Histogram ggplot(abby, aes(x = bing_pos)) + geom_histogram() # Density plot ggplot(abby, aes(x = bing_pos)) + geom_density() ``` c. - Boxplot shows min, max, median, 1st and 3rd quartile easily. (It shows median, 1st and 3rd quartile directly as lines) - Histogram and density plot show min and max but the mean and median aren't shown directly--we have to roughly guess based on the peak of the distribution ```{r eval = TRUE} # Summary statistics abby %>% select(bing_pos) %>% summary() abby %>% summarize(mean(bing_pos, na.rm = TRUE), median(bing_pos, na.rm = TRUE), sd(bing_pos, na.rm = TRUE)) ``` d. The distribution of sentiment scores is roughly tri-modal, ranging from 0 to 1. There's a group of very negative letters (with roughly 0\% of words being positive), a group of very positive letters (with roughly 100\% of words being positive), and a group of mostly negative letters. The typical sentiment is around 0.34. ## Exercise 5: Box plots vs. histograms vs. density plots a. Boxplots very clearly show key summary statistics like median, 1st and 3rd quartile b. Boxplots can oversimplify by not showing the shape of the distribution. ## Exercise 6: Returning to our context, looking ahead - Answers will vary ## Exercise 9: Read in and get to know the weather data ```{r eval = TRUE} weather <- read_csv("https://raw.githubusercontent.com/Mac-STAT/data/main/weather_3_locations.csv") ``` ## Exercise 10: Exploring the data structure Check out the basic features of the weather data. ```{r eval = TRUE} # Examine the first six cases head(weather) # Find the dimensions of the data dim(weather) ``` A case represents a day of the year in a particular area (Hobart, Uluru, Wollongong as seen by the `location` variable). ## Exercise 11: Exploring rainfall The `raintoday` variable contains information about rainfall. - `raintoday` is categorical (No, Yes) - It is more common to have no rain. ```{r eval = TRUE} # Visualization ggplot(weather, aes(x = raintoday)) + geom_bar() # Numerical summaries weather %>% count(raintoday) ``` ## Exercise 12: Exploring temperature The `maxtemp` variable contains information on the daily high temperature. - `maxtemp` is quantitative - The *typical* max temperature is around 23 degrees Celsius (with an average of 23.62 and a median of 22 degrees). The max temperatures ranged from 8.6 to 45.4 degrees. Finally, on the typical day, the max temp falls about 7.8 degrees from the mean. There are multiple modes in the distribution of max temperature---this likely reflects the different cities in the dataset. ```{r eval = TRUE} # Visualization ggplot(weather, aes(x = maxtemp)) + geom_histogram() # Numerical summaries summary(weather$maxtemp) # There are missing values (NAs) in this variable, so we add # the na.rm = TRUE argument weather %>% summarize(sd(maxtemp, na.rm = TRUE)) ``` ## Exercise 13: Customizing! (CHALLENGE) a. ```{r eval = TRUE} ggplot(weather, aes(x = maxtemp)) + geom_histogram() + labs(x = "Maximum temperature", y = "Number of days", title = "Distribution of max temperatures in Perth") ``` b. ```{r eval = TRUE} # Make the bars green ggplot(weather, aes(x = raintoday)) + geom_bar(fill = "green") ```