--- title: "Knn-bias-variance" subtitle: "Notes and in-class exercises" format: html: embed-resources: true toc: true --- ```{r include = FALSE} knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, fig.height = 2.75, fig.width = 4.25, fig.env='figure', fig.pos = 'h', fig.align = 'center') ``` You can download the .qmd file for this activity [here](../activity_templates/L08-knn-bias-variance.qmd) and open in R-studio. The rendered version is posted in the [course website](https://mutasim221b.github.io/Mac-STAT-253-Sp-26/) (Activities tab). I often experiment with the class activities (and see it in live!) and make updates, but I always post the final version before class starts. To be sure you have the most up-to-date copy, please download it once you’ve settled in before class begins. # Learning Goals {.unnumbered} - Clearly describe / implement by hand the KNN algorithm for making a regression prediction - Explain how the number of neighbors relates to the bias-variance tradeoff - Explain the difference between parametric and nonparametric methods - Explain how the curse of dimensionality relates to the performance of KNN \ \ # Notes: K Nearest Neighbors (KNN) {-} ## Context {.unnumbered .smaller} ![](../images/MLdiagram.jpg){width=100%} - **world = supervised learning** We want to model some output variable $y$ using a set of potential predictors ($x_1, x_2, ..., x_p$). - **task = regression** $y$ is quantitative - **(nonparametric) algorithm = K Nearest Neighbors (KNN)** \ ## Goal {.unnumbered .smaller} Our usual *parametric* models (eg: linear regression) are too rigid to represent the relationship between $y$ and our predictors $x$. Thus we need more *flexible* **nonparametric models**. \ ## KNN Regression {.unnumbered .smaller} **Goal** Build a **flexible** regression model of a quantitative outcome $y$ by a set of predictors $x$, $$y = f(x) + \varepsilon$$ . . . **Idea** Predict $y$ using the data on "neighboring" observations. Since the neighbors have similar $x$ values, they likely have similar $y$ values. . . . **Algorithm** For tuning parameter K, take the following steps to estimate $f(x)$ at each set of possible predictor values $x$: - Identify the K nearest neighbors of $x$ with respect to Euclidean distance. - Observe the $y$ values of these neighbors. - Estimate $f(x)$ by the average $y$ value among the nearest neighbors. . . . **Output** KNN does *not* produce a nice formula for $\hat{f}(x)$, but rather a set of rules for how to calculate $\hat{f}(x)$. . . . **In pictures (from ISLR)** ![](../images/KNN_ISLR.png) \ \ \ ## Example 1: Review {.unnumbered .smaller} Let's review the KNN algorithm using a **shiny app**. Run the code below and ignore the syntax!! a. Click "Go!" *one time only* to collect a set of sample data. b. Check out the KNN with K = 1. - What does it mean to pick K = 1? - Where are the jumps made? - Can we write the estimated $f(x)$ (red line) as $\beta_0 + \beta_1 x + ....$? c. Now try the KNN with K = 25. - What does it mean to pick K = 25? - Is this more or less wiggly / flexible than when K = 1? d. Set K = 100 where 100 is the number of data points. Is this what you expected? ```{r} #| code-fold: true # Load packages and data library(shiny) library(tidyverse) library(tidymodels) library(kknn) library(ISLR) data(College) college_demo <- College %>% mutate(school = rownames(College)) %>% filter(Grad.Rate <= 100) head(college_demo) ``` ```{r} #| eval: false #| code-fold: true # Define a KNN plotting function plot_knn <- function(k, plot_data){ expend_seq <- sort(c(plot_data$Expend, seq(3000, 57000, length = 5000))) #knn_mod <- knn.reg(train = plot_data$Expend, test = data.frame(expend_seq), y = plot_data$Grad.Rate, k = k) knn_results <- nearest_neighbor() %>% set_mode("regression") %>% set_engine(engine = "kknn") %>% set_args(neighbors = k) %>% fit(Grad.Rate ~ Expend, data = plot_data) %>% augment(new_data = data.frame(Expend = expend_seq)) %>% rename(expend_seq = Expend, pred_2 = .pred) ggplot(plot_data, aes(x = Expend, y = Grad.Rate)) + geom_point() + geom_line(data = knn_results, aes(x = expend_seq, y = pred_2), color = "red") + labs(title = paste("K = ", k), y = "Graduation Rate", x = "Per student expenditure ($)") + lims(y = c(0,100)) } # BUILD THE SERVER # These are instructions for building the app - what plot to make, what quantities to calculate, etc server_KNN <- function(input, output) { new_data <- eventReactive(input$do, { sample_n(college_demo, size = 100) }) output$knnpic <- renderPlot({ plot_knn(k = input$kTune, plot_data = new_data()) }) } # BUILD THE USER INTERFACE (UI) # The UI controls the layout, appearance, and widgets (eg: slide bars). ui_KNN <- fluidPage( sidebarLayout( sidebarPanel( h4("Sample 100 schools:"), actionButton("do", "Go!"), h4("Tune the KNN algorithm:"), sliderInput("kTune", "K", min = 1, max = 100, value = 1) ), mainPanel( h4("KNN Plot:"), plotOutput("knnpic") ) ) ) # RUN THE SHINY APP! shinyApp(ui = ui_KNN, server = server_KNN) ``` \ \ ## KNN Details: Weighted Average {-} The `tidymodels` KNN algorithm predicts $y$ using **weighted** averages. [kknn package documentation](https://www.rdocumentation.org/packages/kknn/versions/1.3.1/topics/kknn) The idea is to give more weight or influence to closer neighbors, and less weight to "far away" neighbors. Optional math: Let ($y_1, y_2, ..., y_K$) be the $y$ outcomes of the K neighbors and ($w_1, w_2, ..., w_K$) denote the corresponding weights. These weights are defined by a "kernel function" which ensures that: (1) the $w_i$ add up to 1; and (2) the closer the neighbor $i$, the greater its $w_i$. Then the neighborhood prediction of $y$ is: $$\sum_{i=1}^K w_i y_i$$ The [kernel wikipedia page](https://en.wikipedia.org/wiki/Kernel_(statistics)) shows common kernel functions. The default "optimal" kernel used in `kknn` was defined in a [2012 Annals of Statistics paper](https://projecteuclid.org/journals/annals-of-statistics/volume-40/issue-5/Optimal-weighted-nearest-neighbour-classifiers/10.1214/12-AOS1049.full). \ \ ## Example 2: Bias-Variance Tradeoff {.unnumbered .smaller} *What would happen if we had gotten a different sample of data?!?* ::: incremental - **Bias: ** On average, across different datasets, how close are the estimates of $f(x)$ to the observed $y$ outcomes? - We have *high bias* if our estimates are far from the observed $y$ outcomes - We have *low bias* if our estimates are close to the observed $y$ outcomes - **Variance: **How variable are the estimates of $f(x)$ from dataset to dataset? Are the estimates stable or do they vary a lot? - We have *high variance* if our estimates change a lot from dataset to dataset - We have *low* variance if our estimates don't change much from dataset to dataset :::
a. To explore the properties of *overly flexible* models, set K = 1 and click "Go!" several times to change the sample data. How would you describe how KNN behaves from dataset to dataset: - low bias, low variance - low bias, high variance - moderate bias, low variance - high bias, low variance - high bias, high variance b. To explore the properties of *overly rigid* models, repeat part a for K = 100: - low bias, low variance - low bias, high variance - moderate bias, low variance - high bias, low variance - high bias, high variance c. To explore the properties of more *balanced* models, repeat part a for K = 25: - low bias, low variance - low bias, high variance - moderate bias, low variance - high bias, low variance - high bias, high variance \ \ ## Example 3: Bias-Variance Reflection {.unnumbered .smaller} In general... a. Why is "high bias" bad? > On average, our prediction errors are large or high b. Why is "high variability" bad? > The model is not stable / trustworthy, changes depending on the sample data c. What is meant by the **bias-variance tradeoff**? > Ideally, both bias and variance would be low. BUT when we improve one of the features, we hurt the other. \ \ ## Example 4: Bias-Variance Tradeoff for Past Algorithms {.unnumbered .smaller} a. The LASSO algorithm depends upon tuning parameter $\lambda$: - When $\lambda$ is too small, the model might keep too many predictors, hence be overfit. - When $\lambda$ is too big, the model might kick out too many predictors, hence be too simple. With this in mind: - For which values of $\lambda$ (small or large) will LASSO be the most biased? - For which values of $\lambda$ (small or large) will LASSO be the most variable? b. The bias-variance tradeoff also comes into play when comparing *across* algorithms, not just *within* algorithms. Consider LASSO vs least squares: - Which will tend to be more biased? - Which will tend to be more variable? - When will LASSO beat least squares in the bias-variance tradeoff game? \ \ # Exercises {-} ## Instructions {.unnumbered .smaller} **Context** Using the `College` dataset from the `ISLR` package, we'll explore the KNN model of college graduation rates (`Grad.Rate`) by: - instructional expenditures per student (`Expend`) - number of first year students (`Enroll`) - whether the college is `Private` ```{r} #| eval: true # Load packages library(tidymodels) library(tidyverse) library(ISLR) # Load data data(College) # Wrangle the data college_sub <- College %>% mutate(school = rownames(College)) %>% arrange(factor(school, levels = c("Macalester College", "Luther College", "University of Minnesota Twin Cities"))) %>% filter(Grad.Rate <= 100) %>% filter((Grad.Rate > 50 | Expend < 40000)) %>% select(Grad.Rate, Expend, Enroll, Private) ``` Check out a codebook from the console: ```{r} #| eval: false ?College ``` . . . **Goals** - Understand how "neighborhoods" are defined using multiple predictors (both quantitative and categorical) and how data *pre-processing* steps are critical to this definition. - Tune and build a KNN model in R. - Apply the KNN model. - It is easier to review code than to deepen your understanding of new concepts outside class. *Prioritize and focus on the concepts over the R code.* You will later come back and reflect on the code. . . . **Directions** - Stay engaged. [Studies show](https://www.sciencedirect.com/science/article/pii/S0360131512002254 ) that when you're playing cards, watching vids, continuously on your message app, it impacts both your learning and the learning of those around you. - Be kind to yourself. You will make mistakes! - Be kind to each other. Collaboration improves higher-level thinking, confidence, communication, community, & more. - actively contribute to discussion - actively include all other group members in discussion - create a space where others feel comfortable making mistakes & sharing their ideas - stay in sync - As you go, consider: **W.A.I.T.** (Why Am/Aren't I Talking?) \ \ ## Questions {.unnumbered .smaller} ### Part 1: Identifying neighborhoods {-} The KNN model for `Grad.Rate` will hinge upon the **neighborhoods** defined by the 3 `Expend`, `Enroll`, and `Private` predictors. And these neighborhoods hinge upon how we **pre-process** our predictors. We'll explore these ideas below using the results of the following chunk. Run this, but DON'T spend time examining the code! ```{r} #| eval: true recipe_fun <- function(recipe){ recipe <- recipe %>% prep() %>% bake(new_data = college_sub) %>% head(3) %>% select(-Grad.Rate) %>% as.data.frame() row.names(recipe) <- c("Mac","Luther", "UMN") return(recipe) } # Recipe 1: create dummies, but don't standardize recipe_1 <- recipe(Grad.Rate ~ Expend + Enroll + Private, data = college_sub) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) recipe_1_data <- recipe_fun(recipe_1) # Recipe 2: standardize, then create dummies recipe_2 <- recipe(Grad.Rate ~ Expend + Enroll + Private, data = college_sub) %>% step_nzv(all_predictors()) %>% step_normalize(all_numeric_predictors()) %>% step_dummy(all_nominal_predictors()) recipe_2_data <- recipe_fun(recipe_2) # Recipe 3: create dummies, then standardize recipe_3 <- recipe(Grad.Rate ~ Expend + Enroll + Private, data = college_sub) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) %>% step_normalize(all_numeric_predictors()) recipe_3_data <- recipe_fun(recipe_3) ``` \ 1. **Feature space** Check out the feature space of our 3 predictors and take note of which school is the closer neighbor of Mac: UMN or Luther. ```{r} #| fig-width: 5 #| fig-height: 3 #| eval: true ggplot(college_sub, aes(x = Expend, y = Enroll, color = Private)) + geom_point(alpha = 0.5) + geom_text(data = head(college_sub, 3), aes(x = Expend, y = Enroll, label = c("Mac","Luther", "UMN")), color = "black") ``` \ 2. **What happens when we don't standardize the predictors?** Of course, KNN relies upon mathematical metrics (Euclidean distance), not visuals, to define neighborhoods. And these neighborhoods depend upon how we pre-process our predictors. Consider the pre-processing `recipe_1` which uses `step_dummy()` but not `step_normalize()`: ```{r} #| eval: true recipe_1_data ``` a. Use this pre-processed data to calculate the Euclidean distance between Mac and Luther: ```{r eval = FALSE} sqrt((14213 - ___)^2 + (452 - ___)^2 + (1 - ___)^2) ``` ```{r} sqrt((14213 - 8949)^2 + (452 - 587)^2 + (1 - 1)^2) ``` b. Check your distance calculation, and calculate the distances between the other school pairs, using `dist()`. ```{r} dist(recipe_1_data) ``` c. By this metric, is Mac closer to Luther or UMN? So is this a reasonable metric? If not, *why* did this happen? \ 3. **What happens when we standardize then create dummy predictors?** The metric above was misleading because it treated enrollments (people) and expenditures ($) as if they were on the same scale. In contrast, `recipe_2` first uses `step_normalize()` and then `step_dummy()` to pre-process the predictors: ```{r} #| eval: true recipe_2_data ``` Calculate the distance between each pair of schools using these pre-processed data: ```{r} dist(recipe_2_data) ``` By this metric, is Mac closer to Luther or UMN? So is this a reasonable metric? \ 4. **What happens when we create dummy predictors then standardize?** Whereas `recipe_2` first uses `step_normalize()` and then `step_dummy()` to pre-process the predictors, `recipe_3` first uses `step_dummy()` and then `step_normalize()`: ```{r} #| eval: true recipe_3_data ``` a. How do the pre-processed data from `recipe_3` compare those to `recipe_2`? RECALL: The standardized dummy variables lose some contextual meaning. But, in general, negative values correspond to 0s (not that category), positive values correspond to 1s (in that category), and the further a value is from zero, the less common that category is. b. Calculate the distance between each pair of schools using these pre-processed data. By this metric, is Mac closer to Luther or UMN? ```{r} dist(recipe_3_data) ``` c. How do the distances resulting from `recipe_3` compare to those from `recipe_2`? ```{r} dist(recipe_2_data) ``` d. Unlike `recipe_2`, `recipe_3` considered the fact that private schools are relatively more common in this dataset, making the public UMN a bit more unique. Why might this be advantageous when defining neighborhoods? Thus why will we typically first use `step_dummy()` before `step_normalize()`? ```{r} college_sub %>% count(Private) ``` \ \ ### Part 2: Build the KNN {-} With a grip on neighborhoods, let's now build a KNN model for `Grad.Rate`. For the purposes of this activity (focusing on concepts over R code), simply run each chunk and note what object it's storing. You will later be asked to come back and comment on the code. **STEP 1: Specifying the KNN model** ```{r} knn_spec <- nearest_neighbor() %>% set_mode("regression") %>% set_engine(engine = "kknn") %>% set_args(neighbors = tune()) ``` **STEP 2: Variable recipe (with pre-processing)** Note that we use `step_dummy()` before `step_normalize()`. ```{r} variable_recipe <- recipe(Grad.Rate ~ ., data = college_sub) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) %>% step_normalize(all_numeric_predictors()) ``` **STEP 3: workflow specification (model + recipe)** ```{r} knn_workflow <- workflow() %>% add_model(knn_spec) %>% add_recipe(variable_recipe) ``` **STEP 4: estimate multiple KNN models** This code builds 50 KNN models of `Grad.Rate`, using 50 possible values of K ranging from 1 to 200 (roughly 25% of the sample size of 775). It then evaluates these models with respect to their 10-fold CV MAE. ```{r} set.seed(253) knn_models <- knn_workflow %>% tune_grid( grid = grid_regular(neighbors(range = c(1, 200)), levels = 50), resamples = vfold_cv(college_sub, v = 10), metrics = metric_set(mae) ) ``` \ \ ### Part 3: Finalize and apply the KNN {-} 5. **Compare the KNN models** Plot the CV MAE for each of the KNN models with different tuning parameters K. NOTE: This is the same function we used for the LASSO! ```{r} knn_models %>% autoplot() ``` a. Use this plot to describe the goldilocks problem in tuning the KNN: - When K is too small, CV MAE increases because the model is too ... - When K is too big, CV MAE increases because the model is too ... b. Why did we try a *range* of K values? c. In KNN modeling, we typically want to *minimize the prediction errors*. Given this goal, is the range of K values we tried *wide enough*? Or might there be a better value for K outside this range? d. In KNN modeling, why won't we typically worry about "parsimony"? \ 6. **Pick K** a. Identify which value of K *minimizes* the CV MAE. Make sure this matches up with what you observe in the plot. ```{r} best_k <- knn_models %>% select_best() best_k ``` b. Calculate and interpret the CV MAE for this model. ```{r} #| eval: false # Plug in a number or best_k$neighbors knn_models %>% collect_metrics() %>% filter(neighbors == ___) ``` ```{r} ``` \ 7. **Final KNN model** Build your "final" KNN model using the optimal value you found for K above. NOTE: We only looked at roughly every 4th possible value of K (K = 1, 5, 9, etc). If we wanted to be very thorough, we could re-run our algorithm using *each* value of K close to our optimal K. ```{r} final_knn <- knn_workflow %>% finalize_workflow(parameters = best_k) %>% fit(data = college_sub) ``` What does a `tidy()` summary of `final_knn` give you? Does this surprise you? ```{r eval = FALSE} # DO NOT REMOVE eval = FALSE final_knn %>% tidy() ``` \ 8. **Make some predictions** Check out Mac and Luther. NOTE: This is old data. Mac's current graduation rate is one of the highest (roughly 90%)! ```{r} #| eval: true # Check out Mac & Luther mac_luther <- college_sub %>% head(2) mac_luther ``` a. Use your KNN model to *predict* Mac and Luther's graduation rates. How close are these to the truth? ```{r} #| eval: false # Prediction ___ %>% ___(new_data = ___) ``` ```{r} ``` b. Do you have any idea why Mac's KNN prediction is higher than Luther's? If so, are you using context or something you learned from the KNN? \ 9. **KNN pros and cons** a. What assumptions did the KNN model make about the relationship of `Grad.Rate` with `Expend`, `Enroll`, and `Private`? b. What did the KNN model tell us about the relationship of `Grad.Rate` with `Expend`, `Enroll`, and `Private`? c. Reflecting upon a and b, name one *pro* of using a *nonparametric* algorithm like the KNN instead of a *parametric* algorithm like least squares or LASSO. d. Similarly, name one *con*. e. Consider another "con". Just as with parametric models, we could add more and more predictors to our KNN model. However, the KNN algorithm is known to suffer from the **curse of dimensionality**. Why? (A quick Google search might help.) \ 10. **Parametric or nonparametric** a. Suppose we wanted to model the relationship of room and board costs vs out-of-state tuition. Would you use the parametric least squares algorithm or the nonparametric KNN algorithm? ```{r} #| eval: true ggplot(College, aes(x = Outstate, y = Room.Board)) + geom_point() ``` b. In general, in what situations would you use least squares instead of KNN? Why? \ 11. **R code reflection** Revisit all code in Parts 2 and 3 of the exercises. Comment upon each chunk: - What is it doing? - How, if at all, does it differ from the least squares and LASSO code? \ 12. **Data drill** a. Calculate the mean `Enroll` for public vs private schools. ```{r} ``` b. Plot the relationship of `Grad.Rate` vs `Enroll`, `Private`, and `Expend`. ```{r} ``` c. Identify the private schools with first year enrollments exceeding 3000. ```{r} ``` d. Ask and answer another question of interest to you. ```{r} ``` \ \ # Notes: R Code {-} Suppose we want to build a model of response variable `y` using predictors `x1` and `x2` in our `sample_data`. _________________________________ ```{r eval = FALSE} # Load packages library(tidymodels) library(kknn) ``` \ **Build the model** ```{r eval = FALSE} # STEP 1: KNN model specification knn_spec <- nearest_neighbor() %>% set_mode("regression") %>% set_engine(engine = "kknn") %>% set_args(neighbors = tune()) ``` STEP 1 notes: - We use the `kknn`, not `lm`, engine to build the KNN model. - The `knn` engine requires us to specify an argument (`set_args`): - `neighbors = tune()` indicates that we don't (yet) know an appropriate value for the number of neighbors $K$. We need to tune it. - By default, neighbors are defined using Euclidean distance. We could but won't change this in the `set_args()`. ```{r eval = FALSE} # STEP 2: variable recipe # (You can add more pre-processing steps.) variable_recipe <- recipe(y ~ x1 + x2, data = sample_data) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) %>% step_normalize(all_numeric_predictors()) ``` ```{r eval = FALSE} # STEP 3: workflow specification (model + recipe) knn_workflow <- workflow() %>% add_recipe(variable_recipe) %>% add_model(knn_spec) ``` ```{r eval = FALSE} # STEP 4: Estimate multiple KNN models using a range of possible K values # Calculate the CV MAE & R^2 for each set.seed(___) knn_models <- knn_workflow %>% tune_grid( grid = grid_regular(neighbors(range = c(___, ___)), levels = ___), resamples = vfold_cv(sample_data, v = ___), metrics = metric_set(mae, rsq) ) ``` STEP 4 notes: - Since the CV process is random, we need to `set.seed(___)`. - We use `tune_grid()` instead of `fit()` since we have to build multiple KNN models, each using a different tuning parameter K. - `grid` specifies the values of tuning parameter K that we want to try. - the `range` specifies the lowest and highest numbers we want to try for K (e.g. `range = c(1, 10)`. The *lowest* this can be is 1 and the *highest* this can be is the size of the smallest CV training set. - `levels` is the *number* of K values to try in that range, thus how many KNN models to build. - `resamples` and `metrics` indicate that we want to calculate a CV MAE for each KNN model. \ \ **Tuning K** ```{r eval = FALSE} # Calculate CV MAE for each KNN model knn_models %>% collect_metrics() # Plot CV MAE (y-axis) for the KNN model from each K (x-axis) autoplot(knn_models) # Identify K which produced the lowest ("best") CV MAE best_K <- select_best(knn_models, metric = "mae") best_K # Get the CV MAE for KNN when using best_K knn_models %>% collect_metrics() %>% filter(neighbors == best_K$neighbors) ``` \ **Finalizing the "best" KNN model** ```{r eval = FALSE} # parameters = final K value (best_K or whatever other value you might want) final_knn_model <- knn_workflow %>% finalize_workflow(parameters = ___) %>% fit(data = sample_data) ``` \ **Use the KNN to make predictions** ```{r eval = FALSE} # Put in a data.frame object with x1 and x2 values (at minimum) final_knn_model %>% predict(new_data = ___) ``` # Wrapping Up {-} ## Today's Material {.unnumbered .smaller} - Finish all the unfinished activities! - Remember that there’s an R code reference section at the bottom of these notes! \ \ # Solutions {-} ## Small Group Discussion {.unnumbered .smaller} 1. **Review**
Solution a. done b. KNN with K = 1: - predict grad rate using the data on the 1 closest neighbor - where the neighorhood changes (in between observed points) - no c. KNN with K = 25: - predict grad rate by average grad rate among 25 closest neighbors - less wiggly / less flexible d. probably not. it's not a straight line. You might have expected a horizontal line at one value. There is a bit more going on to the algorithm (a weighted average).

2. **Bias-Variance Tradeoff**
Solution a. low bias, high variance b. high bias, low variance c. moderate bias, low variance

3. **Bias-Variance Reflection**
Solution a. On average, our prediction errors are large or high b. The model is not stable / trustworthy, changes depending on the sample data c. Ideally, both bias and variance would be low. BUT when we improve one of the features, we hurt the other.

4. **Bias-Variance Tradeoff for Past Algorithms**
Solution a. . - large. too simple / rigid - small. too overfit / flexible b. . - LASSO. it's simpler - least squares. it's more flexible - when the least squares is overfit


## Exercises {.unnumbered .smaller} ```{r} #| eval: true #| echo: false # Load packages library(tidymodels) library(tidyverse) library(ISLR) # Load data data(College) # Wrangle the data college_sub <- College %>% mutate(school = rownames(College)) %>% arrange(factor(school, levels = c("Macalester College", "Luther College", "University of Minnesota Twin Cities"))) %>% filter(Grad.Rate <= 100) %>% filter((Grad.Rate > 50 | Expend < 40000)) %>% select(Grad.Rate, Expend, Enroll, Private) recipe_fun <- function(recipe){ recipe <- recipe %>% prep() %>% bake(new_data = college_sub) %>% head(3) %>% select(-Grad.Rate) %>% as.data.frame() row.names(recipe) <- c("Mac","Luther", "UMN") return(recipe) } # Recipe 1: create dummies, but don't standardize recipe_1 <- recipe(Grad.Rate ~ Expend + Enroll + Private, data = college_sub) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) recipe_1_data <- recipe_fun(recipe_1) # Recipe 2: standardize, then create dummies recipe_2 <- recipe(Grad.Rate ~ Expend + Enroll + Private, data = college_sub) %>% step_nzv(all_predictors()) %>% step_normalize(all_numeric_predictors()) %>% step_dummy(all_nominal_predictors()) recipe_2_data <- recipe_fun(recipe_2) # Recipe 3: create dummies, then standardize recipe_3 <- recipe(Grad.Rate ~ Expend + Enroll + Private, data = college_sub) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) %>% step_normalize(all_numeric_predictors()) recipe_3_data <- recipe_fun(recipe_3) ``` ### Part 1: Identifying neighborhoods {.unnumbered .smaller} 1. **Feature space**
Solution Luther

2. **What happens when we don't standardize the predictors?**
Solution ```{r } #| echo: true #| eval: true # a sqrt((14213 - 8949)^2 + (452 - 587)^2 + (1 - 1)^2) # b dist(recipe_1_data) ``` c. UMN. The quantitative predictors are on different scales

3. **What happens when we standardize then create dummy predictors?**
Solution Luther. Yes. ```{r} #| eval: true recipe_2_data dist(recipe_2_data) ```

4. **What happens when we create dummy predictors then standardize?**
Solution ```{r} #| echo: true #| eval: true recipe_3_data ``` a. `Private_Yes` is now 0.6132441 or -1.6285680, not 1 or 0. b. Luther ```{r} #| echo: true #| eval: true dist(recipe_3_data) ``` c. The distance between Mac and Luther is the same, but the distance between Mac and UMN is bigger. ```{r} #| echo: true #| eval: true dist(recipe_2_data) ``` d. Since public schools are more "rare", the difference between Mac (private) and UMN (public) is conceptually bigger than if private and public schools were equally common.


### Part 2: Build the KNN {-}
Solution No "answers" here. Just run the provided code and make note of what it's doing. **STEP 1: Specifying the KNN model** ```{r} #| eval: true #| echo: true knn_spec <- nearest_neighbor() %>% set_mode("regression") %>% set_engine(engine = "kknn") %>% set_args(neighbors = tune()) ``` **STEP 2: Variable recipe (with pre-processing)** Note that we use `step_dummy()` before `step_normalize()`. ```{r} #| eval: true #| echo: true variable_recipe <- recipe(Grad.Rate ~ ., data = college_sub) %>% step_nzv(all_predictors()) %>% step_dummy(all_nominal_predictors()) %>% step_normalize(all_numeric_predictors()) ``` **STEP 3: workflow specification (model + recipe)** ```{r} #| eval: true #| echo: true knn_workflow <- workflow() %>% add_model(knn_spec) %>% add_recipe(variable_recipe) ``` **STEP 4: estimate multiple KNN models** This code builds 50 KNN models of `Grad.Rate`, using 50 possible values of K ranging from 1 to 200 (roughly 25% of the sample size of 775). It then evaluates these models with respect to their 10-fold CV MAE. ```{r} #| eval: true #| echo: true set.seed(253) knn_models <- knn_workflow %>% tune_grid( grid = grid_regular(neighbors(range = c(1, 200)), levels = 50), resamples = vfold_cv(college_sub, v = 10), metrics = metric_set(mae) ) ```
\ \ ### Part 3: Finalize and apply the KNN {-} 5. **Compare the KNN models**
Solution ```{r} #| eval: true #| echo: true knn_models %>% autoplot() ``` a. When K is too small, CV MAE increases because the model is too **flexible (overfit)**. When K is too big, CV MAE increases because the model is too **rigid (simple)**. b. Because we can't know in advance what a good value of K is. c. Yes, it's wide enough. We observe that CV MAE is minimized when K is around 25, it then increases from there. d. KNN always uses all of the predictors we give it. The concept of parsimony in the sense of simplicity/complexity of how many variables are included is not relevant here. Changing $K$ does not lead to a simpler/more complex model in the way changing $\lambda$ does for LASSO.

6. **Pick K**
Solution a. 33 neighbors ```{r} #| echo: true #| eval: true best_k <- knn_models %>% select_best() best_k ``` b. We expect the predictions of grad rate for *new schools (not in our sample)* to be off by 11.3 percentage points. ```{r} #| echo: true #| eval: true # Plug in a number knn_models %>% collect_metrics() %>% filter(neighbors == 33) # an equivalent but more reproducible way to do this: knn_models %>% collect_metrics() %>% filter(neighbors == best_k$neighbors) ```

7. **Final KNN model**
Solution ```{r} #| echo: true #| eval: true final_knn <- knn_workflow %>% finalize_workflow(parameters = best_k) %>% fit(data = college_sub) ``` Since this is a nonparametric method, we can't summarize the functional relationship between our outcome and predictors with parameters and thus `tidy()` doesn't have estimates to give us.

8. **Make some predictions**
Solution a. . ```{r} #| echo: true #| eval: true # Prediction mac_luther <- college_sub %>% head(2) final_knn %>% predict(new_data = mac_luther) ``` b. If you have any ideas, it's probably based on context because the KNN hasn't given us any info about *why* it returns higher or lower predictions of grad rate.

9. **KNN pros and cons**
Solution a. essentially none -- just that colleges with similar expenditures, enrollments, and school types (private vs public) would have similar graduation rates b. not much -- we can just use it for predictions c. KNN doesn't make assumptions about relationships (which is very flexible!) d. KNN doesn't provide much insight into the relationships between y and x e. When calculated by more and more predictors, our nearest neighbors might actually be far away (thus not very similar).

10. **Parametric or nonparametric**
Solution a. least squares. the relationship is linear, so doesn't require the flexibility of a nonparametric algorithm b. when the relationships of interest *can* be reasonably represented by least squares, we *should* use least squares. It provides much more insight into the relationships.

11. **R code reflection**
Solution Come to office hours to discuss.

12. **Data drill**
Solution ```{r} #| echo: true #| eval: true # a college_sub %>% group_by(Private) %>% summarize(mean(Enroll)) # b ggplot(college_sub, aes(y = Grad.Rate, x = Enroll, size = Expend, color = Private)) + geom_point(alpha = 0.5) # c college_sub %>% filter(Enroll > 3000, Private == "Yes") ``` d. Answers will vary. Come to office hours to discuss.


\ \