Week 9 — Models & Prediction (10/17–21)

This week talks more about regression, simulation, and introduces the idea of minimizing a loss function.

🧐 Content Overview

Element	Length
🎥 More Regression	3m7s
🎥 Simulation	14m48s
📃 Random Numbers	1044 words
🎥 Variance and Sums of Squares	5m59s
🎥 Overfitting	10m31s
📃 Example of overfitting and underfitting in machine learning	1180 words
🎥 Bias-Variance	9m16s
📃 Understanding the Bias-Variance Tradeoff	3000 words
🎥 Optimizing Loss	15m23s

This week has 0h59m of video and 5224 words of assigned readings. This week’s videos are available in a Panopto folder.

🎥 Introduction

This video introduces the week.

Video (3m7s)

Slides

CS 533INTRO TO DATA SCIENCE Michael Ekstrand MODELING AND PREDICTION Learning Outcomes (Week) Simulate to understand the behavior of a system Understand overfitting and apply appropriate design to mitigate it Select and engineer features for a model Photo by Jamie Street on Unsplash Taking Our Bearings We’ve seen: Basic data operations Visualization Probability and inference Correlations and linear models Prediction and inference Where we’re going: Classification Mostly prediction going forward Evaluating model accuracy More data types Text Time Workflows This Week More on variance Simulation methods Prediction accuracy evaluation Features Wrapping Up We’re going to learn more about building and assessing models. This will set us up to move from regression to classification. Photo by vorster vanzyl on Unsplash

🎥 Simulation

This video talks more about simulation as a method for studying statistical techniques, which you are doing in the assignment. I also describe more of NumPy’s random number generation facilities.

Video (14m48s)

Slides

CS 533INTRO TO DATA SCIENCE Michael Ekstrand SIMULATION Learning Outcomes Simulate data to understand statistical behavior Understand the difference between simulation and the bootstrap Photo by Markus Spiske on Unsplash Simulation We can make up data (with a random number generator) We know the population parameters (we set them) We can take unlimited samples (it’s just an RNG) Result: we can study the behavior of sampling and statistical methods Setting Up the RNG Modern: np.random.Generator Create with np.random.default_rng() Can provide a seed (starting point) - integer Recommended! Same seed + software versions + platform ⇒ same results Legacy: np.random functions Python standard library also has random module Drawing from Distributions integers(k, size=n) — n integers in range [0,k) integers(j, k, size=n) — n integers in range [j,k) random(n) — n floats uniformly from [0,1) normals(μ, σ, n) — n normally-distributed floats exponential(λ, n) — n exponentially-distributed floats Random and uniform are not the same thing. More Randomness Operating on arrays: choice(a, n) — n draws from array a w/ replacement permutation(a) — random reordering of a More distributions: scipy.stats Go NUTS: PyMC3, STAN Example: Mean Simulating Means (Law of Large Numbers) Simulating Means — Check for Normality Exponential Simulation We have a proof of the Central Limit Theorem More complex methods are much harder to characterize. Simulation lets us explore behavior Also: useful for learning, gaining intuition Not the Bootstrap Bootstrap Unknown population parameters Resample actual observations Estimate sampling distribution Purpose: estimate population parameters Simulation Known population parameters Sampling from theoretical distributions, making up data Characterize sampling distributions & other properties Purpose: understand estimate / parameter relationships Wrapping Up Simulation lets us study the behavior of statistical methods by drawing many samples with known population parameters. This is commonly used in statistics research. You do it in A4. Photo by Aditya Chinchure on Unsplash

Tip

You should set random seeds for all work that will need randomness, including train/test splits for evaluating predictors.

📓 Random Numbers

Read the Random Numbers notebook.

🎥 Variance, R², and the Sum of Squares

This video provides more detail on explained variance and what the \(R^2\) means.

Video (5m59s)

Slides

Resources

• Penn State R² notes

• Penn State Sum of Squares Notes

• Proof of SSTO = SSR + SSE (on Page 2)

🎥 Overfitting

This video introduces the idea of overfitting: learning too much from the training data so we can’t predict the testing data.

Video (10m31s)

Slides

CS 533INTRO TO DATA SCIENCE Michael Ekstrand PREDICTION AND OVERFITTING Learning Outcomes Understand overfitting Design an evaluation that reduces the likelihood of overfitting Understand that we can overfit process not just an individual model Photo by Jennifer Burk on Unsplash Prediction Goals Predict future unseen data We can just look at the data we have to learn about it We want to forecast or predict the future One way of thinking about generalizability Overfitting Overfitting Train-Test Data Train Test Model Split Predict Test data is unseen Can model generalize to it? Fitting to Data The model structure: Choice of model (linear, tree, etc.) Selection & transformation of features Hyper-parameter values The model parameters: Coefficients learned from data Both can overfit. Train-test addresses parameters. Tuning Data Data Train Test Train Test Tune Wrapping Up Overfitting arises when our model learns too much, so it can’t generalize to new data. Splitting our data into training, tuning, and testing sets helps with this. Photo by Darren Nunis on Unsplash

📓 Overfitting Simulation

• Overfitting Simulation notebook

📃 Overfitting Example

Read Example of overfitting and underfitting in machine learning.

🎥 Replication, Bias, and Variance

Video (9m16s)

Slides

CS 533INTRO TO DATA SCIENCE Michael Ekstrand REPLICATION, BIAS, AND VARIANCE Learning Outcomes Reason about repeated training on different samples Introduce the bias-variance tradeoff Photo by Nick Reynolds on Unsplash Replication Remember bootstrapping and simulation? Repeated samples Compute expectation, variance, etc. of parameters What about prediction? Test Error Population Test Error (Repeated Training) Population Bias-Variance Tradeoff Variance Bias2 Noise Wrapping Up Expected test error is the error we expect from predicting new, unobserved data points. When training from samples, expected error can be decomposed into model bias, model variance, and noise. Photo by Mitchell Luo on Unsplash

📃 Bias-Variance Tradeoff

Read Understanding the Bias-Variance Tradeoff.

Resources

Further reading: Lecture 12: Bias-Variance Tradeoff.

🎥 Optimizing Loss

Video (15m23s)

Slides

CS 533INTRO TO DATA SCIENCE Michael Ekstrand LOSS AND OPTIMIZATION Learning Outcomes Understand a loss function Optimize model for a loss function See attached notebook for code! Photo by Nick Reynolds on Unsplash Least Squares Regression Solve Least Squares Minimization Many problems are minimization (or maximization) problems! scipy.optimize provides a minimize function Model and Loss def predict_mass_uv(params, data=penguin_std): b0, b1 = params return b0 + data['flipper'] * b1 def squared_loss_uv(params, data=penguin_std): preds = predict_mass_uv(params, data) err = data['mass'] - preds return np.sum(np.square(err)) Minimization init_guess = rng.standard_normal(2) result = spopt.minimize(squared_loss_uv, init_guess) # get model parametersresult.xarray([1.12829633e-09, 8.71201757e-01]) Statsmodel OLS model General Optimization Understand data (features and outcome variables) Define loss (or gain/utility) function Define predictive model Search for parameters that minimize loss function Train and Test We optimize the loss function on the train data Evaluate on the test data Sometimes loss function is eval; sometimes not Mean squared error: often used for both Augmented Loss We can add more things to the loss function Penalize model complexity Penalize “strong” beliefs Requires predictive utility to overcome them Wrapping Up Least squares generalizes into minimizing loss functions. This is the heart of machine learning, particularly supervised learning. Photo by Mitchell Luo on Unsplash

Links

• The scipy.optimize.minimize() function.

• The minimization regression notebook notebook.

• Our introduction to the idea of an objective function.

🚩 Week 9 Quiz

Take the Week 9 quiz in Canvas (will be up by end of Saturday).

Since this is the second of two very closely intertwined weeks, there are questions about 📅 Week 8 — Regression (10/10–14) in the quiz ads well.

✅ Practice

There are several ways you can practice the material so far:

• Practice more regressions with World Bank data

• Measure World Bank data predictive accuracy with train-test evaluation and mean squared error

📩 Assignment 4

Assignment 4 is due October 24, 2021.