An Introduction to Statistical Learning: with Applications in R

1. 1Write down the form your model assumes before fitting it. Linear regression assumes a linear relationship between Y and X1 … Xp. That assumption is the source of bias: the passage says it is unlikely any real-life problem is truly that simple, so some bias in the estimate of f is certain.
2. 2Compare the assumed form with what you know about the true f. In ISLP's Figure 2.11 the true f is substantially non-linear, so a linear fit is structurally mismatched. Judge from the problem, not only from the fit statistics.
3. 3Fit at least two flexibility levels. The passage's figures run from the inflexible orange least squares line to the flexible green curve that follows observations closely. Flexibility ranges roughly from 2 to 20 on the Figure 2.12 axis.
4. 4Plot squared bias, variance and test MSE against flexibility, as in Figure 2.12. Expect squared bias (blue curve) to fall as flexibility rises. Mark the flexibility level with the smallest test MSE, the vertical dotted line in that figure.
5. 5Test whether more data would help. The passage states that when the true f is substantially non-linear, no matter how many training observations you are given, a linear regression cannot produce an accurate estimate. If extra data does not lower the error, the cause is bias, not sample size.
6. 6Increase flexibility only while bias drops faster than variance rises. The flexible green curve has high variance because changing one data point may change the estimate considerably. The inflexible line has low variance but higher bias.
7. 7Do not expect to reach zero error. Test MSE has a floor at Var(ε), the irreducible error shown as the dashed line in Figure 2.12. Reducing bias only attacks the reducible part of E(Y − Ŷ)².

The ISLP passages describe three simulated data sets (Figures 2.9–2.11). In Figure 2.11 the true f is substantially non-linear. You fit the orange least squares line, which is relatively inflexible and has low variance, because moving any single observation causes only a small shift in the line. But linear regression assumes a linear relationship between Y and the predictors, so the line cannot follow a curved f. The result is high bias, and the passage states that more training observations will not produce an accurate estimate. You then fit the flexible green curve, which follows the observations very closely. Its bias is lower, but its variance is high, because changing any one data point may change f-hat considerably. Figure 2.12 plots squared bias, variance and test MSE against flexibility for all three data sets. You read off the flexibility level where test MSE is smallest, marked by the vertical dotted line. That level is the target, not the most flexible model.

• If the true f is substantially non-linear, then do not rely on a linear model, however much data you collect. The bias is structural.
• If more training observations leave test error flat, then suspect bias and raise flexibility. If adding data helps, suspect variance.
• If flexibility is rising and squared bias is falling but test MSE is climbing, then variance now dominates. Step back to the flexibility with the smallest test MSE.
• If your goal is prediction only, then treat f-hat as a black box and accept whatever flexibility minimizes test MSE. If you need interpretability, accept some bias in exchange for a simpler form.
• If test MSE sits near Var(ε), then stop adding flexibility. The remaining error is irreducible.

• Judging a model by training fit alone. A flexible curve that hugs the points looks good in-sample while carrying high variance. The tell: training error is low and test error is much higher.
• Assuming more data fixes a linear model on a non-linear f. The tell: the learning curve plateaus above the noise floor.
• Maximizing flexibility to eliminate bias. The tell: squared bias is near zero but variance is large and test MSE rises, as in Figure 2.12 beyond the dotted line.
• Treating all error as bias. Var(ε) is irreducible and cannot be removed by any method. The tell: you keep adding model complexity while test MSE stays at the same level.
• Treating the linearity assumption as harmless. The tell: no check was ever made of the assumed form against the shape of the data, and residuals show a systematic pattern.

• ▸Model and assumed form: state the functional form the model assumes, such as linear in X1 … Xp.
• ▸Evidence on true f: note whether f is known or suspected to be non-linear, and why.
• ▸Flexibility levels tried: list each model or setting, from least to most flexible.
• ▸Squared bias estimate per level: record how bias changes as flexibility rises.
• ▸Variance estimate per level: record how sensitive f-hat is to changing single data points.
• ▸Test MSE per level: record held-out error for each level and mark the minimum.
• ▸More-data check: note whether added training observations lowered the error.
• ▸Estimated irreducible error Var(ε): record the floor your test MSE is approaching.
• ▸Decision: choose the flexibility level and give the reason in one line.

1. 1Write down the definition you are testing: variance is how much f-hat changes if you re-estimate it on a different training set. Ask the question of the method, not of one fitted curve.
2. 2Place the method on the flexibility scale. Linear regression is inflexible: it produces only linear functions such as a line or plane. Thin-plate splines are flexible: they produce a much wider range of shapes. The source's flexibility-interpretability figure also ranks subset selection and lasso as less flexible, then least squares, generalized additive models, trees, support vector machines, bagging/boosting and deep learning as increasingly flexible. Expect variance to rise as you move up this scale.
3. 3Run the thought experiment behind the definition: draw several different training sets, fit the same method to each, and compare the fitted curves. If small changes in the training data produce large changes in f-hat, the method has high variance. If the curves stay close together, variance is low.
4. 4Check the fit against the training points. A curve that follows the observations very closely, like the flexible green curve in the source's Figure 2.9, is a warning sign. The source's thin-plate spline at a low smoothness level fits the observed data perfectly and is far more variable than the true f. Treat a perfect training fit as evidence of overfitting, not of success.
5. 5Control variance through the tuning knob. For the thin-plate spline, the analyst must select a level of smoothness. A higher level gives a smoother, less variable fit. A lower level allows a rougher fit that tracks the noise. Regularization methods (ridge, lasso) lower variance in the same way. The source defers the selection procedure to Chapter 5, so use cross-validation to choose the setting.
6. 6Evaluate on observations not used in training. Variance is paid for in expected test MSE, which equals Var(f-hat(x0)) + [Bias(f-hat(x0))]^2 + Var(ε). Select the setting that achieves low variance and low bias at once, not the one that minimizes variance alone.
7. 7Keep the floor in mind. Variance and squared bias are both nonnegative, so expected test MSE can never fall below Var(ε), the irreducible error. Stop tuning once variance reduction no longer lowers test error.

The source's thin-plate spline example shows variance directly. The data are fit with a thin-plate spline, a non-parametric method, and the analyst must choose a smoothness level. At the first smoothness level (Figure 2.5), the fit is smooth and is a remarkably accurate estimate of the true f in Figure 2.3. At a lower smoothness level (Figure 2.6), the surface fits the observed data perfectly. But the source says that fit is far more variable than the true function f, which is overfitting. It will not give accurate estimates of the response on new observations outside the training set. The variance reading: a different training sample would pull the rough surface into a different shape, so f-hat changes a lot between training sets. Training error is zero, yet expected test MSE rises through the Var(f-hat) term in equation 2.7. The fix is to raise smoothness, which is regularization of the spline, and to pick the level using the methods of Chapter 5.

• If the method is more flexible (splines, trees, boosting, deep learning versus linear regression), then expect higher variance and plan to constrain it.
• If the fit matches the training data perfectly or follows the observations very closely, then suspect overfitting and high variance, and check test error.
• If small changes in the training data produce large changes in f-hat, then the method has high variance: raise the smoothness or regularization, or move to a less flexible method.
• If you need an interpretable model and variance is a concern, then favor a more restrictive method, since interpretability falls as flexibility rises.
• If test MSE is already near Var(ε), then stop reducing variance, because the irreducible error is a floor you cannot go below.

• Equating a perfect training fit with a good model. The tell: zero training error alongside poor accuracy on new observations, as with the low-smoothness spline.
• Chasing low variance alone. The tell: an overly rigid method such as a line fit to a curved f. Expected test MSE needs low variance and low bias simultaneously, so shrinking one term can raise the other.
• Judging variance from one fitted model. The tell: you inspect a single f-hat and never ask how it would change on a different training set.
• Choosing the smoothness or regularization level by eye or on training data. The tell: the setting is picked because the curve looks like it fits. The source assigns this choice to the methods of Chapter 5.
• Expecting tuning to push test error to zero. The tell: you keep adjusting after test MSE plateaus, ignoring that Var(ε), the irreducible error, bounds accuracy and is almost always unknown.

• ▸Method and flexibility rank: name the method and place it on the low-to-high flexibility scale (e.g., linear regression low, thin-plate spline high).
• ▸Tuning parameter: name the knob that controls variance (smoothness level, regularization strength) and its current setting.
• ▸Training-set resampling plan: describe how you will refit on different training sets (e.g., cross-validation folds, per Chapter 5).
• ▸Training error: record the fit on the data used for training.
• ▸Test or validation error: record the error on observations not used in training.
• ▸Spread of f-hat across training sets: note how much the fitted curve or predictions change between refits (small, moderate, large).
• ▸Overfitting verdict: is the training fit near-perfect while test error is poor? Yes or no, with evidence.
• ▸Adjustment decision: raise smoothness or regularization, reduce flexibility, or keep, and state why.
• ▸Irreducible-error check: is test MSE near the unknown Var(ε) floor? Note whether further tuning is worthwhile.

1. 1Compute training error on the same observations used to fit the model. For regression, this is the training residual sum of squares or its mean (training MSE), taken over the n observations {(x1,y1),…,(xn,yn)}. Label it 'training' every time you report it.
2. 2Never read training error as a measure of how well the model predicts new observations. The passages tie training error to the fitting procedure: least squares picks coefficients to make Y ≈ β0 + β1X1 + … + βpXp on the training data, so the fit is optimized for exactly this number.
3. 3Expect training error to fall whenever you add flexibility. Adding a variable always decreases the residual sum of squares on the training data, 'though not necessarily the testing data.' A drop in training error from a bigger model is therefore not evidence that the model is better.
4. 4Check R², which is computed on the training data. Because it is a function of training RSS, it must increase when you add variables, even weakly associated ones. A tiny rise is a weak signal for the added term. Pair it with the p-value for that variable.
5. 5Hold out observations that played no part in fitting, and compute error on them (test error). Compare the two numbers. Training error below test error is expected; the size of the gap tells you how far the fit has drifted toward the training sample.
6. 6When comparing models of different flexibility, rank them on test error, not training error. Training error will always favor the most flexible candidate, so it cannot select among them.
7. 7Keep irreducible error in view. Even a perfect estimate of f leaves error from ε, which X cannot predict. A training error far below the plausible noise level means the model is fitting ε, not f.

The Advertising data (200 markets) predicts sales from TV, radio, and newspaper budgets. The model using all three media has R² = 0.8972. The model using only TV and radio has R² = 0.89719. Because R² is computed on the training data, adding newspaper had to raise it, since extra variables always lower training RSS. The rise is tiny, about 0.00001. That tiny gain, together with the non-significant p-value for newspaper in Table 3.4, is the evidence for dropping newspaper. The mechanical increase in training fit is not. A reader who took the larger R² at face value would keep a variable that the training numbers reward by construction. The check is whether the variable helps on data not used in fitting, and here the passages indicate it does not.

• If a more flexible model has lower training error, then do not conclude it is better; test error is the arbiter.
• If adding a predictor raises R² only slightly and its p-value is not significant, then treat the gain as the automatic training-data effect and consider dropping the predictor.
• If training error is much lower than held-out error, then the model is fitting noise (the ε term) and you should reduce flexibility.
• If you must choose among models of different flexibility, then compare on observations not used for fitting, never on training RSS or R².
• If someone reports only training error or R², then ask for error on held-out data before accepting the claim.

• Picking the most flexible model because it has the lowest training error. The tell: the 'best' model wins on every training metric by construction as terms are added.
• Treating a rising R² as proof a new variable matters. The tell: R² increases even for a variable with a non-significant p-value, as with newspaper (0.8972 vs 0.89719).
• Reporting training error as expected prediction accuracy. The tell: no held-out data appears anywhere in the evaluation, and the number is presented as how the model will perform on new cases.
• Ignoring irreducible error. The tell: training error is pushed toward zero, which can only happen by fitting the ε term that no function of X can predict.
• Evaluating on data used in any part of fitting or selection. The tell: the 'test' set shared observations with the training data, so the estimate still understates true test error.

• ▸Model description: name the functional form (e.g., linear in TV and radio) and the number of parameters.
• ▸Fitting method: state how parameters were estimated (e.g., ordinary least squares).
• ▸Training set size n: count the observations used to fit.
• ▸Training error: record the training MSE or RSS computed on the fitting data.
• ▸Training R²: record it, noting it is a training-data statistic.
• ▸Held-out set: describe observations excluded from fitting and confirm no overlap.
• ▸Held-out error: record the error on those observations with the same metric as training.
• ▸Gap (held-out minus training): compute it and note whether it is large.
• ▸Decision: keep, simplify, or add flexibility, justified by held-out error, not training error.

1. 1Compute the training MSE and set it aside as a non-diagnostic number. Training MSE falls monotonically as flexibility rises, so a low value alone says nothing about overfitting.
2. 2Obtain an estimate of test MSE. Test data are usually unavailable, so estimate it from the training data with cross-validation (Chapter 5). Training MSE is easy to compute, test MSE is not, and the diagnosis depends on the harder number.
3. 3Fit the same method at several flexibility levels (for a thin-plate spline, several smoothness settings) and plot training MSE and estimated test MSE against flexibility on one chart.
4. 4Look for the signature: training MSE decreasing monotonically while test MSE traces a U-shape. The minimum of the U is the target flexibility. Moving right of it is overfitting.
5. 5Apply the definition strictly: overfitting is the case where a less flexible model would have yielded a smaller test MSE. A test MSE above training MSE alone does not qualify, because that gap is expected almost always.
6. 6Choose the flexibility at the minimum estimated test MSE. Where the minimum falls varies considerably by data set: near-linear truth gives a shallow dip then a rise, strongly non-linear truth gives a rapid drop in both curves before a slow rise.
7. 7If the goal is prediction only, do not default to the most flexible model. A less flexible method often predicts more accurately.
8. 8Compare against the irreducible error Var(ε), the lowest achievable test MSE. A test MSE near that floor means little room remains; a test MSE far above it with a tiny training MSE points to overfitting.

The book fits a thin-plate spline to the Income data (income as a function of years of education and seniority). With a high smoothness level the yellow surface in Figure 2.5 gives a remarkably accurate estimate of the true f. Lowering the smoothness level gives the rough fit in Figure 2.6, which makes zero errors on the training data: it fits the observed points perfectly. Yet that surface is far more variable than the true f, so it will not yield accurate estimates for new observations outside the training set. A training-only check would rank the rough fit best, since its training error is zero. A test-error check ranks it worse than the smooth fit. The rough spline is following noise, meaning patterns produced by random chance rather than by f. The fix is to choose the smoothness level using a test-error estimate such as cross-validation, not training fit.

• If training MSE is small and estimated test MSE is large, then treat the model as overfit and reduce flexibility or increase smoothness.
• If a less flexible model would give a smaller test MSE than the current one, then you are overfitting. If the test MSE gap exists but no simpler model beats the current one, then it is not overfitting.
• If the true relationship looks close to linear, then expect the test MSE to bottom out early and prefer a simple fit such as least squares over a highly flexible curve.
• If the goal is inference or interpretability, then prefer simpler, less flexible methods.
• If no held-out test set exists, then use cross-validation on the training data to estimate test MSE before choosing flexibility.

• Picking the model by training MSE. Tell: the most flexible model always wins, and a perfect fit (zero training error, as in Figure 2.6) looks like success.
• Equating any train/test gap with overfitting. Tell: the test MSE exceeds the training MSE, but a simpler model would not have scored lower on test data.
• Assuming prediction-only work calls for the most flexible model. Tell: a stock-price style task is built with maximum flexibility and no interpretability constraint, and accuracy on new data is worse than a simpler method.
• Assuming one flexibility level suits every data set. Tell: you reuse a setting tuned elsewhere, but the test-MSE minimum sits at a different point for near-linear versus highly non-linear truth.
• Treating patterns found in training data as real. Tell: apparent structure disappears on new data, since it came from random chance. In the S&P data, past returns show little association with future ones, so a model that appears to find strong structure there is suspect.

• ▸Model and flexibility setting: name the method and the flexibility or smoothness parameter being varied.
• ▸Training MSE: record the training MSE at each flexibility level tried.
• ▸Estimated test MSE: record the cross-validation estimate at each level, and note the method used.
• ▸Flexibility at minimum test MSE: identify the level where estimated test MSE is lowest.
• ▸Current model vs. simpler alternative: state whether a less flexible model gives a lower test MSE.
• ▸Irreducible error reference: note Var(ε) if known, or the lowest test MSE observed, as the floor.
• ▸Overfitting verdict: yes only if a less flexible model beats the current one on test MSE; otherwise no.
• ▸Goal (prediction or inference): record the goal, since inference favors less flexible methods.
• ▸Action taken: reduce flexibility, raise smoothness, or keep the current model.

1. 1Hold out data first. Fit every candidate on a training set and score it on a separate test set. Never rank methods by the data they were fit to. The source's own exercise (boosting, bagging, random forests, BART) says to fit on a training set and evaluate on a test set.
2. 2Fix one error metric before fitting anything, and express it in the units of Y. For regression, report the prediction error and also its size relative to the mean of Y. The Advertising example does this: RSE 3.26 against mean sales of about 14,000 gives 23%. For classification, use the test error rate.
3. 3Line up candidates from least to most flexible. Examples from the source: linear regression vs. boosting or random forests, LDA vs. QDA, and the maximal margin classifier vs. a soft-margin support vector classifier vs. an SVM with a non-linear boundary.
4. 4Compute training error and test error for each candidate and plot both against flexibility. Training error keeps falling as flexibility rises. Test error is the quantity that turns upward, giving the U-shape. Pick the flexibility at the bottom of the test curve, not the lowest training error.
5. 5Estimate the irreducible floor. Because Y = f(X) + ε and ε cannot be predicted from X, even a perfect estimate of f leaves error. Once test error stops improving between methods, assume you are near that floor and stop adding flexibility.
6. 6Run a stability check on the winner. Add, drop, or change a single observation and refit. In Figure 9.5, one added point dramatically changed the maximal margin hyperplane and shrank its margin, which signals overfitting. If one point moves the fit this much, treat the training score as unreliable.
7. 7Check dimensionality before trusting a local method. With KNN and large p (the exercise uses p = 100), very few training observations lie near any test point. Estimate what fraction of observations a local neighbourhood actually uses.
8. 8Confirm the goal. If you need prediction, treat f̂ as a black box and select on test error alone. If you need inference (which media drive sales), interpretability can outweigh a small test-error gain.

Take the Advertising data: sales across 200 markets, regressed on TV budget. The linear regression output (Table 3.2) gives RSE = 3.26, so actual sales deviate from the regression line by about 3,260 units on average. Mean sales are about 14,000 units, so the relative error is 3,260/14,000 = 23%. The source adds a key point. Even if the true β0 and β1 were known exactly, predictions would still be off by about 3,260 units, because that error comes from ε, not from the estimate. That is the irreducible floor, so you should not expect any more flexible model to push test error below it. A more flexible model can only attack the reducible part. You would therefore fit richer candidates on a training split and score them on held-out markets. A candidate is worth adopting only if its test error drops meaningfully below the linear baseline; whether 23% is acceptable is a judgment about the problem context, not a statistic.

• If a more flexible model lowers training error but not test error, keep the simpler model: the extra flexibility is fitting ε, not f.
• If one added or removed observation substantially changes the fitted model (as with the maximal margin hyperplane in Figure 9.5), choose a less flexible or softer-margin version, such as the support vector classifier.
• If the true boundary is plausibly linear, do not assume a more flexible method such as QDA wins on test error just because it can represent a linear boundary. Verify on held-out data.
• If p is large and you are considering KNN, first check how few training points fall near a test point; if the neighbourhood is nearly empty, expect poor test performance.
• If the stated goal is inference rather than prediction, prefer the more interpretable model unless a flexible one clearly reduces test error.

• Ranking methods by training fit. The tell: the most flexible model always wins, and its advantage disappears or reverses on new data.
• Reading RSE or training fit as evidence of predictive accuracy. The tell: a small RSE is reported with no held-out evaluation.
• Chasing zero error. The tell: you keep adding flexibility after test error has flattened, ignoring that ε sets an irreducible floor.
• Judging an error number without context. The tell: '3,260 units' is declared good or bad without comparing it to mean sales (about 14,000, so 23%) or to the decision it supports.
• Trusting a fit that perfectly separates the training data. The tell: a tiny margin and a hyperplane that jumps when one observation is added.

• ▸Prediction goal (prediction or inference): state which, because it determines whether interpretability counts.
• ▸Error metric and units: name the metric (e.g., RSE in units of Y, or classification error rate).
• ▸Train/test split: describe how data were divided and confirm no model saw the test set during fitting.
• ▸Candidate models ordered by flexibility: list them from least to most flexible (e.g., LDA, QDA).
• ▸Training error per candidate: record the value for each model.
• ▸Test error per candidate: record the held-out value for each model; the minimum is the selection criterion.
• ▸Relative error vs. mean of Y: divide test error by the mean response (e.g., 3,260/14,000 = 23%) to judge acceptability.
• ▸Single-observation sensitivity: note whether adding or removing one point materially changes the fit.
• ▸Dimensionality check (p and neighbourhood size): for local methods, note how many training points fall near a typical test point.
• ▸Chosen model and justification: name the model at the bottom of the test curve and say why.

1. 1Write down the goal first: inference (understand how each predictor relates to the response) or prediction (accurate estimates only). The passages tie the need for interpretability to inference. For pure prediction, such as the direct-marketing mailing, the company 'simply wants to accurately predict the response.'
2. 2Place each candidate method on the flexibility axis using the Figure 2.7 ordering. From most interpretable and least flexible to least interpretable and most flexible: subset selection and lasso, least squares, generalized additive models, trees, bagging and boosting, support vector machines, deep learning.
3. 3Start with the most restrictive model that could plausibly work. Least squares linear regression is 'relatively inflexible but quite interpretable,' and you can read off how Y relates to X1, X2, ..., Xp directly.
4. 4If a linear model has many predictors, try the lasso. It uses the same linear model but sets a number of coefficients to exactly zero. The response is then related only to the predictors with nonzero estimates, which makes the final model easier to read.
5. 5Move up to a GAM only if a linear fit misses a non-linear relationship. Accept the cost: each predictor's effect is now a curve rather than a single coefficient, so the model is 'somewhat less interpretable.'
6. 6Treat splines, bagging, boosting, SVMs with non-linear kernels, and neural networks as the last resort for inference. They can yield estimates of f so complicated that it is hard to see how any individual predictor is associated with the response.
7. 7Judge each added bit of flexibility against held-out accuracy, not training fit. Figure 2.6's rough spline fits the training data perfectly yet is far more variable than the true f and will not predict new observations. Chapter 5 covers choosing the right amount of smoothness.
8. 8Record the interpretability you gave up and what you got in return, so the choice of a flexible model is explicit rather than accidental.

The passages contrast two settings. A company running a direct-marketing campaign wants to identify people likely to respond to a mailing from demographic variables. It has no interest in 'a deep understanding of the relationships between each individual predictor and the response,' so interpretability is low priority and a flexible method is acceptable if it predicts better. The Advertising data poses inference questions instead: which media are associated with sales, which give the biggest boost, and how large an increase in sales goes with a given increase in TV advertising. These questions require reading individual predictor effects, so you choose least squares or the lasso. In the Advertising linear fit, the RSE is 3.26, so predictions are off by about 3,260 units, roughly 23% of the mean sales of 14,000. That is the accuracy price you weigh against the clarity of a linear model. If the linear fit were inadequate, a GAM would add curves per predictor, and each curve is harder to summarize than one coefficient. Boosting or a neural network would fit more flexibly still but would obscure how TV alone relates to sales.

• If the goal is inference, then choose the most restrictive adequate model (least squares or lasso) and read coefficients directly.
• If the goal is pure prediction, then interpretability is secondary and flexible methods are acceptable, subject to held-out accuracy.
• If a linear model has many predictors and you want a smaller story, then use the lasso: zeroed coefficients leave only a subset of predictors related to the response.
• If a linear fit misses non-linear structure but you still need per-predictor explanations, then use a GAM and interpret each predictor's curve rather than jumping to boosting or deep learning.
• If a flexible fit matches the training data almost perfectly, then suspect overfitting (as in Figure 2.6) and check against new data before trusting it.

• Choosing a flexible method by default for an inference question. The tell: you cannot say how a single predictor, like TV advertising, is associated with the response.
• Assuming more flexible is always better. The tell: a rougher fit that reproduces training data perfectly but is far more variable than the true f.
• Treating the lasso as more flexible than least squares. The tell: you expect it to fit more shapes, when it is more restrictive and sets coefficients to exactly zero.
• Assuming a GAM is as readable as a linear model. The tell: you cite a single slope for a predictor whose relationship is now a curve.
• Using Figure 2.7's ordering as exact. The tell: you treat it as a precise ranking, though the passage calls it a representation of a general tendency.

• ▸Goal (inference / prediction / both): state which questions the model must answer.
• ▸Key questions about predictors: list the specific 'which / how much' questions, such as effect of TV on sales.
• ▸Baseline model: least squares fit; note coefficients and whether they answer the questions.
• ▸Restrictive variant: lasso result; list predictors with nonzero coefficients.
• ▸Non-linearity evidence: note where a linear fit fails, and whether a GAM curve is needed.
• ▸Flexible alternative considered: trees, boosting, SVM, or deep learning, and why.
• ▸Held-out accuracy comparison: record error on new data for each model, not training fit.
• ▸Interpretability given up: describe in one line what you can no longer explain.
• ▸Final choice and rationale: model chosen, tied back to the goal.

1. 1Fix the inferential question before any test runs. The source frames inference as asking 'which predictors are associated with the response?' and notes that often only a small fraction of the available predictors are substantially associated with Y. Write down every hypothesis that question will generate, because error-rate control is defined over that list.
2. 2Define the family explicitly: list each null hypothesis you will test and count them. The chapter outline treats 'The Challenge of Multiple Testing' (13.2) as its own problem. Many tests at once make false positives accumulate, so the count m is the first input to everything below.
3. 3Restate the single-test error logic as in 13.1.2 (Type I and Type II errors). A Type I error is a false positive, which is the error this construct measures. A Type II error is a missed true effect, and it is the cost you pay when you tighten control.
4. 4Choose the error rate to control. FWER (13.3) controls the chance of any false positive across the family, so use it when one false claim is costly. FDR (13.4) controls the expected share of false positives among your discoveries, so use it when you screen many candidates and tolerate some false leads.
5. 5Apply a procedure matched to that rate: an FWER-control approach from 13.3.2, or the Benjamini–Hochberg procedure from 13.4.2 for FDR. The supplied passages give only section titles, not formulas, so take the exact adjustment from those sections or the lab (13.6.2 for FWER, 13.6.3 for FDR) rather than improvising one.
6. 6Report the power cost. Section 13.3.3 is titled 'Trade-Off Between the FWER and Power'. State how many hypotheses you can no longer reject under stricter control, so the reader sees what the guarantee cost.
7. 7If the usual null distribution is doubtful, consider the re-sampling versions of the p-value (13.5.1) and the FDR (13.5.2). Check 13.5.3 ('When Are Re-Sampling Approaches Useful?') before choosing them over the standard route.
8. 8Record the final claim in a fixed form: the family size, the error rate controlled, the procedure used, the level, and the number of rejections. Anyone can then audit the claim.

Take the Advertising data set from the passages: sales in 200 markets, with TV, radio and newspaper budgets as predictors. The inference question is which of the three media are associated with sales. Step 1: the family is three null hypotheses, one per medium, so m = 3. Step 2: with three tests, the chance that at least one false positive appears is already larger than for a single test. That is the multiple-testing challenge of 13.2. Step 3: because a client will act on the answer by changing budgets, a false claim is costly, so FWER control (13.3) is the natural choice. Step 4: apply an FWER procedure from 13.3.2 to the three p-values and note which media remain significant. Step 5: state the power cost, meaning any medium that was significant alone but is no longer under the adjusted threshold. The passages supply no p-values for this data set, so the numbers must come from your own fit. With only three tests the FWER and FDR choices will often agree, and the difference grows as the family does.

• If a single false positive would trigger a costly action, as with the client changing advertising budgets, then control FWER.
• If you test a large number of candidates and only want a mostly-true list of discoveries, then control FDR with the Benjamini–Hochberg procedure instead of FWER.
• If FWER control leaves you with almost no rejections, then check whether the stricter guarantee is costing too much power (13.3.3) before switching to FDR, and say openly that you switched.
• If the null distribution is unclear or the standard p-values are doubtful, then consider the re-sampling p-value and FDR (13.5) after reading when they are useful (13.5.3).
• If you cannot state the family size m, then do not report adjusted results until you can.

• Counting only the tests you report: you ran many comparisons and show the few significant ones, so the stated family is too small. The tell is that the reported number of tests is far below the number of models or predictors examined.
• Treating raw per-test p-values as controlled: you apply a 0.05 cutoff to each test and claim overall error control. The tell is that no adjustment procedure is named.
• Mixing up FWER and FDR: you apply an FDR procedure but describe the result as 'the chance of any false positive is below α'. The tell is wording that promises a guarantee the chosen procedure does not give.
• Ignoring the power side: you tighten control and read the non-rejections as proof of no effect. The tell is the phrase 'not significant, so no association' with no mention of Type II error.
• Choosing the error rate after seeing the results: you switch between FWER and FDR until more hypotheses are rejected. The tell is that the criterion was not recorded before the tests ran.

• ▸Inferential question: write the one question being answered, for example 'which predictors are associated with the response'.
• ▸Family of hypotheses: list every null hypothesis that will be tested.
• ▸Family size m: count the list above, including exploratory tests.
• ▸Cost of a false positive: state who acts on a false claim and what it costs them.
• ▸Error rate chosen (FWER or FDR): name the rate and give the reason from the cost above.
• ▸Procedure and level: name the procedure, such as an FWER approach or Benjamini–Hochberg, and the target level.
• ▸Re-sampling needed (yes/no): record whether the null is doubtful, and why.
• ▸Rejections before and after adjustment: give both counts and name the hypotheses lost, to show the power cost.
• ▸Final claim statement: write one sentence giving m, the rate controlled, the procedure, and the discoveries.

1. 1Name the flexibility knob for your method before you fit anything: degrees of freedom for smoothing splines, polynomial degree or number of knots for regression, tree depth, K in KNN, network size. The passages treat 'degrees of freedom' as the quantity that summarizes the flexibility of a curve.
2. 2Anchor the low-flexibility end with linear regression, which sits at the most restrictive end with two degrees of freedom. Fit it first as the baseline. A linear fit can only produce linear functions, so its failure on a non-linear f is the signal to add flexibility.
3. 3Fit at least three levels of flexibility: one restrictive, one intermediate, one very flexible. Figure 2.9 does this with a linear fit (orange) and two smoothing splines (blue and green) of different smoothness.
4. 4Compute training MSE at each level. Expect it to fall monotonically as flexibility rises. Treat this number as uninformative for selection, because there is no guarantee that the lowest training MSE gives the lowest test MSE.
5. 5Compute test MSE at each level on observations not used for training, using Ave(y0 − f̂(x0))². If you have a test set, use it directly. Plot test MSE against flexibility and look for the U-shape: it declines, levels off, then rises.
6. 6Choose the flexibility at the bottom of the U, not the point where the training curve is lowest. When training MSE is small and test MSE is large, you are overfitting. Visually, the too-flexible fit is wiggly and fits the true f poorly even though it matches the observed data very well.
7. 7If inference is the goal, weigh interpretability next. Flexibility and interpretability trade off: as flexibility rises, interpretability falls (Figure 2.7). A slightly worse test MSE from a restrictive model can be worth paying if you must explain how each predictor relates to Y.
8. 8Compare your best test MSE to the irreducible error Var(ε) when you can estimate it. That is the lowest achievable test MSE for any method, so a fit close to it means more flexibility will not help.

Figure 2.9 simulates data from a known non-linear f and fits three curves of increasing flexibility. The orange linear regression has 2 degrees of freedom and is too inflexible to follow f. The blue smoothing spline has intermediate smoothness. The green smoothing spline is the most flexible and matches the observed points very well, but it is too wiggly and fits the true f poorly. Training MSE (grey curve) falls monotonically across all three, so green has the lowest training MSE. Test MSE (red curve), computed on a very large test set, declines at first, levels off, then rises. Orange and green both have high test MSE, and blue minimizes it. The dashed line at Var(ε) marks the lowest achievable test MSE, and blue sits close to it, so the spline is near-optimal. Selecting by training MSE would have chosen green, the worst kind of mistake here.

• If the linear fit leaves clear curvature (as with Wage vs. age, where wages rise until about 60 and then decline), then add flexibility such as splines rather than staying linear.
• If training MSE keeps falling but test MSE has turned upward, then you have passed the bottom of the U: reduce flexibility (fewer degrees of freedom, smoother fit).
• If the goal is inference about how individual predictors relate to Y, then prefer a restrictive model such as linear regression or the lasso, since splines and boosting can produce estimates too complicated to interpret.
• If the goal is purely prediction on new observations, then pick flexibility by test MSE and accept lower interpretability.
• If your best test MSE is already close to Var(ε), then stop adding flexibility; remaining error is irreducible.

• Selecting the model with the lowest training MSE. The tell: the chosen model is the most flexible one and its curve is wiggly, with test error much larger than training error.
• Assuming more flexibility is always better. The tell: the test MSE curve is U-shaped but you only inspected the training curve, which declines monotonically.
• Using a rigid model on a non-linear f. The tell: the linear fit has high training and high test MSE, so it underfits and neither error is low.
• Choosing a very flexible method when you need to explain predictor effects. The tell: you cannot say how any individual predictor is associated with the response.
• Treating the U-shape as dataset-specific. The tell: you skip the flexibility sweep on a 'simple' problem, though the source says this property holds regardless of data set and method.

• ▸Method and flexibility knob: name the method and the parameter that controls flexibility (degrees of freedom, polynomial degree, knots, tree depth, K, network size).
• ▸Goal: state prediction or inference; this sets how much interpretability matters.
• ▸Flexibility levels tried: list at least three, from most restrictive (e.g., linear, 2 df) to most flexible.
• ▸Training MSE per level: record each; expect a monotone decline.
• ▸Test MSE per level: record each on data not used for training.
• ▸Shape of test curve: note where it declines, levels off, and rises.
• ▸Chosen level and reason: pick the level at the minimum test MSE and say why.
• ▸Irreducible error estimate Var(ε): record if known, to judge distance from the optimum.
• ▸Interpretability check: note whether the chosen fit still lets you explain each predictor's role.

1. 1Diagnose whether you need a penalty at all. Compare n to p: if n is much larger than p, least squares has low variance and plain fitting often suffices. If n is not much larger than p, the fit varies a lot and overfits. If p > n, there are infinitely many least squares solutions, each with zero training error and very high test variance, so a penalty is required.
2. 2Choose the penalty family by goal. Use ridge (shrinkage) when you want to reduce variance and keep all predictors. Use the lasso when you also want some coefficients set exactly to zero, since least squares almost never produces exact zeros. Treat dropout and boosting shrinkage as the same dial in other model classes (Section 10.7 covers dropout and regularization in neural networks).
3. 3Treat the strength as a tuning parameter, not a constant. The source treats lambda as something you select (Section 6.2.3, 'Selecting the Tuning Parameter'), and Chapter 5 supplies the tools: validation set, leave-one-out, and k-fold cross-validation.
4. 4Build a grid of strength values from weak (close to least squares) to strong (coefficients heavily constrained). Fit the model at every grid value on the same training folds so the comparison is fair.
5. 5Score each value by cross-validated test error, not training error. Training error always favors the weakest penalty, because an overfit model can fit the training data perfectly.
6. 6Pick the value at the bottom of the cross-validation error curve. The principle is to give up a small increase in bias to get a substantial drop in variance. Stop strengthening once added bias outweighs the variance saved.
7. 7Refit on all the data at the chosen strength. For lasso, read off which coefficients are zero to judge interpretability. Confirm that the shrinkage did not remove variables you have domain reasons to keep.
8. 8Apply the same logic to other smoothness-type penalties. In the Wage lab, smoothing-spline lambda is mapped to degrees of freedom (df 1, 3, 4, 8, 15). Smaller smoothness penalty means a rougher fit, and a df of 1 (plus intercept) is just a linear fit.

The source's thin-plate spline example shows the tuning problem in a single dial. Fit with a chosen level of smoothness (Figure 2.5), the spline gives a remarkably accurate estimate of the true f. Lower the smoothness penalty (Figure 2.6) and the fit becomes rougher and passes through the observed data perfectly. That perfect fit is far more variable than the true function, so it is overfitting and would predict poorly on new observations. The Wage lab makes the same dial explicit for smoothing splines. It sets lam so the effective degrees of freedom equal 4 (approx_lam returns lam_4, and degrees_of_freedom confirms 4.0000001). It then loops over df in [1, 3, 4, 8, 15] and plots each fit, adding one to df to account for the intercept. df=1 gives a straight line, and df=15 gives a wiggly curve. You would then choose among these by cross-validation, as the source says ('We discuss methods for choosing the correct amount of smoothness in Chapter 5'), not by how well the curve hugs the training points.

• If n is much larger than p and the relationship is roughly linear, expect low variance and keep the penalty weak; if n is barely larger than p, strengthen it.
• If p > n, never use unpenalized least squares: it gives infinitely many zero-training-error solutions with extremely high variance, so a penalty is mandatory.
• If you need a sparse, interpretable model with irrelevant variables removed, use the lasso; if you want to keep all variables and just stabilize them, use ridge.
• If training error keeps falling as you weaken the penalty but cross-validated error rises, you are overfitting; increase the strength.
• If fits at different strengths differ only slightly in cross-validated error, prefer the more constrained (smoother, simpler) one.

• Selecting the strength on training error. The tell: the chosen penalty is the weakest on the grid and the fit is perfect or near-perfect on training data, as with the rough thin-plate spline in Figure 2.6.
• Skipping regularization when p is close to or above n. The tell: huge coefficient variability across resamples, or non-unique least squares solutions with zero training error but poor test performance.
• Over-shrinking. The tell: cross-validated error rises on the strong side of the curve because added bias exceeds the variance saved; the fit approaches a trivial or straight-line model (df=1).
• Expecting ridge to select variables. The tell: no coefficients are exactly zero; only the lasso-type penalty sets coefficients to zero for interpretability.
• Tuning the strength once on a single validation split. The tell: the chosen value shifts noticeably when the split changes; use k-fold or leave-one-out cross-validation instead.

• ▸Model and penalty type: name the method (ridge, lasso, dropout, boosting shrinkage, smoothing spline) and the parameter being tuned.
• ▸n and p: record observations and predictors; note if n is much larger than p, close to p, or p > n.
• ▸Goal: variance reduction only, or also variable selection/interpretability.
• ▸Grid of strengths: list the values tried, from weak (near unpenalized) to strong.
• ▸Resampling scheme: validation set, LOOCV, or k-fold (state k) used to score each value.
• ▸Cross-validated error per value: record the error at each grid point, not the training error.
• ▸Chosen strength and rationale: the value at the error minimum, and whether a simpler neighbor is within noise.
• ▸Coefficients retained/zeroed: list which variables survive and flag any that contradict domain knowledge.
• ▸Final check: refit on all data and note the test or holdout result.

1. 1Name the quantity you need before choosing a method. Model assessment (how well does this model perform?) and model selection (which flexibility level or tuning parameter?) both use resampling, but they answer different questions. Write down which one you are doing.
2. 2Treat training error as unusable for selection. The passages state it can dramatically underestimate test error, so never pick the winner by lowest training error.
3. 3Check whether a large designated test set exists. If it does, estimate test error directly. If not (the usual case), estimate test error from the available training data by holding out subsets.
4. 4Define the grid of tuning values first. In the KNN lab the tuning parameter (hyperparameter) is the number of neighbors, and you do not know a priori which value to use. Loop over K from 1 to 5 and record performance for each.
5. 5Choose the resampling scheme. The validation set approach holds out one subset. Leave-one-out and k-fold cross-validation repeat the hold-out across multiple splits. Cross-validation applies to both regression and classification. Use the bootstrap when the target is the accuracy of a parameter estimate or of the method itself.
6. 6For each tuning value, refit the model on the training portion of each split and score it on the held-out portion with the metric that matches the goal (error rate, or the fraction correct among predicted positives).
7. 7Pick the tuning value or complexity level with the best held-out estimate. Then refit on all the available data with that value.
8. 8Choose the metric before looking at results. In the insurance example the overall error rate is not of interest, only the fraction correct among customers predicted to buy.

The Chapter 4 lab on the insurance data shows why the metric matters when you tune. Only 6% of customers buy insurance, so selling to a random selection gives a 6% success rate, which the company considers too low given visit costs. The overall error rate is therefore not the target. The target is the fraction of predicted buyers who actually buy. Fitting KNN with K = 1 and scoring on test data gives a confusion table with 53 false 'Yes' predictions and 9 true 'Yes' predictions. Of 62 predicted buyers, 9 buy, or 14.5%, about double random guessing. K is a tuning parameter with no known best value, so the lab loops K from 1 to 5 and checks this same metric on test data at each value. The resampling-based version of this loop replaces the single test split with cross-validated held-out estimates, and selects K by that held-out figure rather than by training fit.

• If a very large designated test set exists, estimate test error directly on it; otherwise estimate it by holding out parts of the training data.
• If you are comparing flexibility levels or tuning values such as K, then select on a resampled test-error estimate, never on training error.
• If the question is the variability or accuracy of a parameter estimate or of the method itself, then use the bootstrap rather than cross-validation.
• If the outcome is a rare class (6% buyers), then score the tuning loop on the metric tied to the business goal, not on overall error rate.
• If a mathematical adjustment to training error is an option (Chapter 6 covers these), treat it as an alternative to resampling, not a substitute for judging test error.

• Choosing the tuning value with the lowest training error. The tell: the most flexible setting (for example K = 1) always wins, and performance collapses on new data.
• Confusing model assessment with model selection. The tell: the same held-out score is used both to pick the setting and to report final performance.
• Optimizing the wrong metric. The tell: overall accuracy looks fine while the fraction of correct predicted buyers is barely above the 6% base rate.
• Tuning on a single small held-out split and trusting it. The tell: the best value shifts when you change which observations are held out.
• Using the bootstrap when you need a test error estimate, or the reverse. The tell: you are reporting a standard error where you needed a predictive error, or vice versa.

• ▸Goal: model assessment, model selection, or parameter accuracy.
• ▸Test set available? If a large designated one exists, note it; otherwise state the resampling method.
• ▸Tuning parameter(s): name each hyperparameter (e.g. number of neighbors K).
• ▸Candidate grid: list the values to try (e.g. K = 1 to 5).
• ▸Resampling method: validation set, leave-one-out, k-fold, or bootstrap, with the reason.
• ▸Metric: the quantity that matches the real objective (e.g. fraction correct among predicted buyers).
• ▸Base rate: the random-guess benchmark (e.g. 6% purchase rate).
• ▸Held-out result per candidate: record the estimate for every grid value.
• ▸Selected value and final refit: chosen setting and confirmation of refit on all available data.

1. 1Write down n and p before choosing a method. Use the book's notation: n is the number of observations (rows of X), p is the number of variables available for prediction (columns of X). The Wage data set has n = 3,000 and p = 11. Treat the ratio of observations per predictor as your first diagnostic.
2. 2Classify the regime. n very large with p small favors a flexible method. p very large with n small is the setting where flexible methods are expected to do worse (Exercise 1a/1b). If p is in the thousands or millions (modern biological data, web advertising data), assume high-dimensional behavior from the start.
3. 3Count effective dimensionality, not just raw columns. Noise predictors that are not associated with the response still add to p. When you add features, ask whether each is truly associated with the response. Signal features reduce test error. Noise features raise it.
4. 4If you consider a local method such as KNN, check whether neighbors can exist. The K nearest observations to x0 may be very far from x0 in p-dimensional space. Spreading 50 observations over p = 20 dimensions leaves a given point with no nearby neighbors. Work the exercise arithmetic: a 10% window of the range in one feature captures about 10% of the data, and the fraction shrinks as features are added.
5. 5When observations per predictor are few, default to a parametric method (linear regression). Fit it first as the baseline, then compare against any non-parametric alternative on an independent test set, not training error.
6. 6If p is large and you fit a linear model, use regularization (the lasso or similar). Tune the penalty λ by validation. Expect a larger λ to be chosen as p grows (when p = 20 a small λ was best). Report degrees of freedom, the number of non-zero lasso coefficients, as the flexibility measure.
7. 7Evaluate each change in p on held-out data. Record test MSE as p increases and look for deterioration. Do not read training fit as evidence that added predictors help.
8. 8Weigh interpretability before accepting a flexible fit. Even at low p, if KNN's test MSE is only slightly lower than linear regression's, prefer the linear model: it is described by a few coefficients and gives p-values.

The book's KNN versus linear regression simulation uses 50 training observations and adds noise predictors that are unrelated to the response. At p = 1 or p = 2, KNN beats linear regression, because 50 points are enough to estimate f(X) in one or two dimensions. At p = 3 the results are mixed. At p ≥ 4 linear regression is better. As noise dimensions are added, linear regression's test MSE deteriorates only slightly, while KNN's MSE rises more than ten-fold. The cause is an effective reduction in sample size: at p = 20, the 50 observations are spread so thinly that a test point has no nearby neighbors, and its K nearest neighbors are far from x0. The prediction of f(x0) is therefore poor. The lasso example follows the same pattern: with p = 20 versus p = 2,000, test MSE almost doubles, because noise features invite chance associations with the response on the training set. The practical lesson is that the method ranking flips as p grows with n fixed, so you re-run the comparison at your actual p.

• If observations per predictor are small, then choose a parametric method (linear regression) over a non-parametric one like KNN.
• If n is extremely large and p is small, then a flexible method is a reasonable candidate, but still compare it to linear regression on test MSE.
• If p is large and you are fitting a linear model, then use regularization and select the tuning parameter λ by validation; expect a larger λ as p grows.
• If the added features are not truly associated with the response, then do not add them: they raise test error by increasing overfitting risk with no upside.
• If KNN's test MSE is only slightly lower than linear regression's, then keep linear regression for the simple, few-coefficient model with p-values.

• Assuming more features always improve the model. The tell: training fit improves while test MSE rises (it almost doubles from p = 20 to p = 2,000 in the lasso example).
• Using KNN or other local methods at high p with few observations. The tell: nearest neighbors are far from the test point and test MSE blows up (over ten-fold in the simulation).
• Judging method choice at one dimension only. The tell: KNN wins at p = 1 or 2 and you carry that conclusion to p ≥ 4, where linear regression wins.
• Leaving the regularization parameter at a fixed small value regardless of p. The tell: validation error is best at larger λ as p grows, but your model uses a small λ, so test error is poor.
• Ignoring interpretability when the accuracy gain is small. The tell: you pick a flexible model for a marginal MSE gain and lose the few coefficients and p-values.

• ▸n (observations): enter the number of rows in X.
• ▸p (predictors): enter the number of columns available for prediction.
• ▸Observations per predictor (n/p): compute it; flag it if small.
• ▸Regime: label as n large/p small, p comparable to n, or p much larger than n.
• ▸Candidate noise predictors: list features you suspect are not associated with the response.
• ▸Baseline model: record linear regression test MSE on held-out data.
• ▸Flexible/local model: record KNN or other non-parametric test MSE at the same p.
• ▸Regularization: record the method, the chosen λ, and degrees of freedom (non-zero coefficients).
• ▸Decision and interpretability note: state the chosen model and whether a small MSE gain justifies losing a simple, coefficient-based model.

1. 1Separate your predictors into signal (truly related to the response) and noise (unrelated) before you pick a method. You never observe this split directly, so treat it as a hypothesis and test it with cross-validation.
2. 2Count dimensions against observations: record n and p. The source's hard setting is p = 45 with n = 50; the easy setting is a blood-pressure model with 3 predictors and thousands of patients. As p approaches or exceeds n, noise features carry more risk.
3. 3Fit the least squares baseline first. Record its test or cross-validated error and which coefficients it makes large. Treat it as the no-shrinkage reference point.
4. 4Fit a regularized model (lasso for a sparse suspected signal, ridge or PCR otherwise) and choose the tuning parameter by ten-fold cross-validation. Plot cross-validation error against the penalty and mark the minimum.
5. 5Read the shape of the cross-validation curve. A minimum at small shrinkage with a shallow dip means the data supports little regularization, and many penalty values give near-identical error, so least squares is acceptable. A pronounced dip at heavy shrinkage means noise dominates and regularization is doing real work.
6. 6Inspect the coefficient paths at the cross-validation minimum. If only a few coefficients are non-zero and they stand clearly above the grey mass of the rest, you have found separable signal.
7. 7Add features only if they are plausibly associated with the response. Re-measure test error after each batch. Noise features raise test error with no upside, and even relevant ones can add variance that outweighs the bias reduction.
8. 8When the model is misspecified, don't read the noise-variance estimate literally. A non-linear relationship fit with a linear model inflates the residuals and therefore the estimated σ², so apparent noise can be model error rather than irreducible error.

The source's sparse simulation (Figure 6.13) has p = 45 predictors, n = 50 observations, and only two predictors related to the response. This is a low signal-to-noise setting because 43 of the 45 variables are pure noise. Least squares, shown at the far right of the coefficient plot, assigns a large coefficient to only one of the two signal variables, so it cannot recover the true relationship. Ten-fold cross-validation applied to the lasso gives a minimum error at a penalty where only the two signal variables have non-zero coefficients. The two signal variables carry much larger estimates than the grey noise lines. Cross-validation plus the lasso therefore finds the true structure even in this challenging setting. A second case in the source shows the cost of added noise: in Figure 6.24, test MSE almost doubles as p grows from 20 to 2,000. Similarly, in Figure 3.20, adding noise predictors to a non-linear problem increases KNN's MSE more than ten-fold, while linear regression deteriorates only slightly.

• If the cross-validation minimum sits at small shrinkage and the dip is shallow, then the signal is strong relative to noise and least squares is a reasonable choice.
• If p is close to or above n and most predictors are likely unrelated, then use regularization (lasso if you expect few signals) and tune it by cross-validation.
• If extra features are not clearly tied to the response, then leave them out; test error rises with noise features, so each one needs a reason.
• If you use a flexible, local method such as KNN and p grows beyond a few noise predictors, then switch to a more rigid method such as linear regression, since noise predictors hurt KNN much faster.
• If the residuals look inflated because the true form is non-linear, then fix the model form (for example add a quadratic term) before concluding the data are noisy.

• Equating more features with a better model. Tell: training fit improves while test MSE rises, as when MSE almost doubles from p = 20 to p = 2,000.
• Using least squares in a p ≈ n setting. Tell: large coefficients on only some true signal variables while chance associations get non-zero weight, as in the Figure 6.13 comparison.
• Over-reading a flat cross-validation curve as a precise optimum. Tell: a wide range of penalty values gives near-identical error, so the chosen value is arbitrary.
• Blaming irreducible noise for what is model misspecification. Tell: a clearly non-linear pattern in the data and an inflated σ̂² from the RSS; a quadratic fit brings the standard errors closer to the bootstrap.
• Skipping tuning-parameter selection in high dimensions. Tell: a regularized fit chosen by default rather than by cross-validation, with unstable test error.

• ▸Sample size n and predictor count p: record both and the ratio p/n.
• ▸Suspected signal predictors: list those with a real reason to relate to the response.
• ▸Suspected noise predictors: list those with no known link to the response.
• ▸Least squares baseline error: cross-validated or test MSE with all predictors.
• ▸Regularization method and tuning method: for example lasso with ten-fold cross-validation.
• ▸Cross-validation curve shape: note whether the minimum is at small or large shrinkage and whether the dip is shallow or pronounced.
• ▸Non-zero coefficients at the minimum: count them and check them against the suspected signal list.
• ▸Test error after adding or removing feature batches: log the change each time.
• ▸Model-form check: note any visible non-linearity that would inflate σ̂² and mislead the noise estimate.