Practical Statistics for Data Scientists

1. 1Type every column first. Mark each as continuous, discrete, binary, or ordinal, because data typing tells the software how to process the column. Check that the software's inferred dtype matches your intent. Taxonomies differ between pandas, R, and SQL.
2. 2Decide which methods in your pipeline are distance- or variance-based (KNN is the passage's example). Scaling is mandatory for those. LDA weights also need it if you want to read them as importance.
3. 3Compute location and spread per numeric feature before choosing a scaler. Look at mean vs. median, standard deviation vs. IQR or MAD, and the min and max. Large gaps between the mean and median, or an extreme min or max, flag outliers that will distort a z-score.
4. 4Pick the scaler to match the spread. The default is the z-score: subtract the mean, divide by the standard deviation. If outliers are present, use a robust variant: the median for location and the IQR (or MAD) for scale. If you want bounded inputs, squash into 0–1. The MAD can be multiplied by 1.4826 to put it on the standard-deviation scale for a normal distribution.
5. 5Fit the scaler on the training data only, then apply that same fitted scaler to every later record. In the passage, `scaler.fit(X * 1.0)` is followed by `transform` on both `X` and `newloan`. Multiplying by 1.0 forces integer columns to float.
6. 6Run the distance-based model on the scaled matrix, then map results back to the original scale for reporting. In the loan KNN example, `X.iloc[nbrs[1][0], :]` displays the neighbors in original units while the search ran on scaled data.
7. 7Treat unit variance as a weighting decision, not a neutral default. It assumes every variable is equally important for prediction. If you have subjective knowledge that a variable matters more (the passage suggests payment-to-income ratio for loans), scale that column up deliberately.
8. 8Check the outcome. Pull the nearest neighbors and confirm they are alike on all variables, not just one. Do not expect scaling to fix distribution shape: normalization does not make a skewed variable normal.

The passage's loan example uses KNN with K = 5 to find the loans most similar to a new loan. Run on raw columns, the neighbors are not alike across the variables. This is the symptom of one large-scale variable dominating the distance. The fix is `StandardScaler`. Call `scaler.fit(X * 1.0)`, then `X_std = scaler.transform(X * 1.0)` and `newloan_std = scaler.transform(newloan * 1.0)`. Fit `KNeighborsClassifier(n_neighbors=5)` on `X_std` and `y`, then call `knn.kneighbors(newloan_std)`. Display the neighbors with `X.iloc[nbrs[1][0], :]` so the reader sees original units. The passage reports that the five neighbors are now much more alike in all the variables, a more sensible result. Two caveats follow. First, a z-score gives each variable equal weight, which is arbitrary. If you believe payment-to-income ratio drives default, scale it up. Second, the z-score did not change any distribution's shape. If a column was skewed before, it is still skewed afterward.

• If the method uses distances (KNN) or you will read coefficients as importance (LDA weights), then scale first. The passage says LDA weights measure variable importance only if predictors were normalized beforehand.
• If a feature has outliers, then replace the mean and standard deviation with the median and IQR (or MAD). Variance and standard deviation are especially sensitive to outliers because they use squared deviations.
• If you have domain knowledge that one variable matters more, then scale that variable up after standardizing. Do not leave the equal-importance assumption in place by default.
• If a column is skewed and the method needs normality, then do not rely on scaling to fix it. Standardizing does not change distributional shape.
• If you need all features on a bounded range, then squash into 0–1. Check beforehand that no extreme value will compress everything else into a narrow band.

• Running KNN on raw columns. The tell is that the 'nearest' neighbors match on one large-valued variable but look unrelated on the others.
• Fitting the scaler separately on new records, or on the combined data, so the new loan is not on the training scale. The tell is that new-record neighbors shift when the batch composition changes.
• Standardizing with mean and standard deviation on an outlier-heavy column. The tell is that most values bunch near one end after scaling while one or two sit far out. The mean/SD estimates were themselves distorted by the extremes.
• Reading raw LDA weights as importance. In the passage's unscaled loan output, the weights are 7.18 for borrower_score and -0.0997 for payment_inc_ratio. Their magnitudes mix feature units with importance, so they are not comparable unless the predictors were normalized first.
• Treating unit variance as neutral, or assuming scaling makes data normal. The tell is that a known-important variable has no more influence than a minor one, or that a plot shows the same skew as before scaling.

• ▸Feature name and type: record continuous, discrete, binary, or ordinal, and confirm it matches the software dtype.
• ▸Method sensitivity: state whether the model is distance-based, variance-based, or coefficient-read-as-importance.
• ▸Location and spread check: record mean vs. median and SD vs. IQR/MAD for the column.
• ▸Outlier flag: note whether extreme values are present (yes/no) and the min and max.
• ▸Scaler chosen: z-score, median/IQR, or 0–1 squash, with a one-line reason.
• ▸Fitted on: name the training set the scaler was fit on, and confirm new records use the same fitted scaler.
• ▸Importance weighting: state whether equal weight is assumed or a variable was scaled up, and the domain reason.
• ▸Post-scaling sanity check: record whether the nearest neighbors (or equivalent output) are alike on all variables.
• ▸Reporting scale: confirm results are mapped back to original units for display.

1. 1Write the business objective as one sentence before you pick a metric, and name what a wrong call costs. The source frames it this way: data scientists 'measure' against concrete business or organizational objectives. Example objective: 'audit only the returns most likely to be cheating, within a fixed audit budget.'
2. 2Identify the outcome of interest (the 1s) and check whether it is rare, as with purchases on a website or insurance fraud. If it is, accuracy (the percent of predicted classifications that are correct) is only a first step, not the verdict. Report it, but never use it alone.
3. 3Match the metric to the question. Use recall when the question is 'how good is the model at correctly identifying 1s.' Use specificity for the 0s. Use precision when the question is how clean the flagged set is. Use AUC when the question is how well the model distinguishes 1s from 0s overall. Use lift when the question is how effective the model is at identifying 1s in a ranked list.
4. 4If a resource constraint caps how many records you can act on (audits, calls, reviews), use a lift chart. Rank records by predicted probability of being a 1, compute lift decile by decile starting with the most probable 1s, and draw the cutoff where your capacity runs out.
5. 5Compute every metric on held-out data, never on the training set. In the loan example, the default xgboost model reaches a training error of 0.133043 but a test error of 0.3529. The penalized model (lambda = 1000) shows 0.30966 on training and 0.3286 on test. In-sample numbers flatter the model.
6. 6Plot the metric against model complexity (iterations) for both training and test sets. In the source's Figure 6-10 the default model keeps improving on the training set while getting worse on the test set. The penalized model does not show this. Choose the setting from the test curve.
7. 7If the 1s are rare, improve the model with the imbalance strategies in the source: undersample (downsample) the prevalent class, oversample (upsample) the rare class by bootstrapping if necessary, up-weight the rare class or down-weight the prevalent one, or generate data. Judge the result with the same imbalance-aware metrics you chose in step 3, evaluated on untouched data.
8. 8Record the metric, the cutoff, and the reason for choosing them next to the model, so the choice is reviewable against the objective from step 1.

A tax authority can only afford to audit a limited number of returns, and it wants to spend that budget on the likeliest tax cheats. Cheating returns are the rare 1s, so overall accuracy is a weak guide: it says nothing about how many cheats the audits actually reach. The authority scores each return with the model's predicted probability and sorts from most to least probable. It then builds a lift chart decile by decile, starting with the most probable 1s, to see how much more concentrated the cheats are in the top deciles than in a random draw. Because the budget is fixed, it draws the audit line where its capacity ends. Returns above the line are audited and returns below are left alone. It also reports recall to show what share of the cheats the audited set captures, and AUC as a summary of how well the model separates 1s from 0s. All of this is computed on held-out returns. The loan data shows why: the default xgboost model scores a 0.133043 training error but a 0.3529 test error, so an in-sample metric would have overstated the model badly.

• If the outcome of interest is rare, then do not report accuracy alone. Add recall, specificity, or precision, and say which one matches the cost of a miss.
• If you can act on only a fixed number of records, then rank by predicted probability and use lift by decile to set the cutoff at your capacity.
• If you need one threshold-free number for how well the model separates 1s from 0s, then use AUC, and pair it with lift or recall when the decision is about a specific cutoff.
• If the training metric keeps improving while the held-out metric worsens, then stop adding complexity or add regularization (alpha, lambda) and choose by the held-out curve.
• If you rebalance with undersampling, oversampling, weights, or generated data, then evaluate on data that was not resampled, using the same imbalance-aware metric.

• Reporting accuracy as the headline on rare-event data. The tell: the model looks fine overall, yet nobody can say how many of the actual 1s it catches.
• Evaluating on the training set. The tell: a large gap between training and test error, like 0.133043 versus 0.3529 in the default loan model.
• Picking a metric without a stated business objective. The tell: you cannot say what a false negative or a false positive costs, or why this metric was chosen over recall, precision, AUC, or lift.
• Using AUC when the decision is capacity-limited. The tell: a strong AUC, but no one has checked how many true 1s fall in the top decile or within the budgeted number of cases.
• Tuning the model on test error and never watching it across iterations. The tell: test error rises as trees are added, as with the unpenalized xgboost model, and no one noticed because only the final number was reported.

• ▸Business objective: one sentence stating the decision and what the model is used for.
• ▸Outcome of interest (the 1s): define it and state whether it is rare.
• ▸Cost of a miss vs. a false alarm: say which error hurts more and why.
• ▸Resource constraint: the maximum number or share of records you can act on, or 'none'.
• ▸Primary metric: recall, specificity, precision, AUC, or lift by decile, with a one-line reason tied to the objective.
• ▸Secondary metrics: accuracy plus any others reported alongside the primary.
• ▸Imbalance strategy: undersample, oversample, up/down weight, data generation, or none.
• ▸Evaluation data: describe the held-out set and confirm it was not resampled.
• ▸Train vs. test check: training metric, test metric, and whether the gap grows with iterations.

1. 1Fix the sample statistic you want to characterize (mean, median, model parameter) and the original sample of size n; the bootstrap treats your sample as a stand-in for the population.
2. 2Draw a resample of size n from your data with replacement, so each observation can appear multiple times or not at all; this simulates drawing from an effectively infinite population where draw probabilities stay constant.
3. 3Record the statistic of interest for that resample, then repeat the resample-and-record cycle R times (R is set arbitrarily; more iterations mean a more accurate standard error and interval).
4. 4From the R recorded statistics, compute the standard deviation — this estimates the standard error of your sample statistic — and produce a histogram or boxplot to see how variable the estimate is.
5. 5For an x% confidence interval, sort the R results and trim [(100−x)/2]% off each end; the remaining endpoints are your bootstrap confidence interval (e.g., trim 5% off each tail for a 90% interval).
6. 6Present the estimate as this interval range rather than a single point number, to counteract the tendency to place undue faith in a point estimate.
7. 7Read the interval width diagnostically: widen it if you demand higher confidence or hold less data; narrow it if you can tolerate lower confidence or gather a larger sample.
8. 8In R use boot(data, R=1000, statistic=stat_fun); in Python loop resample() from scikit-learn R times since the major packages lack a built-in bootstrap.

Applying the bootstrap to the incomes of people taking out loans, you define stat_fun to compute the median for a resample and run boot(loans_income, R=1000, statistic=stat_fun). The original estimate of the median is $62,000. Across 1,000 resamples (each drawn with replacement, size n, median recorded), the bootstrap distribution shows a bias of about −$70.56 and a standard error of $209.15. That standard error — the standard deviation of the 1,000 resampled medians — tells you how variable the $62,000 estimate is, not a defect in the median itself. Results vary slightly between consecutive runs because the resampling is random. In Python you reproduce this by looping resample(loans_income) 1,000 times, appending each sample.median() to a list, then examining that Series.

• If a single sensor or data source is intrinsically less accurate, then downweight it via a weighted mean rather than treating all observations equally.
• If the data has extreme values or long tails (like income data with a Bill Gates in the neighborhood), then use the median or trimmed mean, since the median won't move no matter how rich the outlier is.
• If you need higher confidence or have a small sample, then accept a wider interval — both raise uncertainty; if you can tolerate lower confidence or have more data, then the interval narrows.
• If you are a data scientist communicating potential error or deciding whether to collect more data, then a bootstrap confidence interval suffices; if you are publishing or submitting to a regulator, then you need the more formal treatment a researcher uses.
• If the major Python packages offer no bootstrap function, then implement it with scikit-learn's resample in a loop rather than searching for a built-in.

• Interpreting a 90% confidence interval as '90% probability the true value lies in this interval' — the tell is phrasing the claim as 'given this sample, probability about the population'; the interval actually answers 'given a sampling procedure and population, how often intervals contain the statistic.'
• Reporting only a point estimate and placing undue faith in it — the tell is a single number presented with no range, hiding the variability the standard error would expose.
• Assuming standardizing to z-scores makes data normal — the tell is claiming normality after normalizing; z-scoring only puts data on the standard normal scale for comparison, it does not change the distribution's shape.
• Trusting large sample size over sampling method — the tell is the Literary Digest error: polling 10 million self-selected, high-status subscribers still produced sample bias, while Gallup's 2,000 well-selected people predicted correctly.
• Assuming raw data is normally distributed — the tell is modeling skewed income data or discrete binomial data as normal; normality typically applies to errors and to averages/totals in large samples, not raw data.

• ▸Statistic of interest: name what you are estimating (mean, median, parameter).
• ▸Sample size n: record the number of observations in your original sample.
• ▸Resampling method: with replacement, size n each draw — confirm.
• ▸R (iterations): set the number of resamples (e.g., 1000); more improves accuracy.
• ▸Original estimate: the statistic computed on the full original sample.
• ▸Estimated standard error: standard deviation of the R resampled statistics.
• ▸Confidence level x%: the coverage you require (e.g., 90% or 95%).
• ▸Trim per tail: compute (100−x)/2 % and mark the endpoints after trimming.
• ▸Reported interval + use: state the [low, high] range and whether it's for communication or deciding on a larger sample.

1. 1Count your looks before you read results. List every predictor tested, every model fit, and every comparison made on the same data. Each one is another chance for noise to pass as signal, so the count sets your baseline risk.
2. 2Compute the chance of at least one false positive: 1 − (0.95)^k for k tests at alpha = 0.05. For k = 20 the passage gives 0.64. If that number is high, treat any single 'significant' finding as unproven.
3. 3Set aside a labeled holdout set before any model selection. Fit and tune only on the training portion, then assess on data the model has not seen. This is the passage's primary mitigation in supervised learning.
4. 4If you have no labeled holdout (unsupervised or other tasks), assume the risk persists. Treat conclusions as provisional and apply p-value adjustment where your design fits, such as multiple treatment groups or repeated looks in a clinical trial.
5. 5Do not trust in-sample selection alone. Stepwise, forward, backward, and all-subset regression assess and tune models on the same data they were fit on, so validate the chosen model with cross-validation.
6. 6Constrain complexity directly. In regression, use penalized methods (ridge, lasso) that shrink coefficients toward zero. In scikit-learn logistic regression, keep the penalty and C regularization on by default; turn it off only deliberately, as with C = 1e42.
7. 7Cap flexibility in smoothers. In pyGAM, the default n_splines = 20 overfits at larger SqFtTotLiving values; setting n_splines = 12 gives a more reasonable fit.
8. 8Use cross-validation most heavily for flexible, iterative models that respond to local data structure. Linear regression is typically less exposed because of its simple global linear structure.

The source's own example is 20 randomly generated predictors and one randomly generated outcome, with a significance test run on each at alpha = 0.05. The chance a single predictor correctly tests nonsignificant is 0.95. The chance all 20 do is 0.95^20 = 0.36. So the chance at least one predictor falsely tests significant is 1 − 0.36 = 0.64. A model built on that 'significant' predictor fits noise: no real relationship exists, yet the training data seem to support it. This is alpha inflation, and it is the same mechanism as overfitting in data mining. A holdout set exposes the problem, because the spurious predictor has no pull on data the model has not seen. The pyGAM case shows the same risk in a flexible model: with the default n_splines = 20 the fit overfits at larger SqFtTotLiving values, and 12 splines gives a more reasonable fit.

• If you ran many tests or models on the same data, then expect alpha inflation: do not report the best p-value at face value, and adjust p-values where your design allows.
• If a labeled holdout set exists, then judge every model on it, never on the training data alone.
• If you selected variables with stepwise, forward, backward, or all-subset methods, then confirm the result with cross-validation, because these are in-sample methods.
• If the model is flexible or iterative (splines, GAMs, local-structure procedures), then cross-validate and limit complexity, for example by reducing n_splines from the default 20 to 12.
• If you must use p-values as intermediate inputs for feature inclusion, then treat them as one point of information, not a controlling rule.

• Reporting one 'significant' predictor out of many tests. Tell: you cannot say how many tests were run; with 20 tests the false-positive chance is already 0.64.
• Judging a model on the data it was fit to. Tell: strong training performance with no holdout or cross-validated check.
• Trusting stepwise or all-subset selection as final. Tell: the chosen variable set was never validated on out-of-sample data.
• Leaving flexible-model defaults untouched. Tell: wiggly fits in sparse regions, as with default n_splines = 20 at larger SqFtTotLiving values.
• Switching regularization off by accident or without a reason. Tell: scikit-learn LogisticRegression with a huge C (such as 1e42) in a model you meant to protect against overfitting.

• ▸Number of predictors tested: count every variable examined against the outcome.
• ▸Number of models/comparisons run: include all discarded and re-tuned attempts.
• ▸False-positive chance at alpha 0.05: compute 1 − 0.95^k using the total number of looks.
• ▸Holdout set available (yes/no): state where it was split and confirm it was untouched during tuning.
• ▸Selection method used: stepwise, forward, backward, all-subset, penalized, or none.
• ▸Cross-validation performed (yes/no) and result: record the out-of-sample score beside the in-sample score.
• ▸Complexity controls: note the regularization setting (penalty, C) and any spline or flexibility cap (n_splines).
• ▸P-value adjustment applied: describe the method, or write why none fits this design.
• ▸Overall risk call (low/medium/high) and next action: justify from the fields above.

1. 1Name the selection process before you model. Write down how each record reached your data set: random sample, convenience sample, cherry-picked subset, or a time window you chose. Nonrandom sampling, cherry-picked data, time intervals that accentuate an effect, and stopping an experiment when results look 'interesting' are the typical forms of selection bias the passages list.
2. 2Count your searches. The vast search effect says that if you repeatedly run different models and ask different questions of a large data set, you are bound to find something interesting. Log every model, feature set, and question you try, so you know how many chances chance had.
3. 3Split off one holdout set before exploration starts, and sometimes more than one. Validate performance only on data the search never touched. A result that exists only in the data you searched is a candidate, not a finding.
4. 4Run target shuffling (a permutation test, in essence) on any predictive association the model suggests. Shuffle the outcome, refit with the same search procedure, and see whether chance alone produces performance as good as yours.
5. 5Ask whether the result is within the range of normal chance variability. A p-value is useful for this: it is the probability that results as extreme as the observed ones occur under a null hypothesis model. Treat it as one more point of information, not the controlling decision tool.
6. 6Check whether you selected an extreme value. If you chose the best performer, the top segment, or the most striking reading, expect the next measurement to be more central. Skill and luck both contributed to the extreme, and only the skill persists. Do not read the drop as a cause.
7. 7Quantify sampling variability with the bootstrap. Estimate standard error by bootstrap resampling, with no distributional assumptions required, and do not confuse it with standard deviation (variability of individual points versus variability of a sample metric). Remember the square root of n rule: halving the standard error needs four times the sample.
8. 8Inspect residuals if you report confidence intervals for predictions. Plot absolute residuals against predicted values and look for heteroskedasticity. Non-normal errors suggest the model may be missing something. Formal inference needs normal, constant-variance, independent residuals, while predictive accuracy does not.

The passages offer the rookie-of-the-year, sophomore-slump case. In a given season's rookie class, one athlete performs better than all the rest. In nearly all ball-or-puck sports, performance has two elements: skill and luck. Picking the top rookie selects for both, because skill and good luck are probably contributing. In the second year the skill is still there, but very often the luck is not, so performance falls. A naive analyst might attribute the drop to pressure or complacency and build a model around it. The bias-aware analyst sees that the selection rule (best performer) guaranteed the extreme value, so the decline is expected regression to the mean. The same logic applies to the stadium coin-flip illustration: someone who gets 10 heads in a large crowd has shown luck, not special talent. To check a claim like this, compare the selected group's next-period results against the pool's overall central tendency, rather than reading the fall as a causal effect.

• If you ran many models or questions on one data set, then require confirmation on a holdout set (or more than one) before believing any result.
• If a model claims a predictive association, then run target shuffling and ask whether shuffled outcomes can match its performance.
• If you selected a case because it was the best or worst, then expect the next measurement to be more central and do not assign causal meaning to the change.
• If a p-value is above your alpha threshold, then say 'effect not proven' rather than 'no effect'; a larger sample might give a smaller p-value.
• If you need uncertainty on a statistic and don't want to rely on distributional assumptions, then use bootstrap resamples to estimate standard error.

• Treating the top result of a wide search as a discovery. The tell: it was one of many models or questions and was never tested on untouched data.
• Reading regression to the mean as a real effect (a 'slump'). The tell: the subject was chosen for an extreme value and the next value moved toward the center.
• Choosing a time window or subset because it shows the effect. The tell: the effect vanishes or shrinks when the interval or sample is changed.
• Stopping a study when results look interesting. The tell: the stopping point was decided after peeking at outcomes, not in advance.
• Confusing standard deviation with standard error, or treating a p-value as the decision itself. The tell: an interval or conclusion is stated with unwarranted precision, or one threshold alone drives the call.

• ▸Claim being tested: state the finding in one sentence.
• ▸Sampling process: describe how each record entered the data (random, convenience, filtered, time-windowed).
• ▸Selection red flags: note any cherry-picking, chosen time interval, or early stopping.
• ▸Search count: list the number of models, features, and questions tried before this result.
• ▸Holdout plan: say which data was set aside before exploration and the result on it.
• ▸Target-shuffling result: record how shuffled-outcome runs compare with the real performance.
• ▸Extreme-value check: say whether the subject was chosen for an extreme and what a regression-to-the-mean expectation would be.
• ▸Chance check: give the p-value or bootstrap standard error and your alpha threshold, plus the conclusion ('within chance' or 'effect not proven').
• ▸Residual check: note heteroskedasticity or non-normality found if you report prediction intervals.

1. 1Fix the outcome type first. The data type of the outcome decides the metric. A continuous outcome (a closing price) takes an error metric such as RMSE. A binary outcome (the 0/1 'Competitive?' column in the auction table) takes classification metrics such as AUC and recall.
2. 2Check that the sample represents the population the model will score. Data quality often matters more than quantity, and statistics adds representativeness. If the sample misrepresents the population, no accuracy figure computed on it transfers.
3. 3Set the holdout aside before any modeling. Records in it are never used to fit, tune, or choose features. Decide up front how many holdout sets you can afford, because more than one is sometimes needed.
4. 4Draw the holdout by a stated sampling procedure, ideally random so each available record has an equal chance of selection. Use stratified sampling if a subgroup must be present in both fitting and evaluation data.
5. 5Fit models on the training portion only. Score the holdout once per candidate and record the RMSE, or the AUC and recall. Log how many models and questions you tried, since that count is your exposure to the vast search effect.
6. 6Run target shuffling (a permutation test). Shuffle the outcome, refit the same pipeline, and score again. If the shuffled outcomes score nearly as well as the real ones, the 'predictive association' is chance.
7. 7Use the bootstrap to measure the reliability of your accuracy estimate. Resample, re-score, and report the spread rather than a single number.
8. 8Confirm the winning model on a holdout it has never touched. Treat a drop from the earlier score as the honest estimate of performance on new data.

The Literary Digest poll of 1936 shows how accuracy on new data fails through sampling, not volume. The magazine polled its entire subscriber base plus additional lists, over 10 million people, and predicted a landslide victory for Alf Landon. George Gallup polled just 2,000 people biweekly and correctly predicted a Roosevelt victory. The difference was selection: the Digest paid little attention to how people were chosen and ended up polling people of relatively high socioeconomic status. Its 'training sample' was huge but did not represent the population that would vote. Apply this to a model: a holdout drawn from the same skewed source as the training data will report flattering accuracy and still miss on the real population. Before trusting any RMSE or AUC, ask whether the evaluation records were drawn the way future records will arrive. Sample size cannot repair a sample that misrepresents the population.

• If the outcome is binary (e.g., competitive auction 0/1), then report AUC and recall. If it is continuous, then report RMSE.
• If you have scored many candidate models against the same holdout, then treat the best score as inflated (the vast search effect). Confirm on a fresh holdout.
• If a model's score looks impressive but you cannot say why, then run target shuffling. If shuffled outcomes score close to the real ones, then discard the association.
• If you chose records or entities because they performed extremely well, then expect worse results next period. Regression to the mean (the 'rookie of the year, sophomore slump') means the luck component does not repeat.
• If the sampling procedure is nonrandom or its selection is unexplained, then fix representativeness before computing any metric. A bigger sample does not help.

• Tuning on the holdout: you test many models and features against one holdout and report the best. The tell is a large gap between the holdout score and the score on newly arriving data.
• Trusting size over representativeness, the Literary Digest error: you cite millions of records while no one can state how they were selected. The tell is that the sample over-represents one segment, such as high socioeconomic status.
• Mistaking chance for signal: you run many models and questions on a large data set and report the 'interesting' one. The tell is that a target-shuffled run scores nearly as well.
• Selecting on an extreme result: you pick the top performer from one period and forecast the same level next. The tell is that performance falls toward the middle the following period, because skill persists and luck does not.
• Reporting a single point estimate: one RMSE or AUC is presented as exact. The tell is that you cannot state how much it would change on a different sample; bootstrap resampling would show the spread.

• ▸Outcome variable and type: name it and mark continuous or binary. This picks RMSE vs. AUC/recall.
• ▸Target population: who or what the model will score in future.
• ▸Sampling procedure: how training and holdout records were drawn (random, stratified, other) and why they represent the population.
• ▸Holdout definition: how many holdout sets, how they were separated, and the date they were locked.
• ▸Number of models and questions tried: the count that sets your vast-search exposure.
• ▸Holdout metric(s): RMSE, or AUC and recall, from the single scoring of each final candidate.
• ▸Target-shuffle result: the score with shuffled outcomes next to the real score.
• ▸Bootstrap spread: the range of the metric across resamples.
• ▸Fresh-holdout confirmation: the score on data the chosen model never touched, and the gap versus the earlier score.

1. 1State the interpretive question before fitting anything: which variables matter, what structure exists, or what mechanism explains the outcome. The passages distinguish prediction from explanation: 'In data science, the most important use of regression is to predict... In some cases, however, gaining insight from the equation itself... can be of value.' Only the second case needs interpretability work.
2. 2Summarize raw relationships first. For two numeric variables use a contour plot, heat map, or hexagonal binning (two-dimensional density). For two categorical variables build a contingency table with row percentages so each category's outcome mix can be compared directly.
3. 3Check predictor correlation before reading any coefficient. 'When the predictor variables are highly correlated, it is difficult to interpret the individual coefficients.' Near-perfect correlation (multicollinearity) makes the regression unstable or impossible to compute. Treat coefficients from correlated predictors as joint, not individual, effects.
4. 4Encode variables so coefficients stay readable. Encode a factor with P levels as P − 1 dummy variables. Consolidate factors with many levels. Keep ordered factors, such as building grades 1 Cabin through 13 Mansion, as a single numeric variable to preserve the ordering.
5. 5Ask what is missing: a confounding variable (an important predictor omitted from the model) produces spurious relationships. Ask also whether the effect of one predictor depends on another. If the relationship is interdependent, add an interaction term, because a main effect alone is 'independent of other variables.'
6. 6Run diagnostics on the residuals. Use standardized residuals to find outliers, leverage (hat-values) to find influential records, and look for heteroskedasticity, which 'may indicate a predictor missing from the equation.' Use partial residual plots to examine the outcome against one predictor at a time.
7. 7To expose structure across many numeric predictors, use principal components (linear combinations, minimized correlation between them, a few explaining most variance) or K-means clustering. Standardize continuous variables with z-scores before clustering so large-scale variables do not dominate.
8. 8Treat clusters as hypotheses to examine. Inspect each cluster mean (the vector of variable means) to describe what defines the group. Then use the groups directly, analyze them in more depth, or feed them as a feature or outcome to a predictive model.

The Lending Club contingency table crosses loan grade (A high to G low) with loan status (charged off, current, fully paid, late), 450,961 loans in total. Reading the row percentages gives a communicable finding: the charged-off share rises steadily with each grade step: A 0.022, B 0.040, C 0.050, D 0.067, E 0.082, F 0.118, G 0.126. The late share rises the same way, from 0.006 for grade A to 0.061 for grade G. The raw counts alone would mislead. Grade B has more charged-off loans (5,302) than grade A (1,562), but only because B holds 132,370 loans against A's 72,490. Row percentages remove that size effect. The summary the passage draws is that high-grade loans have a very low late/charge-off percentage compared with lower-grade loans. The table shows association, not mechanism, so a next step would be to check for confounders before claiming grade causes default.

• If the goal is only prediction, then do not over-invest in coefficient interpretation; if you must explain the relationship, then examine the equation, correlations, and diagnostics.
• If predictors are highly correlated, then do not interpret individual coefficients as separate effects; report them jointly or remove or combine redundant predictors.
• If a coefficient has a surprising sign or magnitude, then suspect an omitted confounding variable or a correlated predictor before accepting the 'finding.'
• If residual variance differs across ranges of the outcome (heteroskedasticity), then look for a missing predictor, not just a technical fix.
• If you cluster continuous variables with different scales, then standardize first; otherwise the large-scale variables define the clusters.

• Reading a coefficient in isolation when predictors are correlated. Tell: coefficients flip sign or change sharply when a related predictor is added or removed.
• Presenting a spurious relationship from an omitted confounder as a mechanism. Tell: the effect disappears or reverses once the important omitted predictor enters the model.
• Comparing raw counts instead of percentages in a contingency table. Tell: the largest category 'wins' simply because it is largest (grade B charge-offs exceed grade A's in count only).
• Clustering unstandardized data. Tell: clusters line up almost entirely with the one variable that has the largest scale.
• Treating one outlier or influential record as a real pattern, or ignoring it. Tell: dropping a single high-leverage record materially changes the regression equation.

• ▸Interpretive question: state in one sentence what relationship, importance ranking, or mechanism you need to explain.
• ▸Purpose check (predict vs. explain): record which applies; if only predict, note the interpretation depth you will skip.
• ▸Raw relationship view: name the plot or table used (contour, heat map, contingency table with row %) and the pattern it shows.
• ▸Correlated predictor pairs: list pairs that move together and the action taken (drop, combine, report jointly).
• ▸Variable encoding: note how each factor is encoded (P − 1 dummies, consolidated levels, ordered numeric).
• ▸Confounder and interaction check: list candidate omitted variables and interactions tested, with result.
• ▸Diagnostics findings: summarize outliers (standardized residuals), influential records (leverage), and heteroskedasticity.
• ▸Structure method: record if principal components or K-means used, K chosen, whether variables were standardized.
• ▸Plain-language finding and caveat: write the communicable conclusion and one sentence on what it does not prove.

1. 1Type every column before computing anything. Label each as continuous, discrete, binary, ordinal, or nominal categorical, and set categorical columns explicitly (in pandas, specify them in read_csv). The type determines the display, the summary, and how software computes on the variable.
2. 2Estimate location with more than one metric. Compute the mean alongside the median and a trimmed mean. If they diverge, the mean is being pulled by extreme values; if they agree, note that and move on.
3. 3Choose weights when records represent unequal groups. For state murder rates, use the weighted mean (np.average with weights=Population) and the weighted median (wquantiles.median in Python, weightedMedian in R) next to the unweighted versions.
4. 4Measure variability on the same scale as the data. Report standard deviation, IQR, and median absolute deviation from the median. Pull percentiles (5, 25, 50, 75, 95) with quantile() to see the spread, not just a single center.
5. 5Draw each numeric distribution: a boxplot for a fast read (median, box = 25th to 75th percentile, whiskers, outlier points) and a frequency table or histogram for shape. Check which whisker rule your software uses (R and Matplotlib cap whiskers at 1.5 × IQR; other tools may differ).
6. 6Summarize categorical variables as proportions and show them in a bar chart. Treat binned numeric data as categories only when binning is deliberate.
7. 7Examine correlation among predictors and between each predictor and the target. Build a correlation matrix, then confirm each notable coefficient with a scatterplot, since a coefficient between –1 and +1 compresses the relationship into one number.
8. 8Write down what you found before modeling: outliers to investigate, skewed variables, correlated predictor pairs, and any type or scaling issues (standardize, or use Gower's distance for mixed data, before distance-based methods).

The source's state data gives a compact EDA pass. Murder rate percentiles come from quantile(): 5% = 1.600, 25% = 2.425, 50% = 4.000, 75% = 5.550, 95% = 6.510. The median is 4 murders per 100,000, but the 5th percentile is only 1.6 and the 95th is 6.51, so the spread is real and the median alone would hide it. Because states differ in size, you also compute a population-weighted mean (4.445834) and weighted median (4.4). They are about the same, so weighting does not change the story and you record that. A boxplot of Population divided by 1,000,000 shows the median state near 5 million, the middle half between about 2 and 7 million, and several high-population outliers. Those outliers are the reason to prefer the median and percentiles over the mean when summarizing population, and a flag to inspect those states before any model uses the variable.

• If the mean and median differ noticeably, then report the median or a trimmed mean as the location estimate and investigate the extreme values driving the gap.
• If records represent groups of very different size (states, regions), then compute both weighted and unweighted location estimates; if they match, say so, and if they differ, justify which one answers your question.
• If you need a variability number to interpret in the data's own units, then use the standard deviation rather than the variance; if outliers are present, then add the IQR or median absolute deviation.
• If a boxplot shows points beyond the whiskers, then treat them as candidates for inspection, not automatic errors, and confirm which whisker rule your software uses.
• If the data mixes numeric and categorical variables, or variables are on different scales, then standardize or use Gower's distance before any distance-based clustering or dimension reduction.

• Reporting only the mean. The tell: no median or percentiles in your summary, and a skewed variable like state population described by a single average.
• Ignoring group size when records are aggregates. The tell: an unweighted statistic for states or regions that was never compared with a weighted one.
• Treating boxplot outliers as errors, or ignoring them. The tell: points beyond the whiskers are deleted or never mentioned, and you never checked the 1.5 × IQR rule in your software.
• Reading a correlation coefficient without a scatterplot. The tell: a correlation matrix is the only evidence cited for a predictor relationship.
• Leaving data types implicit. The tell: categorical columns are handled as text or numbers by default, and software processes them inappropriately.

• ▸Variable name and type: list each column as continuous, discrete, binary, ordinal, or nominal.
• ▸Location estimates: record mean, median, and trimmed mean; note any gap between them.
• ▸Weighting check: state whether records are aggregates and, if so, the weighted vs. unweighted result.
• ▸Variability and percentiles: record standard deviation, IQR, and the 5/25/50/75/95 percentiles.
• ▸Distribution plot findings: describe the boxplot and histogram shape, and list outliers beyond the whiskers.
• ▸Categorical proportions: list category proportions and the bar chart observation.
• ▸Correlation findings: name predictor pairs and predictor-target pairs with notable correlation, each confirmed by scatterplot.
• ▸Scaling or encoding actions: note standardization, Gower's distance, or categorical declarations needed before modeling.
• ▸Open issues to investigate: list outliers and anomalies to resolve before fitting a model.

1. 1Define the target population in writing before touching data. Use the customer-survey test from the source: does 'customer' mean purchase amount > 0? Do you include past customers, refunds, internal test purchases, resellers, or both billing agent and customer? Each yes/no changes the population you are sampling.
2. 2Check that your accessible population (the list or flow you can actually draw from) matches the target population. Ask who is missing from the list. The Literary Digest drew from subscribers plus marketers' lists of phone and car owners, so it missed lower-income voters.
3. 3Choose the draw procedure and state it as a rule, e.g. 'select 100 customers at random.' Every available member gets an equal chance at each draw. Decide explicitly between with replacement (observations go back for possible reselection) and without replacement (once chosen, unavailable).
4. 4For flows such as web visitors or live transactions, build in timing. A visitor at 10 a.m. on a weekday differs from one at 10 p.m. on a weekend, so draw across the time windows you care about, not just whenever it is convenient.
5. 5If a subgroup matters but would be too thin under simple random sampling, stratify. Divide the population into strata (homogeneous subgroups), sample randomly within each, and overweight small strata to reach equivalent sample sizes, as pollsters do for smaller demographic groups.
6. 6Set sample size by the analytical goal, not by what is available. Prefer a sample of several thousand records you can inspect, plot, and clean (missing values, outliers) over millions you cannot. Reserve massive data for big and sparse problems, like Google search queries: 150,000+ words and over a trillion queries per year.
7. 7Compare the sample to a benchmark or known actual values. If the result suggests bias, suspect a misspecified model or a missing important variable, not just bad luck. Remember that even a random sample is never exactly representative; the issue is systematic, nonrandom difference.
8. 8Fix hypotheses before looking. Specify the hypothesis, then run a well-designed test. Repeated modeling or hunting through many predictors (data snooping, the vast search effect) produces misleading or nonreproducible findings.

The 1936 US presidential poll shows the construct at full scale. The Literary Digest polled its entire subscriber base plus extra lists, over 10 million people, and predicted a Landon landslide over Roosevelt. George Gallup polled just 2,000 people, biweekly, and correctly predicted a Roosevelt victory. The difference was the selection method. The Digest chose quantity and paid little attention to how people were selected; its subscribers, and the phone and automobile owners on marketers' lists, skewed toward relatively high socioeconomic status. That is sample bias: the sample differed in a meaningful, nonrandom way from the electorate. Gallup used more scientifically chosen, random-based methods to get a sample representative of the voting electorate. Applying the lesson: before trusting 10 million records, ask who could never appear in them. Here the answer was lower-income voters, and no volume of additional records from the same lists could fix that.

• If your list or data source systematically excludes part of the target population, then more rows will not help; redefine the accessible population or change the sourcing.
• If a subgroup is too small under simple random sampling to analyze, then stratify and overweight that stratum to get comparable sample sizes.
• If you need to inspect, clean, or chase down missing values and outliers, then use a random sample of a few thousand rather than the full millions of records.
• If the data is both huge and sparse (many columns, mostly zeros, long-tail targets like search queries), then accumulate massive data; random sampling will not retrieve the rare pertinent records.
• If you are sampling from a flow (visitors, transactions), then stratify or spread the draw across time windows such as weekday daytime versus weekend night.

• Equating size with representativeness. Tell: you cite the record count (e.g., 'over 10 million') as proof of validity but cannot describe the selection method.
• Vague population definition. Tell: no one can say whether refunds, test purchases, resellers, or past customers are in or out, so results shift depending on who pulled the data.
• Convenience timing in flow sampling. Tell: all records come from one time window (e.g., weekday mornings) yet conclusions are applied to all visitors.
• Data snooping. Tell: you tried many predictors or models until something looked interesting, and the finding does not reproduce on new data.
• Reading a biased result as noise. Tell: results deviate persistently in one direction from a benchmark (like shots clustering in one quadrant), yet you only rerun the sample instead of checking for model misspecification or a missing variable.

• ▸Analytical goal: state the decision or estimate this sample must support.
• ▸Target population definition: who/what is in, and the explicit in/out calls (refunds, test records, resellers, past customers).
• ▸Accessible population / source list: where records actually come from.
• ▸Known gaps: which target members cannot appear in the source, and why.
• ▸Sampling procedure: simple random vs. stratified; with or without replacement; exact draw rule.
• ▸Strata and allocation: list strata and the sample size per stratum (note any overweighting).
• ▸Timing coverage: time windows covered if sampling from a flow.
• ▸Sample size and rationale: n, and why this size fits inspection/cleaning versus a sparse big-data need.
• ▸Benchmark check: known values or benchmark compared against, and the result (bias signal yes/no).

1. 1Name the statistic and the question first. Decide whether you need a standard error or confidence interval (bootstrap) or a hypothesis test against 'no effect' (permutation). The source treats both as resampling, but only the bootstrap always means sampling with replacement.
2. 2Bootstrap the statistic: draw one value from your sample of size n, record it, replace it, and repeat n times. Compute your statistic (mean, median, or a model parameter) on those n values and store it.
3. 3Repeat that resample-and-record loop R times. The source says R is 'set somewhat arbitrarily' and that more iterations give more accurate results, so choose a large R and rerun with a larger one to see whether the answer moves.
4. 4Summarize the R stored results. Their standard deviation estimates the standard error. A histogram or boxplot shows the shape of the sampling distribution, and the interval from the R results gives the confidence interval. Also compute bias as mean(results) minus the original statistic, as the source's code does.
5. 5For multivariate data, resample whole rows as units, then fit the model on each bootstrap sample. The spread of each parameter across fits estimates its stability.
6. 6For a permutation test, combine the groups, shuffle, and recompute the metric many times. Compare the observed value to this shuffled distribution. The source's illustration: the sum of products of {1,2,3} and {4,5,6} is 32, and no shuffle can exceed it. A raw metric like that only means something relative to the resampling distribution.
7. 7To plan sample size, rerun the bootstrap with different values of n and watch how the sampling distribution narrows.
8. 8Report the original statistic, bias, standard error, and interval together, and state that the bootstrap describes how additional samples from a population like yours would behave. It does not add information beyond your sample.

The source's code computes a bootstrap of the median of loans_income and prints three numbers: the original median, the bias (results.mean() minus the original median), and the standard error (results.std()). Walk through it. You have a sample of loan incomes and want the uncertainty of the median, a statistic with no convenient textbook formula. Draw n incomes with replacement, take their median, and store it. Repeat R times. The standard deviation of the R medians is your standard error. Mean of the R medians minus the original median is the bias, and a value near zero says the resampled medians center on your observed one. A histogram of the R medians shows the sampling distribution, and its interval gives the confidence interval. No normality assumption on incomes is needed. The method gives you variability, not a better median.

• If the statistic has no established mathematical approximation (median, model parameters, many other metrics), then bootstrap it. The sampling distribution of the mean has been established since 1908, but that of many other metrics has not.
• If you are testing whether an observed difference or association could arise by chance, then use a permutation (shuffling) procedure against the observed value instead of a bootstrap.
• If the data is multivariate or you are assessing model stability, then resample rows as units and refit the model on each bootstrap sample.
• If a single tree or model is unstable and you want better predictions, then aggregate predictions over bootstrap samples (bagging): average them, or take a majority vote for classification.
• If your sample is small, then do not expect the bootstrap to rescue it. It cannot create new data or fill holes in the data, so collect more data instead.

• Treating the bootstrap as a fix for a small sample. The tell: a tight-looking interval from very few observations, with confidence that the data does not support. The bootstrap only shows how samples from a population like yours would behave.
• Resampling without replacement when you mean to bootstrap. The tell: every resample reproduces the original data, so the standard error comes out near zero. Bootstrap always implies sampling with replacement.
• Resampling individual values when rows belong together. The tell: model parameters fit on resampled data show relationships broken apart. Sample rows as units for multivariate data.
• Using too few iterations and trusting the output. The tell: the standard error or interval shifts noticeably when you rerun with a different seed or larger R.
• Reading a raw permutation metric on its own. The tell: quoting a sum of products like 32 as 'strong' without comparing it to the shuffled distribution. Its meaning comes only from the resampling distribution.

• ▸Statistic of interest: name the mean, median, parameter, or difference being assessed.
• ▸Goal: standard error, confidence interval, or hypothesis test.
• ▸Procedure: bootstrap (with replacement) or permutation (shuffling combined groups).
• ▸Sampling unit: single values or whole rows.
• ▸Original sample size n: record it, and note whether it is small enough to limit what resampling can tell you.
• ▸Number of iterations R: state the value and the larger value used to confirm stability.
• ▸Original statistic, bias (mean of results minus original), and standard error (SD of results): record all three.
• ▸Interval or p-value: the confidence interval from the R results, or the observed value's position in the shuffled distribution.
• ▸Shape check: describe the histogram or boxplot of the R results.

1. 1Write the hypothesis before you touch data, in a form that names a direction and a comparison, e.g. 'price A is more profitable than the existing price B' or 'the new treatment is better than the control'. The source's pipeline starts here, and a test designed afterward cannot be called an experiment.
2. 2Choose the control deliberately. Use the standard existing treatment, or 'making no change'. The summary recommends a no-change control, so every treatment is judged against what would have happened anyway.
3. 3Fix one metric (the test statistic) before the experiment runs. You may collect several behavior metrics, but if the test must decide between A and B, name the single deciding metric in advance. Picking it afterward opens the door to researcher bias.
4. 4Randomize subjects into the groups. Each subject gets an equal chance of landing in either group, so differences in outcome come from the treatment and not from who was selected.
5. 5Run a power calculation before launch. Specify three things: the minimum effect size worth detecting, the required probability of detecting it (power, e.g. 0.8), and the significance level (alpha, e.g. 0.05). Solve for sample size, as in the statsmodels TTestIndPower example, which returned 116,602.393 for the chosen effect size.
6. 6Run the test to the planned sample size and analyze it only against the pre-set metric. Do not hunt through other metrics or segments afterward.
7. 7Gauge the role of chance with permutation or bootstrap resampling. The source says formal inference is mostly unneeded, but you must still check how much random variation could produce the observed difference.
8. 8Choose the design to fit the question. If you need 'which of several prices is best' and not 'is A different from B', consider a multi-arm bandit. If the traditional test ends inconclusive ('effect not proven'), decide in advance what action you take.

The source's coin thought experiment shows why a predefined hypothesis matters. A person claims they can flip 10 heads in a row. You challenge them, they flip 10 times and get 10 heads. The chance of that by luck is about 1 in 1,000, so you credit them with a special talent. The hypothesis came first and the test came second. Now the stadium announcer asks 20,000 people to each toss a coin 10 times and report 10 heads. The chance that somebody gets 10 heads is above 99% (1 minus the probability that nobody does). Picking that person afterward proves nothing. The same data pattern is evidence in one design and noise in the other. Applied to an A/B test: if you name the metric before running it, a win is a verified result. If you scan many metrics afterward and report the one that moved, you have run the stadium version, which the source calls data snooping.

• If you could not state the hypothesis and the single deciding metric before launch, then treat the result as exploratory (data snooping), not as confirmation.
• If you cannot name the minimum effect size you care about, then do not launch, because you cannot compute the sample size and an underpowered test will return 'effect not proven'.
• If the real question is 'which of several options is best' and not 'is A different from B', then use a multi-arm bandit design instead of a static A/B test.
• If a difference appears but is small relative to plausible chance variation, then run a permutation or bootstrap resample before acting on it.
• If a no-change option exists, then include it as the control; if no existing standard exists, say so explicitly and choose the comparison baseline before the test.

• Choosing the metric after seeing results. The tell: the reported metric was not in the original plan, or several metrics were examined and only the winner is shown.
• Underpowered tests. The tell: no sample-size calculation exists, and the outcome is 'effect not proven' because the sample was not big enough to prove a real effect.
• Data snooping and the vast search effect. The tell: many predictors or segments were tried and the 'finding' fails to reproduce.
• Missing or non-random control. The tell: the comparison group was self-selected or is the 'before' period, so you cannot separate treatment effect from other changes.
• Treating a significance test as the goal in an optimization setting. The tell: the team fixates on p-values for A vs. B when the business question is which of several options performs best, and the test ends without a decision.

• ▸Hypothesis: one sentence stating which treatment is expected to beat which, and on what outcome.
• ▸Control: the standard existing treatment or 'no change'; state exactly what the control group receives.
• ▸Single deciding metric (test statistic): the one metric that decides A vs. B, fixed before launch.
• ▸Secondary metrics: any other collected metrics, marked as descriptive only and not for the decision.
• ▸Randomization method: how subjects are assigned to groups so each has an equal chance of either.
• ▸Minimum detectable effect: the smallest effect size worth detecting.
• ▸Alpha and power: the significance level (e.g. 0.05) and the probability of detecting the effect (e.g. 0.8).
• ▸Required sample size: result of the power calculation, per group.
• ▸Chance check and inconclusive plan: the permutation/bootstrap procedure, and what action you take if the result is 'effect not proven'.

1. 1Fit the full model first as your reference point. In the passages' house example, that means OLS on the predictors PropertyType, NbrLivingUnits, SqFtFinBasement, YrBuilt, YrRenovated and NewConstruction, with categoricals expanded by pd.get_dummies(..., drop_first=True). Record P (number of variables after dummy expansion) and n (number of records).
2. 2Do not judge candidate models by training RMSE or R². Adding variables always lowers RMSE and raises R² on the training data, so both reward complexity and cannot guide selection.
3. 3Score each candidate with a metric that carries a complexity penalty. Use adjusted R² (a function of n and P), or AIC = 2P + n·log(RSS/n), where lower is better and each extra variable costs 2. Use AICc when n is small. Use BIC when you want a stronger penalty. Mallows Cp is another AIC variant.
4. 4Count complexity honestly. One categorical predictor becomes several dummy columns, and each column adds to P. Check P after encoding, not before. In linear and logistic regression, drop one dummy per factor (drop_first=True) so multicollinearity does not distort the count or the fit.
5. 5Search for the model that minimizes AIC (or maximizes adjusted R²). The passage names searching through all candidate models as one way to do it. Compare candidates on the same data and the same outcome.
6. 6If you have holdout data, assess the final candidates there. The passage says data scientists who use holdout assessment do not need to worry about the differences among AIC, BIC and Cp, because those are in-sample metrics. Let holdout error break ties.
7. 7Reduce complexity at the data level too. Binning a numeric variable into an ordered factor shrinks the complexity and size of the data and helps you find relationships early. Standardize numeric features so scale does not distort distance-based methods.
8. 8Treat smoothing and similar settings as complexity controls to tune deliberately. For example, bw_method in pandas density plots sets how smooth the curve is. Pick the simplest setting that still shows the structure.

The source's house-price regression starts with a full OLS model on six predictors: PropertyType, NbrLivingUnits, SqFtFinBasement, YrBuilt, YrRenovated and NewConstruction. PropertyType is categorical, so get_dummies with drop_first=True expands it into several columns, and P counts every one of them. Adding any variable lowers training RMSE and raises R², so those numbers cannot tell you whether YrRenovated earns its place. AIC can. Dropping a variable raises n·log(RSS/n) slightly but removes 2 from the 2P term. If the fit term rises by less than 2, the smaller model has the lower AIC and Occam's razor says to keep it. If you have holdout data, you then check the surviving model there instead of arguing about AIC versus BIC. BIC would simply penalize the extra variables harder than AIC.

• If a candidate model improves only training RMSE or R², then ignore the improvement. Those metrics always favor more variables.
• If two models fit about equally well by AIC or adjusted R², then pick the one with fewer variables (Occam's razor).
• If n is small, then use AICc instead of plain AIC.
• If you want fewer variables than AIC gives you, then use BIC, which has a stronger penalty.
• If you have holdout data, then rely on holdout error for the final choice and treat AIC, BIC and Cp as in-sample screening only.
• If you fit linear or logistic regression with one-hot dummies, then omit one dummy per factor to avoid multicollinearity.

• Choosing the model with the lowest training RMSE or highest R². The tell is that the full model always wins and no variable is ever removed.
• Miscounting P by counting a categorical predictor once when it expands into several dummy columns. The tell is an AIC penalty that looks too small for the number of coefficients in the summary.
• Leaving all one-hot dummies in a regression. The tell is multicollinearity: unstable or inferable coefficients, because one dummy can be computed from the others.
• Treating in-sample AIC/BIC/Cp as proof of predictive performance. The tell is a model that scores well on AIC but is never checked on holdout data.
• Leaving features unstandardized, so scale rather than signal drives the model. The tell is variables with values in the hundreds of thousands dominating distance calculations while ratio variables count for almost nothing.

• ▸Outcome and candidate predictors: list the outcome and every raw predictor before encoding.
• ▸n (records): enter the number of rows used in the fit.
• ▸P after encoding: count variables after dummy expansion and after dropping one dummy per factor.
• ▸Training RMSE / R²: record for reference only; never use these to choose.
• ▸Adjusted R²: record for each candidate.
• ▸AIC (2P + n·log(RSS/n)): compute for each candidate; lower is better.
• ▸Variant check: note whether you need AICc (small n) or BIC (stronger penalty).
• ▸Holdout error: enter error on held-out data for the finalists.
• ▸Decision and reason: name the simplest model that is not clearly worse, and state why.