Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

1. 1Separate parameters from hyperparameters before you search. Parameters (θ0, θ1 in the life-satisfaction model) are what training finds. Hyperparameters are the choices you fix before training: model type, architecture, regularization settings. The source's chapter listings name examples: regularization hyperparameters for decision trees, kernel choice for Kernel PCA, and hidden layers, neurons per layer, learning rate and batch size for neural networks.
2. 2Frame the search as model selection. The source defines it as choosing the type of model and fully specifying its architecture. Write down each candidate as a fully specified configuration (for example, a linear model with one input and one output) so every candidate is a concrete, comparable object.
3. 3Fix one performance measure before you compare anything. The source stresses that you need a utility or cost function to say which values perform best. Choose it first so the winner is not picked after seeing the results.
4. 4Hold data apart for judging. Train each candidate on the training data and score it on validation data, not on the training set. Overfitting means good on training data and poor generalization, so a training score cannot rank configurations. Cross-validation (the source's 'Better Evaluation Using Cross-Validation') is the alternative when you do not want to set aside a single validation set.
5. 5Run the search that fits the space. Use grid search when the space is small and you can list every combination. Use randomized search when there are many hyperparameters or wide ranges. Both appear as separate steps under 'Fine-Tune Your Model' in the source.
6. 6Rank by validation score and inspect the best candidates and their errors ('Analyze the Best Models and Their Errors'). Check whether the winner is overfitting (strong on training, weaker on validation) or underfitting (weak on both). Adjust the search range accordingly and re-run.
7. 7Consider ensemble methods as a further tuning step. The source lists them as a fine-tuning option after grid and randomized search.
8. 8Touch the test set once, at the end. Evaluate only the final chosen system on it ('Evaluate Your System on the Test Set'). Any decision made after seeing test results contaminates the estimate of generalization.

The source's running example models life satisfaction from GDP per capita using Hungary (12,240; 4.9), Korea (27,195; 5.8), France (37,675; 6.5), Australia (50,962; 7.3) and United States (55,805; 7.2). Choosing a linear model, life_satisfaction = θ0 + θ1 × GDP_per_capita, is model selection. Training finds θ0 = 4.85 and θ1 = 4.91 × 10^-5. For Cyprus (22,587), the prediction is 4.85 + 22,587 × 4.91 × 10^-5 ≈ 5.96. Now treat model complexity as the choice. The source shows a high-degree polynomial that fits the training data much better than the line yet is untrustworthy, because it overfits. Judged on training data, the polynomial would win every time. Judged on held-out countries, the simple linear model is the one that earns trust. The tuning rule is to rank candidates by validation performance, not training fit. The source does not report numeric validation errors for this example, so the comparison here is qualitative.

• If a candidate scores much better on training data than on validation data, then it is overfitting. Choose a simpler configuration or stronger regularization, and do not select it on training score.
• If a candidate scores poorly on both training and validation data, then it is underfitting. Widen the search toward more capacity or better features rather than tuning finer.
• If you have few hyperparameters with a short list of values, then use grid search. If there are many hyperparameters or continuous ranges, then use randomized search.
• If a single validation split feels too noisy or wastes scarce data, then use cross-validation to score each configuration.
• If you have already looked at test-set results and then changed the model, then the test score no longer estimates generalization. Treat it as spent and obtain fresh held-out data.

• Selecting on training performance. The tell is that the most complex model (such as a high-degree polynomial) always wins and then disappoints on new data.
• Reusing the test set during tuning. The tell is that you keep iterating after seeing the test score, and the launch results fall short of it.
• Confusing parameters with hyperparameters. The tell is that you try to 'search' over θ values that the training algorithm already finds, or you leave architecture choices untuned.
• Searching a range that was never justified. The tell is that the best value sits at the edge of the grid, meaning the true optimum lies outside what you tried.
• Moving on after the search without error analysis. The tell is that you ship the top-scoring configuration without inspecting where it fails, missing data problems such as nonrepresentative training data or irrelevant features that no hyperparameter will fix.

• ▸Model type: name the algorithm family being tuned.
• ▸Hyperparameters and ranges: list each hyperparameter with the values or range to try.
• ▸Performance measure: state the single metric used to rank candidates, fixed before the search.
• ▸Evaluation method: validation split or cross-validation, and which data is held out.
• ▸Search method: grid search or randomized search, and why it fits the space size.
• ▸Best configuration: record the top-ranked setting and its validation score.
• ▸Training vs. validation gap: note the training score next to the validation score to flag over- or underfitting.
• ▸Edge-of-range check: say whether the best value sits at a boundary and what range to try next.
• ▸Final test result: record the one-time test-set score after the configuration is frozen.

1. 1Put the test set aside before any exploration. Work on a copy of the training set (housing = strat_train_set.copy()) and sample an exploration set if the training set is very large. Do not touch the test set until you are ready to launch a model you are confident about.
2. 2Carve out a validation set from the training set only. Train candidate models and hyperparameter values on the reduced training set (full training set minus validation set) and pick the winner on the validation set.
3. 3Replace a single validation split with K-fold cross-validation when one split is too noisy or too costly. cross_val_score(model, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10) trains and evaluates 10 times, each time holding out a different fold and training on the other 9.
4. 4Handle the sign convention. Scikit-Learn expects a utility function (greater is better), so the scoring is the negative MSE. Compute np.sqrt(-scores) to recover RMSE per fold, then read the mean and the spread across folds.
5. 5Never trust the training-set score. A Decision Tree that scores RMSE 0.0 on the training set has most likely badly overfit. Judge it only on cross-validated scores.
6. 6Run hyperparameter search inside the validation loop. Use GridSearchCV for few combinations. Use RandomizedSearchCV when the space is large, since it lets you set the compute budget through the iteration count. Treat data-preparation choices (e.g., add_bedrooms_per_room, outlier handling, feature selection) as hyperparameters in the same search.
7. 7Retrain the chosen model on the full training set, including the validation set. Evaluate it once on the test set to get the generalization-error estimate.
8. 8If training data differs from production data (e.g., web flower photos vs. mobile-app photos), make sure the validation and test sets contain only representative production-like data. Use a train-dev set to separate overfitting from data mismatch.

The source describes a regularization search that goes wrong. You train 100 models with 100 values of the regularization hyperparameter and pick the one with the lowest error on the test set: 5%. In production it produces 15% errors. The cause is that you measured generalization error many times on the test set and adapted the model to that particular set. The fix is holdout validation: select on a validation set, retrain the winner on the full training set, then touch the test set once. In the housing project, the Decision Tree scores RMSE 0.0 on the training set, which signals overfitting rather than perfection. 10-fold cross-validation gives the honest number. The grid search then finds max_features=8 and n_estimators=30 with RMSE 49,682, slightly better than the default-hyperparameter score of 50,182. That score still comes from cross-validation, so the test set stays untouched until the final evaluation.

• If you have measured the same test set more than once while choosing among models, then the test estimate is contaminated. Take your choices from a validation set or cross-validation instead.
• If the validation set is too small, then model comparisons are imprecise and you may pick a suboptimal model. If it is too large, then candidates train on a much smaller set than the final model, like picking the fastest sprinter for a marathon. In either case, move to repeated cross-validation with many small validation sets.
• If training time is the constraint, then remember that cross-validation multiplies training time by the number of folds. Use a single holdout split or RandomizedSearchCV with a capped iteration count.
• If the hyperparameter space is large, then use RandomizedSearchCV: 1,000 iterations explores 1,000 values per hyperparameter, and you control the budget. If you are exploring only a few combinations, then GridSearchCV is fine.
• If your training data is plentiful but unrepresentative of production, then build validation and test sets from representative data only and hold out a train-dev set to diagnose mismatch.

• Tuning on the test set. The tell: a low test error (e.g., 5%) that becomes a much higher production error (e.g., 15%).
• Trusting the training-set score. The tell: an RMSE of 0.0 or near zero on a flexible model such as an unpruned Decision Tree.
• Misreading cross-validation scores. The tell: negative MSE values treated as errors, or RMSE computed without negating the scores first.
• Choosing the validation size carelessly. The tell: the best model flips between runs (set too small), or candidates are compared on a training set far smaller than the final one (set too large).
• Validating on non-representative data. The tell: strong validation results on abundant web data (e.g., downloaded flower photos) that fail on the real production inputs (e.g., mobile-app photos).

• ▸Test set locked away (date/method): record when and how the test set was separated and who may access it.
• ▸Exploration data: state whether you work on a copy of the training set or a sample of it.
• ▸Validation scheme: holdout, K-fold (give K), or repeated CV, and why it fits your data size and time budget.
• ▸Scoring metric and sign: name the metric and note whether Scikit-Learn returns it negated.
• ▸Training-set score vs. CV mean ± spread: record both to expose overfitting.
• ▸Search method and budget: GridSearchCV or RandomizedSearchCV, with the number of combinations or iterations.
• ▸Preparation steps treated as hyperparameters: list the options included in the search.
• ▸Representativeness check: say how validation/test data match production data and whether a train-dev set is used.
• ▸Final test-set evaluation (one time only): record the single test score and the date it was run.

1. 1Measure the model on the training set first. Compute the same error metric (the passages use RMSE via mean_squared_error) on the data it was fit on. Training error tells you whether the model can even learn the structure.
2. 2Read the training error against the scale of the target. If a typical error is large relative to the target's normal range, treat it as underfitting. The passages' example: a $68,628 error against median_house_values mostly between $120,000 and $265,000.
3. 3Check for the opposite signature. A training error near zero (a Decision Tree returning 0.0) is a warning, not a win. The passages ask 'Could this model really be absolutely perfect?' and the answer is that it probably overfits the training data.
4. 4Confirm overfitting on data the model has not seen. The definition is good on training, poor on new data, so you need a held-out set or cross-validation, as the Testing and Validating and cross-validation sections cover. Compare training error with held-out error.
5. 5If underfitting, apply the three fixes in order of cost: select a more powerful model with more parameters, feed better features (feature engineering), or reduce constraints such as the regularization hyperparameter. Rule out any option that does not apply, e.g. an unregularized model has no regularization to reduce.
6. 6If overfitting, constrain the model with regularization (ℓ1, ℓ2, dropout, max-norm in the neural-net chapter) and check the data. Noisy or too-small training sets and uninformative features, such as a country's name, invite the model to fit noise.
7. 7Treat the regularization strength as a hyperparameter set before training and held constant during it. Tune it by validation. A very large value gives an almost flat model (slope near zero) that will not overfit but is unlikely to find a good solution. That is underfitting you caused.
8. 8Re-measure after each change, one change at a time, so you know which move shifted the gap between training and held-out error.

The passages' housing project shows both states in sequence. A Linear Regression model gets an RMSE of 68,628.198 on the whole training set. Most districts' median_housing_values fall between $120,000 and $265,000, and the first prediction is off by close to 40%. Error is poor on the training data itself, so this is underfitting: the features lack information or the model is too weak. The model is not regularized, so reducing constraints is ruled out. Next, a DecisionTreeRegressor, a more powerful model, is trained, and its training RMSE is 0.0. A perfect score is implausible, so the tree has probably memorized the training data. Training error alone cannot settle it, so you evaluate on data not used for fitting. Same data, same metric, two opposite failures, and the fix is different for each.

• If training error is already large relative to the target's range, then treat it as underfitting and add model power, better features, or less regularization. More data will not help first.
• If training error is near zero (like RMSE 0.0), then do not trust it. Evaluate on held-out data or cross-validation before concluding anything.
• If training error is low but held-out error is high, then overfitting: simplify the model, add regularization, or clean and enlarge the training set.
• If the model is unregularized and underfits, then the reduce-constraints option is unavailable. Choose a more complex model or add features such as the log of population.
• If you raise the regularization hyperparameter and both errors rise, then you have pushed from overfitting into underfitting. Back it off.

• Judging a model by training error only. The tell: a perfect score like 0.0 that nobody checked on unseen data.
• Fixing underfitting with more regularization, or overfitting with a more powerful model. The tell: the error gets worse after the change, because you moved the wrong way along the bias/variance trade-off.
• Setting the regularization hyperparameter too high. The tell: an almost flat model with slope near zero and poor error on both sets.
• Feeding many uninformative attributes to a complex model. The tell: a model that finds patterns in noise (e.g. the country's name) and does well in training but fails on new instances.
• Tuning the hyperparameter during training or by the learning algorithm's own updates. The tell: confusion over a value that should be fixed beforehand, since a hyperparameter is not affected by the learning algorithm.

• ▸Model and version: name the algorithm and key settings tested.
• ▸Error metric: state the metric used (e.g. RMSE) and the target's normal value range.
• ▸Training error: record the value on the training set.
• ▸Held-out/cross-validation error: record the value on data not used for fitting.
• ▸Gap and scale check: note the difference between the two errors and the error relative to the target range.
• ▸Diagnosis: underfitting, overfitting, or acceptable, with the evidence that decided it.
• ▸Regularization hyperparameter: record the current value and whether it was set before training.
• ▸Action taken: one change from the fix list (more powerful model, better features, less or more regularization, cleaner or larger data).
• ▸Re-measured result: record both errors after the change and the decision.

1. 1Define the performance measure P before training. For the spam filter, the passages name accuracy (ratio of correctly classified emails). Only commit to one metric after checking it fits the task: the passages list RMSE for regression, and precision/recall, the confusion matrix, and the ROC curve for classification.
2. 2Create the test set first, before exploring the data in depth. Pick instances randomly, typically 20% of the dataset or less if it is very large (1% of 10 million instances is 100,000). Set it aside and never look at it. This prevents data snooping bias.
3. 3Make the split reproducible. The passage's split_train_test shuffles with np.random.permutation, which gives a different test set on every run. Fix the random seed so the same instances stay in the test set, or the model eventually sees the whole dataset.
4. 4Carve a validation (dev) set out of the training set. Train every candidate model and every hyperparameter value (for example, 100 regularization values) on the reduced training set, and score each on the validation set. Never touch the test set during selection.
5. 5Select the best candidate on the validation set. Retrain it on the full training set, including the validation set, to get the final model.
6. 6Evaluate the final model once on the test set. The result is your estimate of the generalization error (out-of-sample error).
7. 7Read the gap between training error and generalization error. Low training error with high generalization error means overfitting. If both are poor, the model is too simple and underfits.
8. 8After launch, keep measuring. Monitor live performance, monitor the input data for abnormal values (an anomaly detection algorithm can do this), and be ready to switch learning off and revert to a previously working state if performance drops.

The passages describe this failure. You are choosing a regularization hyperparameter for a linear model. You train 100 models with 100 different values and score each on the test set. The best one shows just 5% error. You launch it, and production returns 15% errors. The cause is that you measured generalization error many times on the same test set and tuned the model to that particular set, so the 5% figure was optimistic. The fix is holdout validation. Split a validation set off the training set, train the 100 models on the reduced training set, and pick the winner on the validation set. Retrain that winner on the full training set. Then touch the test set once, to get an honest estimate. If that estimate is near 15% rather than 5%, you learn it before users do.

• If you are about to score a model on the test set for the second time to make a choice, stop. Use a validation set for selection and keep the test set for one final estimate.
• If training error is low but generalization error is high, then treat it as overfitting: simplify the model, add regularization, or get more or better data.
• If the dataset is very large (for example, 10 million instances), then a hold-out of 1% (100,000 instances) is enough for the test set. Otherwise default to about 80/20.
• If the validation set is too small, then model evaluations will be imprecise and you may pick a suboptimal model. Enlarge it before trusting the ranking.
• If production performance drops or inputs look abnormal, then switch learning off and revert to the last working state rather than letting the system keep adapting.

• Reporting training-set performance as success. The tell: a strong training score with no held-out evaluation. The passages say good training performance is insufficient; the goal is new instances.
• Data snooping bias. The tell: you inspected the test set and chose a model type because of a pattern you saw there, and the estimate came out too optimistic.
• Repeatedly tuning on the test set. The tell: a test error like 5% that becomes 15% in production, because the hyperparameters fit that particular set.
• Non-reproducible splits. The tell: rerunning the split code yields a different test set each time, so over repeated runs the model sees the whole dataset.
• Launching and not monitoring. The tell: no alerting on live accuracy or input anomalies, and no way to revert, so a performance drop goes unnoticed until users complain.

• ▸Task and performance measure P: name the task and the metric (accuracy, RMSE, precision/recall, F1, ROC AUC) and why it fits.
• ▸Dataset size: total instances, which sets the split ratio.
• ▸Test set fraction and seed: for example 20% or less if large, with the fixed random seed or reproducible method used.
• ▸Test set locked on (date): record when it was set aside and confirm it was not inspected.
• ▸Validation set size: how much of the training set you hold out for model selection.
• ▸Candidates and hyperparameters tried: list model types and the hyperparameter values compared on the validation set.
• ▸Selected model and validation score: the winner and its validation result.
• ▸Final test-set generalization error: the single estimate from one evaluation of the final model.
• ▸Training error vs generalization error gap: note whether it signals overfitting or underfitting.
• ▸Production monitoring and rollback: live metric tracked, input anomaly check, and the trigger and steps for switching learning off and reverting.

1. 1Deploy the model as a web service. Save the trained model with joblib, upload it to Google Cloud Storage, create a new model version in Google Cloud AI Platform pointing at the GCS file, and let the service handle load balancing and scaling. The web app sends JSON requests with the input data (e.g., a district) and receives JSON predictions.
2. 2Write monitoring code that checks live performance at regular intervals and triggers alerts when it drops. Cover two patterns: a steep drop (likely a broken infrastructure component) and a gentle decay that can go unnoticed for a long time.
3. 3Pick the performance signal. If downstream metrics reveal model quality (e.g., a recommender system's results), use them. If not, send human raters a sample of the model's outputs, weighted toward cases where the model was not so sure. Raters can be experts, crowdsourced workers (e.g., Amazon Mechanical Turk), or the users themselves via surveys or repurposed captchas.
4. 4Define the failure process up front: what to do when performance drops and how to prepare for it. The source warns this is often much more work than building and training the model.
5. 5Monitor input data quality separately from model performance. A malfunctioning sensor sending random values or another team's output going stale can degrade performance slightly, and the cause may take a while to surface.
6. 6Automate the refresh cycle: collect and label fresh data regularly; run a script that trains and fine-tunes hyperparameters on a schedule (daily or weekly, per your needs); run a second script that evaluates the new and previous models on the updated test set.
7. 7Gate deployment on that comparison: deploy the new model only if performance has not decreased. If it did decrease, investigate why before anything ships.
8. 8Keep evaluation slices beyond the full test set, such as a subset of only the most recent data or specific input kinds (e.g., inland versus near-ocean districts), to see strengths and weaknesses.

Take the chapter's housing-price model, deployed on Google Cloud AI Platform: the model file sits in GCS, and a web app sends JSON for a district and gets back a predicted median price. The previous process had experts estimating prices manually, often off by more than 20%, which gives you a reference for performance. You write monitoring code that checks live predictions at regular intervals and alerts on a drop. Because the true median price is sometimes found later, you can compare predictions to it; where it isn't, you send a sample to human raters. A scheduled script collects fresh data, labels it, retrains, and tunes hyperparameters. A second script scores the new and old models on the updated test set and deploys the new one only if performance has not decreased. Separately, you check input quality, since another team's stale output feeding the pipeline would quietly erode accuracy. You also keep slices such as recent districts and inland versus near-ocean districts to see where the model weakens.

• If performance drops steeply, suspect a broken infrastructure component first; if it declines gently, suspect model rot from a changing world and plan retraining.
• If downstream metrics can reveal model quality, use them; if not, set up human raters and prioritize samples the model was unsure about.
• If the new model does not match or beat the previous model on the updated test set, do not deploy it and investigate why.
• If performance degrades slightly, check input data quality (malfunctioning sensors, stale upstream output) before blaming the model.
• If the data keeps evolving, automate data collection, labeling, training, and evaluation rather than retraining by hand.

• Treating deployment as the finish line: no monitoring code exists, and the tell is that nobody learns of a drop until users complain.
• Watching only for sudden breaks: a gentle decay from model rot goes unnoticed for a long time, and the tell is a slowly worsening metric no alert caught.
• Letting a broken pipeline component go unnoticed: downstream components keep running on its last output, so data gets stale and performance falls with no error raised.
• Ignoring input quality: a faulty sensor or stale upstream team output causes slight degradation, and the tell is a model that looks fine but receives bad signals.
• Retraining on a schedule and deploying automatically without comparing to the previous model, so a worse model reaches production; the tell is no evaluation on the updated test set before release.

• ▸Serving setup: where the model file lives and how requests/responses (JSON) flow.
• ▸Performance signal: downstream metric, human raters, or both, and who the raters are.
• ▸Check interval and alert trigger: how often performance is checked and what drop fires an alert.
• ▸Input data quality checks: which inputs and upstream sources you verify (e.g., sensors, other teams' outputs).
• ▸Failure process: the steps taken when an alert fires.
• ▸Refresh schedule: how often fresh data is collected, labeled, and the model retrained (daily/weekly).
• ▸Deployment gate: comparison of new vs. previous model on the updated test set, and the rule for deploying.
• ▸Evaluation slices: recent-data and special-case subsets (e.g., inland vs. near-ocean) to track.
• ▸Investigation owner: who looks into why a new model performed worse.

1. 1Classify the problem first. The book sets two rough bands: very simple problems typically need thousands of examples, and complex problems such as image or speech recognition may need millions. Write down which band you are in before you collect anything.
2. 2Count what you actually have, as training instances only. Hold-out data (test set, and any validation or train-dev sets) does not count toward training. Compare that count to your band.
3. 3Check whether you can reuse parts of an existing model. The book's 'millions of examples' figure carries the exception 'unless you can reuse parts of an existing model' (transfer learning). Settle this before you budget for large-scale data collection.
4. 4Before switching to a fancier algorithm, weigh more data against a better algorithm. Banko and Brill (2001) found that very different algorithms, including fairly simple ones, performed almost identically well on a complex problem once given enough data (Norvig et al., 'The Unreasonable Effectiveness of Data'). For complex problems, data often matters more than the algorithm.
5. 5Start with simpler techniques such as Random Forests and Ensemble methods. The book says most problems are solved quite well by them. Save Deep Learning for complex problems where you have enough data, computing power, and patience.
6. 6If performance is poor, test whether quantity is the cause, not just assume it. The book lists learning curves (Chapter 4) as the tool for this, and separates quantity from three neighboring data problems: nonrepresentative data, poor-quality data, and irrelevant features. More rows do not fix those.
7. 7Treat the count as a claim about coverage, not just size. Nonrepresentative training data is a separate challenge from insufficient quantity, so check that your examples cover the cases the model will see.

The book contrasts a toddler with a learning algorithm. A toddler needs you to point at an apple and say 'apple' a few times, then recognizes apples in all sorts of colors and shapes. A typical ML algorithm cannot do that. Even a very simple problem usually needs thousands of examples, and image or speech recognition may need millions. The evidence for data over algorithm is Banko and Brill's 2001 Microsoft paper. They compared very different ML algorithms, including fairly simple ones, on a complex language problem (the book's footnote gives choosing between 'to,' 'two,' and 'too' from context as the kind of task). All the algorithms performed almost identically well once the data was large enough. The practical reading: if you have a complex problem and a mediocre result, ask whether more data would help before you swap in a more elaborate model. The same logic explains why neural networks are now more useful. The book cites 'a huge quantity of data available' as one reason, and notes that ANNs frequently outperform other techniques on very large, complex problems.

• If your problem is simple (a toddler could learn it from a few examples, but the algorithm cannot), then plan for thousands of examples, not dozens.
• If your problem is complex, such as image or speech recognition, then plan for millions of examples, or look for an existing model whose parts you can reuse.
• If you are choosing between a more sophisticated algorithm and more data on a complex problem, then lean toward getting more data. Very different algorithms performed almost identically given enough of it.
• If you do not have enough data, computing power, and patience, then stay with simpler techniques such as Random Forests and Ensemble methods instead of Deep Learning.
• If you add data and results do not improve, then check representativeness, quality, and feature relevance. The book lists each as a separate challenge from quantity.

• Assuming ML learns like a toddler: you train on a few dozen examples and expect general recognition. The tell is that the model only works on the cases it was shown.
• Counting hold-out data as training data: you quote a dataset size that includes the test and validation sets. The tell is that the training instances are fewer than your headline number.
• Chasing a fancier algorithm on a complex problem when the real limit is data. The tell is that very different models give near-identical results, as in Banko and Brill.
• Collecting millions of examples when you could have reused parts of an existing model. The tell is that you budget for large-scale data collection without ever asking whether transfer learning applies.
• Treating volume as a cure for bad data: you add more rows that are nonrepresentative, noisy, or full of irrelevant features. The tell is that performance on new cases does not improve with size.

• ▸Problem statement: one line describing what the model must predict or recognize.
• ▸Complexity band: simple (thousands of examples) or complex, e.g. image or speech recognition (millions)? Pick one and say why.
• ▸Usable training instances: count of examples in the training set only, excluding test, validation, and train-dev data.
• ▸Gap to band: the band's order of magnitude minus your training count.
• ▸Reuse option: does an existing model exist whose parts you can reuse (transfer learning)? Yes or no, and which.
• ▸Baseline technique: the simpler method you will try first (e.g. Random Forest or an ensemble) and its result.
• ▸Data-versus-algorithm call: if results are weak, will you add data or change the algorithm? State the reason.
• ▸Adjacent-risk check: are the data representative, clean, and limited to relevant features? Note any doubt.
• ▸Next action and date: the one step you will take (collect, reuse, or stop) and when.

1. 1Name the population first. Write down the cases the model must generalize to (the new instances at inference time), then compare your training set against that description before you touch an algorithm. The source frames the failure as 'bad algorithm' versus 'bad data'; check the data first.
2. 2Audit how the sample was collected. Ask who or what could not enter the set. Look for sampling bias in the collection channel (a search engine that favors popular items) and for nonresponse bias (people who did not answer). Record each excluded group.
3. 3Scan for errors, outliers, and noise. Flag instances that are clearly outliers and decide per case: discard them, or fix the error manually. Poor-quality measurements make the underlying pattern harder to detect.
4. 4Quantify missing values per feature (for example, 5% of customers did not specify age). Choose one of four options and write down why: ignore the attribute altogether, ignore the affected instances, fill in the missing values (e.g., with the median age), or train one model with the feature and one without.
5. 5Prune irrelevant features. Run feature engineering: feature selection (pick the most useful features among existing ones), feature extraction (combine existing features into a more useful one; dimensionality reduction can help), and creating new features by gathering new data.
6. 6Check for patterns that only exist by chance. Ask of any attribute, 'Is there a plausible reason this predicts the target?' An uninformative attribute such as a country's name lets a complex model latch onto noise.
7. 7Match model complexity to data quality and quantity. If the training set is noisy or small, simplify the model (fewer parameters, fewer attributes, or regularization), gather more data, or reduce noise by fixing errors and removing outliers.
8. 8Re-evaluate after each cleaning change on held-out data, not on the training set. Better training fit proves nothing about generalization.

The source gives a life-satisfaction model trained on country-level data. Suppose you add many extra attributes, including the country's name, and fit a complex model. It finds that every training country with a 'w' in its name has life satisfaction above 7: New Zealand (7.3), Norway (7.4), Sweden (7.2), Switzerland (7.5). The rule is perfect on the training set and meaningless elsewhere. Would you trust it for Rwanda or Zimbabwe? The pattern arose by pure chance, and the model cannot distinguish a real pattern from noise. The fix follows the data-quality steps: drop the irrelevant attribute (country name), remove outliers or fix errors, keep only features with a plausible link to life satisfaction (the source suggests candidates like employment rate, health, air pollution), and simplify or regularize the model. The same logic applies to representativeness. A funk-music video classifier trained on YouTube search results for 'funk music' inherits a bias toward popular artists, and if you live in Brazil it returns many 'funk carioca' videos, which sound nothing like James Brown. The training set misdescribes the population the model must serve.

• If the training set cannot be shown to mirror the cases the model will see, then treat it as nonrepresentative and fix the sampling before tuning the model.
• If a group could not or did not respond or enter (nonresponse), then assume bias and either recruit that group or state the model's limits.
• If instances are clearly outliers, then discard them or fix the errors manually.
• If a feature is missing for some instances, then pick one of four options explicitly: drop the attribute, drop the instances, impute (e.g., the median), or train models with and without the feature.
• If a complex model fits training data far better than a simple one but you have noisy or small data, then suspect it is fitting noise; reduce noise, remove irrelevant features, add data, or constrain the model.

• Convenient sampling mistaken for a representative sample (e.g., using search results as 'all' funk videos). The tell: the training set over-represents popular or easy-to-reach cases and the model fails on the rest.
• Ignoring nonresponse. The tell: your sample consists only of people who chose to answer, as in the Literary Digest poll, and key groups are silently absent.
• Skipping cleaning because modeling feels more productive. The tell: outliers and measurement errors remain and the system struggles to find the underlying pattern, even though most data scientists spend a significant part of their time on cleaning.
• Feeding in every available attribute. The tell: a complex model finds spurious rules like the 'w in the country name' pattern that cannot generalize.
• Imputing or dropping missing values with no recorded rationale. The tell: a feature missing for a share of customers (e.g., 5% without age) is handled inconsistently, so the choice is not reproducible or tested.

• ▸Target population: describe the cases the model must generalize to at inference.
• ▸Collection method: how the training data was gathered and by which channel.
• ▸Excluded or underrepresented groups: who or what could not enter (sampling bias, nonresponse bias).
• ▸Outliers/errors found: count them and mark each as discard or fix manually.
• ▸Missing-value table: feature, percent missing, chosen option (ignore attribute / ignore instances / fill in / two models).
• ▸Irrelevant feature candidates: list attributes with no plausible link to the target and say drop or keep.
• ▸Feature engineering actions: selection, extraction, or new data to gather.
• ▸Complexity vs data check: model type, number of parameters, data size and noise level, and the simplify/regularize/gather decision.
• ▸Held-out validation result: score on data not used in training after cleaning.

1. 1Diagnose first. Check the error on the training examples themselves. If predictions are inaccurate even there, the model is underfitting. The source lists three fixes: a more powerful model, better features, or fewer constraints (e.g., a lower regularization hyperparameter). Choose feature engineering only when the evidence points to features.
2. 2Inventory every available attribute and mark each as relevant, doubtful or irrelevant. The source's rule is 'garbage in, garbage out': the system learns only if the data has enough relevant features and not too many irrelevant ones.
3. 3Select. Keep the most useful features among those you already have and drop the irrelevant ones. This is the first of the three feature-engineering steps.
4. 4Clean what you keep before transforming it. Discard or manually fix clear outliers. For missing values, choose one of four options: ignore the attribute, ignore the affected instances, fill in the values (e.g., with the median), or train one model with the feature and one without.
5. 5Extract. Combine existing features into a more useful one. This is the 'Experimenting with Attribute Combinations' step. Dimensionality reduction algorithms can also produce combined features.
6. 6Create. If no combination of existing columns carries the signal, gather new data and add it as a new feature.
7. 7Make the features algorithm-ready. Encode text and categorical attributes (one-hot vectors or embeddings), and apply feature scaling. Package every step in a transformer or pipeline so the same recipe runs on training and new data.
8. 8Validate each change with cross-validation, and check feature importance where the model provides it (e.g., Random Forests). Keep a feature only if it improves measured performance. Evaluate on the test set once, at the end.

The source's own case is a linear model of life satisfaction. It underfits because reality is more complex than the model, so its predictions are inaccurate even on the training examples. Adding model power or loosening regularization are two ways out. The third is to feed better features, which is feature engineering. You inventory the attributes and drop any that are irrelevant. You then ask whether combining two existing attributes produces a more informative one. If nothing combines usefully, you gather new data to add a feature. Take the source's missing-data case as well: if 5% of customers did not specify their age, you decide whether to drop the attribute, drop those customers, fill the gap with the median age, or train two models, one with the feature and one without. After each change you compare cross-validated performance. The change stays only if the training-set error falls and the cross-validated score holds up.

• If error is high on the training set itself, then treat it as underfitting and work on features, model power or regularization before collecting more of the same data.
• If a feature adds noise and no signal, then drop it. Too many irrelevant features is a named failure.
• If a feature is missing for a small share of instances (the source's example is 5% of customers missing age), then choose deliberately among ignoring the attribute, ignoring the instances, filling with the median, or training with and without it.
• If no combination of existing attributes carries the needed information, then gather new data and create a new feature instead of reshuffling old ones.
• If two features sit on very different numeric ranges, then apply feature scaling inside a pipeline so training and new data are transformed identically.

• Adding features indiscriminately. The tell: the model has more inputs but no better validation score, which is the source's 'irrelevant features' problem.
• Treating a feature problem as a model problem. The tell: you keep swapping algorithms while training-set error stays high, when better features were one of the three listed fixes for underfitting.
• Fixing missing values or outliers by habit instead of by decision. The tell: you cannot say why you filled with the median instead of dropping the instances or the attribute.
• Preprocessing training and new data differently. The tell: good validation numbers but poor results on fresh data, because the scaling or encoding was not packaged in a pipeline or transformer.
• Judging a feature by training performance alone. The tell: a feature raises training accuracy but not cross-validated accuracy. Check with cross-validation and the test set.

• ▸Symptom: record the training-set error and the validation error, and state whether this is underfitting.
• ▸Candidate feature: name the existing attribute, the combination of attributes, or the new data you would gather.
• ▸Type of change: select, extract (combine), or create (new data).
• ▸Hypothesis: one line on why this feature should carry signal.
• ▸Data cleaning decision: record how you handle outliers and missing values (ignore attribute, ignore instances, fill, or two models).
• ▸Encoding and scaling: note the one-hot or embedding choice and the scaling applied, and confirm both live in the pipeline.
• ▸Cross-validated score before: record the baseline.
• ▸Cross-validated score after: record the result with the change.
• ▸Keep or drop: decide, and note the feature-importance reading if available.

1. 1Start with the simplest model that fits the problem (the book's own ladder starts at Linear Regression) and treat every added parameter as a cost you must justify with data. Complexity is relative to data volume: the passages note that simple problems typically need thousands of examples and complex ones like image or speech recognition may need millions.
2. 2Count effective parameters before you train. Write down the weights, layers and neurons per layer, polynomial degree, tree depth, or kernel settings, then compare that count to the number of training examples. The Hopfield example below shows how fast the count outruns what a model can use.
3. 3Hold out data before you tune anything. Split into training and test sets, and use cross-validation (the 'Better Evaluation Using Cross-Validation' step) for model selection so the test set is touched once, at the end.
4. 4Diagnose with two numbers, training error and validation error. Low on training but much worse on validation points to overfitting (too much complexity). Poor on both points to underfitting (too little). The book covers Learning Curves and the Overfitting/Underfitting sections for this exact diagnosis.
5. 5Move one complexity knob at a time. Examples named in the passages: polynomial degree in Polynomial Regression, number of hidden layers and neurons per hidden layer in neural nets, the regularization hyperparameters of a Decision Tree, and the number of clusters in a Gaussian mixture. Re-measure validation error after each move.
6. 6If you are overfitting, constrain before you shrink. Apply regularization (Ridge Regression and the other Regularized Linear Models, tree regularization hyperparameters) or add training data. If you are underfitting, add capacity or better features, and relax regularization.
7. 7Search the knob systematically with Grid Search or Randomized Search over the complexity hyperparameters, scored by cross-validation, not by training fit.
8. 8Before paying for a more complex algorithm, ask whether more data would do the work. Banko and Brill found very different algorithms, including simple ones, performed almost identically given enough data. Check how costly extra data is, because small and medium datasets are still common and extra data is not always cheap.

The passages give a clear case of complexity out of proportion to what the model delivers: the Hopfield net. To classify 28 × 28-pixel images you need 784 fully connected neurons, which means 306,936 weights. The network's memory capacity is only roughly 14% of the number of neurons, so it learns about 110 different characters (14% of 784). That works out to roughly 2,800 weights per learnable character, which the source sums up as 'a lot of parameters for such a small memory.' It also fails in a second way: some patterns that were not in the training set end up with low energy, so the network sometimes settles on spurious patterns it never learned. The lesson is to count parameters against what the model actually retains and generalizes. Raw parameter count is a poor proxy for useful capacity, so judge capacity by held-out behavior, not by size.

• If training performance is strong but validation performance is clearly worse, then treat it as overfitting: regularize, simplify, or add data before adding capacity.
• If both training and validation performance are poor, then treat it as underfitting: raise capacity (degree, layers, neurons, depth), improve features, or reduce regularization.
• If the model's parameter count is large relative to the examples or to what it can retain (as with the Hopfield net), then reduce or constrain the model instead of scaling it.
• If different algorithms score nearly the same on your task, then spend effort on getting more or better data (the Banko and Brill finding) rather than on algorithm swaps, unless data is expensive or scarce.
• If you are choosing among complexity settings, then pick by cross-validated score and keep the test set for one final check.

• Judging complexity by training fit alone: the model looks excellent in-sample and fails on new data. The tell is a large gap between training and validation scores.
• Tuning complexity against the test set, which leaks the test set into model selection. The tell is test results that look great but fall apart after launch.
• Assuming bigger is better and stacking layers, neurons, or polynomial degree without checking returns. The tell is parameters growing while validation error stays flat or worsens (the Hopfield net: 306,936 weights for about 110 characters).
• Fixing underfitting with more data alone. The tell is that both curves stay poor however many examples you add, which signals the model is too simple.
• Ignoring spurious or unlearned behavior in over-capacity models. The tell is confident outputs on patterns that were never in the training set.

• ▸Model and version: name the algorithm and its main complexity knobs (degree, depth, layers, neurons, kernel, clusters).
• ▸Effective parameter count: write the number of weights or parameters, or your best estimate.
• ▸Training examples: record the number of examples available for fitting.
• ▸Parameters-to-examples ratio: divide parameters by examples and note whether it looks too high.
• ▸Training score: record performance on the training set.
• ▸Validation score (cross-validated): record performance on held-out folds, and never use the test set here.
• ▸Diagnosis: label as overfitting, underfitting, or balanced based on the gap and absolute levels.
• ▸Next action and single knob changed: record one change (regularize, add capacity, add data) and the hyperparameter it touches.
• ▸Re-measured validation score: record the result after the change and whether to keep it.

1. 1Diagnose first: confirm overfitting before adding constraint. The passages define overfitting as a model too complex relative to the amount and noisiness of the training data. Compare training performance with performance on data the model did not see. A big gap points to overfitting; poor performance on both points to underfitting.
2. 2Try the cheaper fixes in parallel. The passages list alternatives to constraining: select a model with fewer parameters (a linear model rather than a high-degree polynomial), reduce the number of attributes, gather more training data, and reduce noise (fix data errors, remove outliers). Regularize what remains.
3. 3Pick the constraint form that fits the model. For linear models and Logistic Regression use ℓ1 or ℓ2 penalties. For neural networks the passages list ℓ1/ℓ2 regularization, dropout, MC dropout, and max-norm regularization. For decision trees they point to the regularization hyperparameters. Early stopping is also named as a form of constraint.
4. 4Locate the correct strength knob and read its direction. Most linear models use alpha, where larger means more regularization. Scikit-Learn's LogisticRegression uses C, the inverse of alpha, where a higher C means less regularization. It also adds an ℓ2 penalty by default, so you are never at zero constraint unless you change it.
5. 5Set the value before training and hold it fixed. A hyperparameter belongs to the learning algorithm, not the model. The algorithm does not change it, so it must be set prior to training and stays constant during training. Treat each candidate value as a separate training run.
6. 6Sweep the strength across a wide range and compare candidates on held-out data, not training data. Training fit will always worsen as you constrain harder (the regularized model 'does not fit the training data as well'). You are looking for the value where unseen-data performance is best.
7. 7Check both extremes. A very large value gives an almost flat model (slope close to zero) that will not overfit but is unlikely to find a good solution. Too little constraint leaves the original overfit. Pick the setting between them, and re-tune whenever the data volume or noise level changes.

The passages' own case is the life-satisfaction linear model with two parameters, θ0 (height) and θ1 (slope), which gives two degrees of freedom. Trained on only some countries, the model picks up a spurious pattern: it ranks countries like Switzerland (7.5) in a way that holds by pure chance, and you cannot be confident it extends to Rwanda or Zimbabwe. If you force θ1 = 0, the model has one degree of freedom and ends up around the mean. If you let θ1 vary but keep it small, the model has somewhere between one and two degrees of freedom. In Figure 1-23, the regularized model (solid line) has a smaller slope than the unregularized one (dotted line) trained on the same circle countries. It fits the training circles worse, but it generalizes better to the square countries it never saw. Push the regularization hyperparameter to a very large value and the slope approaches zero, an almost flat model that cannot overfit but is unlikely to find a good solution. The lesson: the right strength lies between those two lines, and you find it by testing on unseen examples.

• If training performance is far better than performance on unseen data, increase regularization strength (raise alpha, or lower C in LogisticRegression).
• If performance is poor on both training and unseen data, the model is underfitting: decrease regularization, or use a more complex model, before anything else.
• If you are using Scikit-Learn LogisticRegression, remember the knob is C, the inverse of alpha: higher C means less regularization. Do not apply alpha intuition to it.
• If the model is overfitting and you can cheaply collect more data or clean outliers and data errors, do that alongside regularizing; overfitting depends on complexity relative to data amount and noise.
• If the fitted slope or coefficients collapse toward zero and predictions look almost flat, you have over-regularized: back the strength off.

• Reading C like alpha. You raise C expecting a stronger constraint and the model overfits more. The tell: results move opposite to what you intended after each change.
• Judging strength by training fit. Regularization always lowers training fit, so picking the value that fits training data best selects almost no regularization. The tell: the 'best' value is the smallest one tried and unseen-data performance is poor.
• Over-regularizing into underfitting. A very large value yields an almost flat model with slope near zero. The tell: both training and validation performance are bad and the predictions barely vary.
• Assuming no regularization is applied by default. Scikit-Learn's LogisticRegression adds an ℓ2 penalty unless told otherwise, so a 'baseline' is already constrained. The tell: an unexpectedly smooth fit or unexplained coefficient shrinkage.
• Treating the strength as something training will adjust. It is a hyperparameter, fixed before and during training. The tell: you expect a single run to find the right value instead of running a sweep.

• ▸Model and penalty type: name the model and the constraint (ℓ1, ℓ2, dropout, max-norm, tree hyperparameters, early stopping).
• ▸Strength hyperparameter and direction: record its name (alpha or C) and whether larger means more or less regularization.
• ▸Default in effect: note any penalty applied by default (e.g., ℓ2 in LogisticRegression).
• ▸Baseline gap: training performance versus unseen-data performance with minimal constraint.
• ▸Candidate values tried: list each strength value in the sweep, spanning very small to very large.
• ▸Training performance per value: record it to watch fit degrade as the constraint tightens.
• ▸Unseen-data performance per value: record it as the deciding metric.
• ▸Chosen value and reason: state the value and why it beats both extremes.
• ▸Non-regularization fixes considered: note more data, fewer attributes, simpler model, noise cleanup.