Data Science for Supply Chain Forecasting

1. 1Split the full dataset into a training set and a final test set before any experiment. Lock the test set away: it is used once, at the very end, to report the accuracy you can expect on new data.
2. 2Cut the training set into k folds (the passages' example uses k = 5). In scikit-learn, set k with the cv parameter of RandomizedSearchCV. Its default is 3, so set it explicitly.
3. 3For each candidate parameter set, run k experiments. Each trains on k−1 folds and validates on the one remaining fold, rotating the validation fold so every fold is held out once (Experiment #1 validates on Fold #5, #2 on Fold #1, and so on).
4. 4Score a candidate as the average accuracy across its k validation folds. Never rank a candidate on its training-set accuracy or on a single fold.
5. 5Pick candidates with random search, not an exhaustive grid. Define a param_dist dictionary of ranges (add None to max_depth to leave depth unrestricted) and let RandomizedSearchCV sample from it. Set n_jobs=-1 to run trees in parallel and use verbose to monitor progress.
6. 6Keep the parameter set with the best average validation score. All selection happens inside the training set, so the test set is never used to choose anything.
7. 7Refit the chosen model and score it once on the untouched test set. Report that number as expected accuracy. If it is far below the cross-validation score, treat the model as overfit rather than re-tuning against the test set.
8. 8Automate the split, search and final scoring as one workflow so the process repeats identically and manual steps cannot introduce random mistakes.

The passages' own first attempt at tuning a tree trained several versions on the training set and kept the one that scored best on the test set. The text calls this 'a terrible idea' that gives 'outstanding but artificial results': picking parameters on the test set means optimizing the model to fit the test set. The fix is the 5-fold design in Tables 14.1 and 14.2. The training set is split into Folds #1–#5 and the test set is held out. Experiment #1 trains on Folds #1–#4 and validates on Fold #5. Experiment #2 trains on Folds #2–#5 and validates on Fold #1. This continues until Experiment #5 trains on Folds #5, #1, #2, #3 and validates on Fold #4. Each parameter set gets the average of its five validation accuracies. Random search then tests many sets, and the best average wins. The sample comparison in the text: 25 random tests tried 10 distinct max_depth values and 9 distinct min_samples_split values, where a 25-point grid tries only 5 of each. The test set is touched once, at the end, to report the final result.

• If you are about to compare model versions on the test set, stop. Compare them on cross-validation validation folds instead, and keep the test set for the final report.
• If a result looks astonishing, treat it with suspicion and ask whether it replicates on new, unseen data before you accept it.
• If the model has many parameters that may interact, use random search with k-fold cross-validation rather than tuning one parameter at a time or relying on a uniform grid. Random search tests more distinct values of the parameters that matter.
• If a model performs well only on the training set, do not trust it. Validate it on the test set, and accept it only if it performs there too.
• If you want to cut computation time, raise n_jobs (-1 uses all processors) before you cut the number of folds or the search breadth.

• Selecting parameters by best test-set score. The tell: you ran several model versions and kept the winner on the test set, and your reported accuracy beats anything you see on new data.
• Using the test set to select or train a model, then calling it 'validated'. The tell: the test set was used more than once or influenced any choice.
• Trusting training-set accuracy as evidence. The tell: strong training results with no separate validation score, so overfitting goes unnoticed.
• Leaving cv at its default of 3 without checking. The tell: you describe the setup as 5-fold but never set cv, so the folds differ from what you reported.
• Tuning each parameter independently on a coarse grid. The tell: you tried only a handful of values per parameter and missed interactions or the one parameter that mattered.

• ▸Dataset and forecast target: name the data and what is predicted.
• ▸Train/test split: how the final test set was cut, and confirmation it is locked until the end.
• ▸Number of folds (k): the value passed as cv, not the default.
• ▸Parameter search space (param_dist): list each parameter's range, including None where allowed.
• ▸Search method and number of candidates: random search, and how many parameter sets were sampled.
• ▸Mean validation accuracy per candidate: average across the k validation folds, with the best one marked.
• ▸Chosen parameter set: the winner, selected on validation folds only.
• ▸Final test-set accuracy (single use): the one-time result, with the date it was run.
• ▸Gap check: compare cross-validation score and test score; flag a large gap as possible overfitting.

1. 1Plot the training demand next to the model's fitted forecast before you read any KPI. Fit quality is a question about pattern: does the forecast follow the level, the trend, the seasonality and the driver-linked swings?
2. 2Read the errors for structure. In the book's weekday/weekend case, the simple exponential smoothing model over-forecasts weekends and then under-forecasts the first weekdays. Errors that lean the same way at the same point in the cycle mean the model is missing information (underfitting). Errors that look like random scatter mean the model has learned what it can.
3. 3Name the missing piece. Ask whether the demand has an explanatory variable the model never received (weekday vs. weekend, promotion, advertisement, pricing, marketing expense). Ask also whether it has a structural component the model cannot represent (trend, seasonality).
4. 4Apply one of the two fixes the book gives for underfitting: add explanatory variables, or move to a more complex model (add a trend, add a seasonality). Exponential smoothing cannot take explanatory variables, so a driver-based fix means moving to the machine learning models (Chapter 20).
5. 5Do not rely on one linear driver rule. 'Demand increases by 10% for every degree above 25°C' can look fine on one range of data but fail outside it. Ice cream sales flatten at a minimum temperature instead of falling to zero. Check driver relationships across the full range, and check interactions such as weekend vs. Monday or several hot days in a row.
6. 6Size the dataset before you judge the fit. With n periods, the history input length x_len and the forecast horizon y_len, you get loops = 1 + n − x_len − y_len. Keep a full season (s_len loops) aside as the test set, so the model is judged on every seasonal pattern: test_loops = s_len.
7. 7Give the training set at least two full years of loops. For monthly data that means train_loops = 24, so n ≥ 35 + x_len + y_len. Two seasonal cycles let the model learn any seasonal relationship and the full test season checks it.
8. 8Judge fit on the held-out season, not on the training fit alone. A model complex enough to memorize noise looks good in training and fails on the test season. Raise complexity only while the test-season result also improves.

The book's weekday/weekend case shows underfitting. Simple exponential smoothing, f_t = α·d_{t−1} + (1−α)·f_{t−1}, produces one smooth level. The demand has a weekly pattern, so the model over-forecasts the weekends and then under-forecasts the first weekdays. The same-sign, same-position errors are the tell that the model is too simple. The fix is a model duo: one simple exponential smoothing model for weekdays and one for weekends. Figure 4.3 shows this duo fitting the demand much better, because the model now has a new explanatory variable, 'Is it a weekday?'. The book names the lesson as lack of information, not bad tuning. Dataset sizing works the same way. The Norway car dataset has 121 monthly periods. Suppose you choose x_len = 12 and y_len = 1 (illustrative choices, not from the book). Then loops = 1 + 121 − 12 − 1 = 109. Keeping 12 as the test season leaves 97 training loops, well above the 24-loop minimum. The minimum n would be 35 + 12 + 1 = 48.

• If errors lean the same direction at the same point of a cycle (weekends over-forecast, early weekdays under-forecast), then treat the model as underfit and add an explanatory variable or a trend/seasonality component. Retuning alpha will not fix it.
• If demand moves with promotions, advertisement, pricing or similar drivers and the model cannot see them, then expect underfitting. Move to a model that accepts exogenous variables rather than polishing exponential smoothing.
• If a single linear driver rule works on one range of data, then test it at the extremes before trusting it. Sales can flatten (the ice cream and temperature case) instead of continuing the line.
• If n < 35 + x_len + y_len on monthly data, then you lack two seasonal cycles plus a test season. Shorten x_len or y_len, or treat the seasonal fit as unproven.
• If added complexity improves the training fit but not the held-out season, then stop adding it. The extra fit is memorized noise, not a learned pattern.

• Judging fit on a single accuracy number and skipping the visual and error-pattern check. The tell: a decent average error with errors that are systematically signed at weekends or in specific months.
• Retuning smoothing parameters when the problem is missing information. The tell: every alpha setting leaves the same over/under-forecast pattern.
• Extrapolating a linear driver relationship (temperature → sales) beyond the observed range. The tell: forecasts that go implausibly low or negative at extremes, while real sales flatten.
• Testing on a window shorter than a season. The tell: good test results that collapse when an unseen part of the seasonal cycle arrives.
• Training on less than two seasonal cycles, or spending too much history on x_len and y_len. The tell: very few loops (1 + n − x_len − y_len is small) and unstable seasonal forecasts.

• ▸Dataset and frequency: name the series (e.g., monthly car sales, Norway) and its number of periods n.
• ▸Input/output window: record x_len (periods used per prediction) and y_len (periods forecast at once).
• ▸Loop count: compute loops = 1 + n − x_len − y_len.
• ▸Test season: set test_loops = s_len (12 for monthly) and train_loops = loops − s_len. Check train_loops ≥ 24.
• ▸Error pattern observed: note where errors share a sign (weekends, specific months, promotion periods) or write 'no pattern'.
• ▸Missing information suspected: list candidate explanatory variables the model never saw (weekday/weekend, promotion, advertisement, price, marketing spend).
• ▸Structural gap suspected: mark missing trend, missing seasonality, or neither.
• ▸Chosen fix: pick add explanatory variables, add trend/seasonality, or no change, and write why.
• ▸Held-out season result: compare training vs. test-season error; if training improves and test does not, flag possible memorized noise.

1. 1Treat any astonishing accuracy as a suspect. Before you celebrate, write down the question the source poses: can I replicate these results on new (unseen) data? Training accuracy alone does not answer it.
2. 2Set aside unseen periods before you fit or tune anything. Score the model on those periods only after the parameters are fixed. A model that is excellent on the fit data and weak on the unseen data is overfit.
3. 3Audit every initialization for data leakage. If the starting level or forecast uses information from several periods ahead (for example, the average of the first five periods), the early forecasts are given the answer. Prefer initializations that use as little future information as possible, such as a0 = d0 and b0 = d1 − d0.
4. 4If you initialize with an average or a linear regression over the first n observations, test different n. Use a small fixed value (3 to 5) or a value tied to the learning rate (1/α, or proportional to the average of 1/β and 1/α). Leakage grows as n grows, so keep n low.
5. 5Check whether the initialization makes a forecast perfect by construction. With a0 = d0 and b0 = d1 − d0, f1 = a0 + b0 = d1 exactly. Remove that first forecast from the forecast error computation so the method gets no unfair advantage. On small datasets the advantage can cause overfitting.
6. 6Resist adding special rules and exceptions to fix individual misses. Each quick fix makes the model worse on new data and harder to understand. When a pattern keeps failing, look for a structural fix instead.
7. 7Automate the experiment loop (data workflow plus model). Many experiments are needed to find the best model, and a manual loop adds random mistakes. Run each candidate through the same unseen-data test and share good and bad results with your team so others can challenge them.
8. 8Don't let a few failed trials discourage you, and don't let one lucky trial convince you. Judge a configuration by its repeated performance on data it never saw.

The source gives a clean case of leakage-driven overfitting. You initialize the forecast as the average of the first five demand periods. In effect you ask the model: can you forecast the next period, knowing that the average demand of the next five periods is 10? The model delivers good accuracy on those initial periods because you handed it their average, but it cannot repeat that accuracy later. The same pattern appears in double exponential smoothing with simple initialization. Setting a0 = d0 and b0 = d1 − d0 gives f1 = d0 + (d1 − d0) = d1, a perfect first forecast by construction. On a big dataset this is one good prediction among many and does little harm. On a small dataset it inflates apparent accuracy and can mislead you. The fix is to drop that first forecast from the error computation. Linear regression initialization over the first n observations has the same weakness, since leakage increases as n grows. The remedy is to keep n small (3 to 5, or tied to 1/α) and judge the model on periods it did not see.

• If a model's accuracy looks astonishingly good, then assume luck or overfitting until it is replicated on new, unseen data.
• If an initialization uses several future periods (an average or a regression over the first n), then reduce n and drop those initial periods from the error calculation.
• If the initialization makes the first forecast exact (f1 = d1), then exclude that forecast from the error, especially on a small dataset.
• If you are tempted to add another special rule or exception for a miss, then look for a structural fix, because accumulating quick fixes erodes performance on new data.
• If you need to test many configurations, then automate the workflow and judge every result on unseen data, not on the data used to fit.

• Judging a model by fit on the training data. The tell: strong historical accuracy that disappears on new periods.
• Leaky initialization, such as starting with the average of the first five periods. The tell: unusually good early forecasts that cannot be repeated later.
• Counting the free perfect first forecast on a small dataset. The tell: the initialization method looks better than its rivals only because of f1 = d1.
• Piling up special rules and exceptions to fix individual misses. The tell: the model grows more complicated, its interdependencies become hard to explain, and it behaves worse on new data.
• Trusting a lucky result from manual, ad hoc experiments. The tell: a standout outcome nobody can reproduce, while random mistakes creep into the workflow.

• ▸Model and configuration: name the model and its parameters (for example α, β, n for initialization).
• ▸Fit-data accuracy: the error KPI on the data used to fit and tune.
• ▸Unseen-data accuracy: the same KPI on periods never used for fitting or tuning.
• ▸Gap between the two: if the unseen error is clearly worse, flag overfitting.
• ▸Initialization method: describe how a0, b0 or f0 are set and how many future periods they use.
• ▸Leakage check: does the initialization contain information about multiple periods ahead? Yes or no, and the value of n.
• ▸First-forecast exclusion: is any forecast perfect by construction removed from the error? Yes or no.
• ▸Special rules added: list any exceptions or quick fixes, and whether a structural fix exists.
• ▸Replication verdict: can the result be reproduced on new unseen data? Yes, no, or untested, with the next action.

1. 1Plot the raw demand series by period before fitting anything. Mark by eye any level shifts, a trend, a repeating seasonal shape, and isolated spikes. This tells you which of the book's limitations (no trend, no seasonality) a simple model will hit.
2. 2Fit a naïve forecast (moving average n = 1, 'tomorrow will be just as today') and a long moving average (n = 8). Record the error of each. The naïve forecast reacts instantly but passes noise and outliers straight into the forecast. The n = 8 average is stable but slow to react to a level change. The gap between them shows how much of the series is noise and how much is real movement.
3. 3Fit simple exponential smoothing, f_t = α·d_(t−1) + (1−α)·f_(t−1), with 0 < α ≤ 1. A high α learns more from the last observation, so it is more reactive and more noise-sensitive. A low α is smoother but slower. Treat α as the dial for the reactivity-versus-smoothness trade-off.
4. 4Add structure only where the plot shows it. Add a trend model (Chapters 5 and 7) if demand drifts. Add a seasonal model (Chapters 9 and 11) if a pattern repeats. Add one layer at a time and re-measure after each. Start simple and iterate; do not build everything at once.
5. 5Keep the latest periods out as a test set and never optimize on them. Tune α and other parameters on the training data only. The test error is your honest estimate of accuracy on new demand.
6. 6Check the training error for underfitting. If the model is poor even on history, either its complexity is too low for the pattern or it lacks explanatory variables. If so, change the model class or add drivers. Do not keep tuning parameters.
7. 7Stop adding complexity when the test error stops improving meaningfully. What is left is the noise of the series, not a modeling gap. Do not chase it with more rules and exceptions ('simplicity over complexity').
8. 8Treat any astonishing result as suspect. Ask whether you can replicate it on unseen data. If not, you have overfit the noise rather than beaten it.

The book's Figure 4.1 uses a dummy demand series that follows a noisy quadratic curve (y = −x²). A linear regression fits it with an RMSE of 34%. It misses the changing level and would keep forecasting the average historical demand, even though future demand is likely close to zero. Simple exponential smoothing tracks the level change and reaches an RMSE of 30%. Two lessons follow. First, the 4-point gain came from matching the model to the structure (a moving level), which is a modeling fix. Second, even the model that understands the pattern keeps a 30% RMSE, because the series is noisy. The passages do not say whether 30% is the floor. They do show that the remaining error is not a sign the model is broken. The next step is therefore to test whether a richer model beats 30% on a held-out test set. If it only does so on training data, you are fitting noise.

• If the naïve forecast (n = 1) beats your model on the test set, then your model is chasing noise or the series is mostly random. Simplify before adding anything.
• If the forecast swings with every spike, then lower α or lengthen the moving average. If it lags a real level change, then raise α or shorten the average. This is the reactivity-smoothness trade-off.
• If the plot shows drift, then move to a trend model (Chapters 5 and 7). If it shows a repeating pattern, then move to a seasonal model (Chapters 9 and 11). A simple model structurally cannot capture these.
• If error is high on the training set as well as the test set, then the model is underfit: raise model complexity or add explanatory variables. If training error is excellent but test error is much worse, then it is overfit: simplify.
• If a result looks too good, then re-run it on unseen data before trusting it or reporting it.

• Tuning on the test set. The tell: test accuracy looks great, then collapses on new demand, because you no longer have an unbiased estimate.
• Treating all error as a modeling failure. The tell: you keep adding rules and exceptions for small accuracy gains that vanish on fresh data.
• Using a model class that cannot express the pattern, for example a linear model on a changing level or a moving average on seasonal demand. The tell: poor fit on the training data too, as with the 34% RMSE linear fit.
• Reading a lucky result as skill. The tell: astonishing accuracy on one dataset that cannot be replicated on unseen periods.
• Setting α or n to a single extreme and ignoring the trade-off. The tell: a naïve-like forecast that mirrors every spike, or an n = 8-like forecast that is slow to follow a genuine level shift.

• ▸Series and bucket: name the item and period length (monthly, daily) and the number of observations.
• ▸Visible structure: tick level shifts, trend, seasonality, or spikes seen in the plot.
• ▸Naïve (n = 1) test error: record the error of the 'tomorrow = today' forecast as your baseline.
• ▸Moving average (n = 8) test error: record it, and note how slowly it reacted to level changes.
• ▸Simple exponential smoothing: record α used and its training and test errors.
• ▸Added structure tried: list the trend or seasonal model added and the test error after each layer.
• ▸Test-set discipline: confirm which periods were held out and that no tuning used them.
• ▸Gain per added layer: record the test-error change; stop when it is marginal.
• ▸Replication check: note whether the best result held on new, unseen data (yes or no).

1. 1Fix the split before you fit anything. Keep the most recent period(s) aside as the test set. In the book's quarterly example the last loop is held out: demand from Y2Q4 to Y3Q3 is the input, and Y3Q4 is the target. The model never sees these relationships during training.
2. 2Train and tune only on the training set. In the book's words, never use the test set to select or train a model; it is only for a final idea of the accuracy you can expect. If you need to compare models or parameters, use a validation set carved from the training data, not the test set.
3. 3Define the error per period as e_t = f_t − d_t (forecast minus demand). Overshoot gives a positive error and undershoot a negative one. Compute it on every test period with the same formula you used in training, so the two are comparable.
4. 4Report accuracy and bias separately, because the book treats them as different things. Bias is the average error. Add MAE and RMSE for accuracy. The goal is a forecast that is both accurate and unbiased.
5. 5Compute the same KPIs on the training set and on the test set, and put them side by side. A model that only performs well on the training set cannot be trusted. Only the test-set numbers count as the generalizable result.
6. 6Check whether the test result replicates. Ask: can I replicate these results on new (unseen) data? Treat astonishing results with suspicion. They signal luck or overfitting, which the book calls the number one curse for data scientists.
7. 7If test accuracy is far worse than training accuracy, simplify the model and add more data. More data reduces the chance of seeing fake patterns. Do not add special rules or quick fixes, because the model loses reliability on new data. Return to step 2 without peeking at the test set.
8. 8Choose KPIs by experimentation. No single indicator fits every business, and each avoids some pitfalls while being prone to others. Run the test-set comparison over several KPIs before you commit to one.

The book's machine-learning dataset is quarterly demand for several products, restructured into loops. Each loop uses 4 quarters of history (X) to predict the next quarter (Y). In the training set, loop #1 for product #1 reads 5, 15, 10, 7 → 6. Loop #2 slides the window forward: 15, 10, 7, 6 → 13. The same pattern continues for the other products and loops. The last loop is held out as the test set. Demand from Y2Q4 to Y3Q3 is the input, and Y3Q4 is the target. The model is fitted on the training loops only. You then forecast the held-out Y3Q4 for each product and compute e_t = f_t − d_t. From those errors you compute bias, MAE and RMSE on the test set. Those test numbers, not the training ones, tell you what to expect on future demand. If training error is low and test error is much higher, the model has overfit.

• If a result looks astonishingly good, then treat it with suspicion and re-test on new unseen data before trusting it.
• If you are tempted to tune a model using test-set results, then stop. Tune on a validation set and use the test set only once, as the final check.
• If bias is clearly positive or negative while the error KPIs look acceptable, then the forecast is systematically over- or under-shooting. Treat that as a separate defect, because the goal is accurate and unbiased.
• If test error is much worse than training error, then reduce model complexity or add more data. Do not add special rules or quick fixes.
• If one KPI ranks the models differently from another, then do not pick by habit. Experiment on your own data to see which KPI suits your business, since each KPI avoids some pitfalls and is prone to others.

• Tuning on the test set. The tell: you rerun the test evaluation after every tweak and the test score keeps improving. The test set has then become part of training and no longer shows true performance.
• Reporting training accuracy as the forecast accuracy. The tell: the numbers look great, but nobody has run the model on held-out periods.
• Relying on a single KPI. The tell: the KPI looks fine while the forecast consistently runs above or below demand. Check bias next to MAE and RMSE.
• Letting a complex seasonal model fit a simple pattern. The tell: it finds seasonality where there is none and extrapolates it, so the test error jumps compared with training. Use a simpler model or more data.
• Piling on special rules to fix individual misses. The tell: each patch improves the history but the model becomes harder to explain and performs worse on new data.

• ▸Model name and version: label what is being evaluated.
• ▸Training window: the periods the model was fitted on.
• ▸Test window: the held-out periods, which must not overlap the training window and must not have been used to choose the model.
• ▸Training bias, MAE, RMSE: the KPIs computed on the training set.
• ▸Test bias, MAE, RMSE: the same KPIs computed on the test set, using e_t = f_t − d_t.
• ▸Train-vs-test gap: note how much worse the test KPIs are than the training KPIs.
• ▸Replication check: record whether the result holds on another unseen window or dataset (yes/no, with evidence).
• ▸Test set touched during tuning? Answer yes or no. If yes, the test result is invalid, so re-split.
• ▸Decision: accept, simplify the model, or gather more data, with a one-line reason.

1. 1Fix the baseline first. Let the statistical or ML engine produce the forecast baseline, and treat every human input as an edit to that baseline. Judgment is a layer on top of the model, not a replacement for it.
2. 2Restrict judgment to where it is appropriate: products with no history to model, or periods of significant change such as new client behavior, new competition, changing legislation or a pandemic. Everywhere else, the default is to leave the baseline untouched.
3. 3Map the incentive and power structure before you collect any input. List each contributor (channel manager, production planner, sales team, executive), what they gain from a high or low number, and who has the final sign-off. These are the blind spots the source flags: misaligned incentives, and power and influence.
4. 4Collect inputs independently before any group discussion. The loudest person in the room otherwise sets the number. Gather each contributor's adjustment separately, then discuss the gaps.
5. 5Separate the forecast from the goal. Keep the forecast distinct from the yearly budget or sales target, so nobody adjusts the number to hit a goal. An executive who does not want to sign off declining sales is the typical pressure to watch for.
6. 6Attach a name and a rationale to every override: who changed it, by how much, and what information justified it. Access to information is a source of influence, so the rationale makes that information visible.
7. 7Track accuracy and bias for the baseline and for each human step on the same periods and the same metrics. Use MAE and bias, as the book's results tables do, so you see error size and direction.
8. 8Apply the forecast value added test: a human step stays only if it beats the version before it. Remove or retrain the steps that do not, so the team stops spending effort that adds nothing.
9. 9Review at portfolio level on a fixed cadence. Judge the whole portfolio KPI, not single wins, and separate wanting to be right from wanting to have been right (Quine): record misses without defending them.

Table 25.1 in the book compares seven models on the same dataset, with test-set MAE and bias: Regression 17.8 / 1.6, Tree 18.9 / 2.9, Forest 17.6 / 2.7, ETR 17.5 / 2.5, AdaBoost 17.9 / 3.9, XGBoost 17.2 / 3.6, NN 17.5 / 3.7. Apply the same scoring logic to a human layer. Take the best-MAE model, XGBoost, as the baseline: test MAE 17.2, bias 3.6. A planner's override is scored in the same two columns on the same periods. The override has to land below 17.2 MAE to justify the effort. The bias column also matters. Plain regression has a worse MAE (17.8) but the lowest test bias (1.6), so a layer that only shifts the bias deserves a separate look. Training figures can mislead: AdaBoost shows 10.8 train MAE against 17.9 test, so judge every layer, human or model, only on out-of-sample periods. If the override does not beat the baseline over repeated periods, drop that step, which is the forecast value added decision.

• If the product has history and no major change is under way, then leave the baseline alone and require a documented reason for any override.
• If the product is new, lacks history, or the environment is shifting (client behavior, competition, legislation, pandemic), then use judgment, and score it against whatever baseline you can build.
• If one stakeholder or team has outsized power over the number, then collect inputs independently before the group meets and record who has final sign-off.
• If the forecast is being steered toward a budget or target, then split the forecast from the goal and track the gap between them explicitly.
• If a human step does not beat the previous step on tracked error over several periods, then remove it or change who does it.

• Letting the loudest voice set the number. The tell: group consensus always lands near one person's figure.
• Using the forecast as the budget goal. The tell: forecasts drift to match targets, and declining sales never appear in signed-off numbers.
• Overriding the baseline everywhere out of habit. The tell: many small edits with no recorded rationale and no accuracy comparison to the baseline.
• Judging a model or a layer on training or in-sample results. The tell: a large gap between train and test error, as AdaBoost's 10.8 vs 17.9 MAE.
• Tracking error size but not direction. The tell: MAE is acceptable while bias is persistently positive, as in every test-set bias in Table 25.1.

• ▸Product / segment and period: identify what is being forecast and when.
• ▸Baseline forecast (model): record the statistical or ML baseline before any human input.
• ▸Reason for judgment: state why judgment applies (no history, new competition, legislation, client change, pandemic, other).
• ▸Contributor and role: name who proposes the change and their stake (channel, production, sales, executive).
• ▸Independent proposal (submitted before group discussion): record each person's number prior to the meeting.
• ▸Final forecast and sign-off owner: record the agreed number and who decided last.
• ▸Rationale and information source: record what evidence supports the change.
• ▸Actual demand and errors: after the period, record MAE and bias for baseline versus final forecast.
• ▸Value added verdict: mark whether the override beat the baseline (yes/no) and keep, retrain or remove the step.

1. 1Separate three documents before any adjustment happens: the forecast (the best unbiased estimate of future demand), the budget or sales target, and the production plan. Write the definition on the process page: a demand forecast is not a goal. Any adjustment that serves a goal is a bias source, not an input.
2. 2Map each step of the forecasting flow and name the owner: ML baseline, demand planners, sales team, senior management. For each, note the likely distortion. Misaligned incentives: sales may under- or over-forecast depending on bonus design, customer service over-forecasts to secure inventory, a project manager pushes a new product. Power and influence: the loudest voice in the room multiplies those incentives. Pressure to return to the yearly budget turns the forecast into a goal.
3. 3Freeze the baseline before humans touch it. Record the ML (or statistical) forecast as the benchmark. Every later step is stored as its own version, so each person's change can be compared with the version before it.
4. 4Track two measures per stakeholder or team: FVA against the previous step (MAE, RMSE and bias) and the hours spent. Compute FVA as the previous step's error minus this step's error. A positive number means the step helped, a negative one means it hurt.
5. 5Run the analysis over multiple forecast cycles, not one. Anyone can be lucky or unlucky once. If channels or business units differ in information, teams and buying behavior, track FVA separately for each one.
6. 6Counter unintentional bias in the meeting design. Do not let one person open with a number such as 'I believe we will sell 20% more than last quarter'. Anchoring on an unrelated figure is documented (Tversky and Kahneman's wheel-of-fortune experiment). Collect estimates from several people independently, then average them, as in a group forecast or an ensemble. Invite different information sources and value dissenting views.
7. 7Read the results and act. Cut hours at steps with no FVA, and point effort at the SKUs where the model has the highest weighted error or where planners think it lacks insight. Expect small tweaks to add nearly no value and consume time. Larger adjustments are more likely to help, because they demand more explanation and carry personal risk.
8. 8Handle the underlying need separately. If sales wants secured inventory, work with supply planners to secure it through the plan, not through a biased forecast.

The source's Table 27.6 shows a three-step process analysed with FVA. The ML model benchmark has MAE 45% and bias −3%. Demand planners spend 72 hours and improve MAE by 2 points (to 43%) and RMSE by 4, with bias +2 in their favor (to 1%). That is the step worth keeping. The sales team then spends 20 hours and worsens MAE by 2 points (back to 45%) and bias by 4 points (to 5%). The tell is the positive bias: they are inputting a high forecast to secure inventory, which is intentional judgmental bias. Senior management spends 8 hours: MAE is unchanged, RMSE rises by 1, bias improves by 1 (to 4%). They are still overforecasting, likely entering what they hope to sell, not what they expect to sell. The fixes: work with supply planners to secure inventory outside the forecast, separate forecast ('how much we think the demand will be') from plan ('how much we should produce/source'), and have sales and management spend less time tweaking and focus on a few products they are most sure about. Without the data-driven FVA, convincing senior management would be very difficult.

• If a step's FVA is zero or negative over several cycles, then cut the time it spends editing the forecast and restrict it to the few items where it holds real information.
• If a team's adjustments consistently push bias in one direction (e.g., sales +4 bias points), then treat it as intentional bias tied to incentives and fix the incentive or the inventory-securing channel, not just the number.
• If an adjustment is small and frequent, then expect nearly no value and stop it; if it is large, then require an explanation, since large adjustments are more likely to improve accuracy.
• If a forecast comes in near the budget or target when the baseline says otherwise, then suspect the forecast is being used as a goal and split forecast from plan.
• If one person's number is voiced first in a meeting, then expect anchoring: gather independent estimates and average them before discussion.

• Judging a stakeholder on a single cycle. The tell: praise or blame flips from month to month. FVA needs multiple cycles because anyone can be lucky.
• Mixing forecast, budget and plan in one number. The tell: the forecast tracks the yearly budget after a shortfall, or management 'forecasts' what it hopes to sell.
• Rewarding many small tweaks. The tell: planners log many edits and many hours, but FVA is near zero.
• Running the group forecast as a debate led by the loudest person. The tell: the group converges on the first number spoken, so cognitive biases do not offset each other.
• Pooling all business units in one FVA. The tell: results are muddy where channels have different information, teams and buying behavior, hiding which unit's bias is hurting accuracy.

• ▸Forecast cycle / period: label each cycle so FVA is reviewed across multiple cycles.
• ▸Business unit or channel: track separately where information and buying behavior differ.
• ▸Process step and owner: ML baseline, demand planner, sales, management, or an individual.
• ▸Hours spent: record the time this step consumed on the forecast.
• ▸Error before step (MAE, RMSE, bias): from the previous version.
• ▸Error after step (MAE, RMSE, bias): from this step's submitted version.
• ▸FVA (before minus after): positive = helped, negative = hurt, per metric.
• ▸Suspected bias source: incentive misalignment, power/influence, anchoring, confirmation, or none.
• ▸Action: cut time, restrict to specific SKUs, split forecast from plan, or keep.

1. 1Write down every step that touches the forecast, in order: a naive benchmark, then each statistical or ML model, then any judgmental adjustment. The book's own model ladder is moving average, exponential smoothing, double smoothing, damped trend, triple smoothing, then machine learning. Judgmental forecasts come after that in Part III. Forecast value added (FVA) is measured at each rung, not just at the end.
2. 2Fix the naive benchmark before you look at any results. Use something you can compute with no tuning, such as a moving average or the last observation. Freeze it. A benchmark chosen after the fact can be picked to flatter the process.
3. 3Choose one forecast KPI and keep it fixed across all steps. Bias, MAPE, MAE and RMSE are separate measures in Chapter 2. Switching KPIs between steps makes the deltas meaningless. Report bias next to the error KPI, because a step can reduce error while adding bias.
4. 4Score every step on the same holdout (test) data, on the same items and the same periods. Never use the test set to select or train a model. It is only for a final read on the accuracy you can expect. Anything scored on the training set is not value added.
5. 5Compute two deltas for each step: KPI(step) versus the naive benchmark, and KPI(step) versus the prior step. The first is the cumulative value of the effort. The second is the marginal value of that step. Show both, because a step can look good against naive and still add nothing over the step before it.
6. 6Check replicability before you claim the value. Treat astonishing gains with suspicion and ask whether the result holds on new, unseen data. A complex model that beats the previous step only on training data is overfitting, not adding value.
7. 7Automate the pipeline so every step is re-scored the same way each cycle (data in, forecasts, KPI, deltas). The book says automation is the key to fast experimentation and to reliability. Manual scoring creates random mistakes and cannot scale across a portfolio.
8. 8Aggregate across the portfolio (Chapter 27.1, Portfolio KPI) and then look at segments. Keep a step only where it shows positive added value on the chosen KPI. Where a step shows none, drop it or simplify it.

The passages give no published FVA numbers, so the figures below are illustrative. Only the ladder and its logic come from the book. Take one seasonal SKU. The naive benchmark is a moving average. Double exponential smoothing adds a trend, which the book calls a major improvement over simple smoothing. It still has no seasonality, so on a seasonal product it should not beat the benchmark by much. Triple exponential smoothing adds multiplicative seasonal factors, such as +20% in January and −30% in February, and that is where the gain should appear. Suppose test-set MAE is 100 units for the moving average, 95 for double smoothing, and 70 for triple smoothing. Then double smoothing adds 5 units versus naive (5%) and 5 versus its prior step. Triple smoothing adds 30 versus naive (30%) and 25 versus its prior step. A machine-learning model at 68 adds only 2 over triple smoothing, so you ask whether that gain survives on unseen data and justifies the extra complexity. If a judgmental override then pushes MAE to 74, it has negative value. That is the blind-spot risk Chapter 26 warns about, and FVA is the measurement that exposes it.

• If a step's KPI is not better than the prior step's on the holdout data, then remove it or do not adopt it, however sophisticated it is.
• If a model beats naive on training data but not on the test set, then treat it as overfitting and do not count any value added.
• If a step lowers error but raises bias, then do not report it as value added until you have reviewed the bias.
• If the value added is positive only for a few items or segments, then apply the step to those segments only and keep the simpler model elsewhere.
• If a human override has negative FVA against the statistical forecast, then stop it or restrict it to cases where it has shown positive value.

• Scoring on the training set. The tell: large gains in the lab that vanish on new data.
• Moving the benchmark or the KPI between steps. The tell: deltas that cannot be reproduced and a KPI that changes from report to report.
• Reporting only the cumulative gain versus naive. The tell: late steps that add nothing, but nobody sees it because the total still looks good.
• Counting error reduction while ignoring bias. The tell: error falls while forecasts drift consistently high or low.
• Manual, one-off scoring. The tell: FVA figures that differ each cycle with no change in process, and results nobody can rerun.

• ▸Process step: name each step in order, from the naive benchmark through each model to any judgmental override.
• ▸Benchmark definition: state the naive method and freeze it, with its date.
• ▸KPI used: record the single KPI (MAE, MAPE, RMSE) and the bias measure used for every step.
• ▸Holdout scope: list the items, periods and test set used. It must be identical across steps.
• ▸KPI value per step: enter the test-set score for each step.
• ▸Delta vs naive: benchmark KPI minus step KPI, and the same as a percentage.
• ▸Delta vs prior step: previous step KPI minus this step KPI.
• ▸Bias check: note the bias of this step versus the prior step.
• ▸Decision: keep, simplify or drop the step, with one line of reasoning and a replication check on unseen data.

1. 1Start with the simplest model and add complexity one layer at a time. The book's rule is to tackle questions with simple models first, then iterate. Begin with a moving average or simple exponential smoothing and record its error as your benchmark.
2. 2Split the data before you compare anything. Keep the latest periods out of the training set as a test set. Never use the test set to tune parameters, or you lose the only honest estimate of accuracy on new demand.
3. 3Pick one error KPI (RMSE or MAE) as the objective and use it for every model. Compute it on the training set to diagnose underfitting and on held-out data to diagnose overfitting. Skip the error for the first observations where it makes no sense, and never compute it on future periods that have no actual demand.
4. 4Diagnose underfitting first. If a model scores poorly on the training data, its inner complexity is too low or it lacks explanatory variables. Move up one step: simple smoothing, then double smoothing (trend), then damped trend, then triple smoothing (seasonality). If the history cannot explain the demand (the ice cream near a school case), add demand drivers as features rather than a fancier model.
5. 5Move to machine learning (tree, forest, ExtraTrees, AdaBoost, XGBoost) only when the statistical ladder plateaus or you have drivers and leading indicators to feed it. Build the training and test datasets first, run a linear regression as a baseline, then a tree.
6. 6Optimize parameters by automated search. Use Solver in Excel or a random search with k-fold cross-validation in Python. Machine learning models have many parameters, and cross-validation limits the overfitting risk of testing many sets. Accept that overfitting can never be removed completely, and that it grows with model complexity.
7. 7Run an ensemble comparison. A forest averages many trees on the wisdom-of-the-crowd principle, which requires independent information, independent decisions, and a way to aggregate. Keep it only if it beats the simpler benchmark on unseen data.
8. 8Treat any astonishing result with suspicion and ask whether you can replicate it on new data. Keep the more complex model only if it beats the benchmark on the test set, and share the results, good and bad, with your team.

The book's Figure 4.1 uses a dummy demand history that follows a noisy quadratic curve (y = −x²). Two models are fitted. The linear regression is too simple and fits poorly, with an RMSE of 34%. If you used it to forecast the next periods, which are likely to be close to zero, it would keep predicting the average historical demand. Simple exponential smoothing reaches an RMSE of 30%, because it recognizes that the demand level changes over time. The lesson is that the linear model was not complex enough for the relationship in the data, so you move up the ladder. The same logic applies in Part II. After random search with cv=10, the tuned tree still did not beat the benchmark, so the book moves to a forest. Each step is judged against the benchmark on data the model has not seen.

• If a model performs badly on the training set, then it is underfitting: increase complexity (add trend or seasonality) or add explanatory variables before touching anything else.
• If the training error is excellent but the test error is poor, then it is overfitting: simplify the model, or use cross-validation and a holdout set.
• If demand depends on external factors, such as ice cream sales near a school, then history alone will underfit. Add demand drivers through a machine learning model instead of a more elaborate smoothing model.
• If a complex model does not beat the simple benchmark on the test set, then keep the simple model. The book favors simplicity over complexity.
• If you are tempted to optimize on the test set, then stop: use a validation set or k-fold cross-validation and keep the test set for the final accuracy check.

• Choosing a model that is too simple for the pattern, such as a linear model on curved demand. The tell is a poor fit on the training data itself, and a forecast that flatlines at the historical average.
• Jumping straight to a machine learning model without a benchmark. The tell is that you cannot say whether the added complexity improves on a moving average or exponential smoothing.
• Tuning on the test set. The tell is excellent test accuracy that disappears on new demand, because you no longer know the accuracy to expect.
• Running huge parameter searches and trusting the winner. The tell is a best-of-many score that looks astonishing, since more parameters and more experiments increase overfitting.
• Piling on special rules and exceptions to a model. The tell is a growing model nobody on the team can explain or replicate.

• ▸Product/demand pattern: describe level, trend, seasonality, and noise in the history.
• ▸Error KPI and objective: choose RMSE or MAE and use it for every model.
• ▸Train/test split: state which latest periods are held out and confirm they are never used for tuning.
• ▸Benchmark model and test error: record the moving average or simple smoothing result.
• ▸Candidate model and added complexity: name the next step (trend, damped trend, seasonality, tree, forest, boosting).
• ▸Training error vs test error: record both to flag underfitting or overfitting.
• ▸Explanatory variables available: list demand drivers or leading indicators, or write 'none'.
• ▸Tuning method and number of experiments: record random search, k-fold value, and trials run.
• ▸Decision: keep the simpler model or move up, with the one-line justification shared with the team.

1. 1Name the parameters and their legal ranges before running anything. For exponential smoothing: alpha (also beta, gamma) must satisfy 0 < α ≤ 1. For tree and ensemble models: tree depth, learning rate, number of estimators. Write each range down first.
2. 2Pick one forecast KPI (bias, MAPE, MAE or RMSE) as the score. The source says no KPI fits every case and only experimentation shows which suits you, so fix the KPI before comparing configurations. Otherwise each configuration will look best on a different metric.
3. 3Wrap the model in a function that takes parameters and returns a score. Chapter 6 builds models such as moving_average(d, extra_periods, n) as functions so an optimization engine can call them repeatedly.
4. 4Run simple experiments first: vary one parameter at a time across a small grid and record the score. This shows the direction of the trade-off, for example low alpha reacts slowly while high alpha chases noise.
5. 5Move to smarter experiments when parameters interact or ranges are wide: random search over the ranges, with each candidate scored by cross-validation, not by one in-sample fit.
6. 6Score on data the model did not train on. Use a training / validation / holdout split (the Feature Optimization chapter's training set, validation set, and holdout dataset). Choose parameters on validation and report the final result on holdout once.
7. 7Check initialization and inputs for leakage. Initializing a forecast with the average of the first five periods gives the model information about the future and flatters the early-period score.
8. 8Rerun the best configuration, then check the chosen values are plausible. An extreme value (alpha near 1 turns simple exponential smoothing into a naïve forecast) signals a problem, not a win. Log the result.

Take simple exponential smoothing, f_t = α·d_(t−1) + (1−α)·f_(t−1). The source's Figure 3.4 compares α = 0.1 and α = 0.8 on the same demand. At 0.1 the forecast reacts slowly to changing demand. At 0.8 it follows demand fluctuations too closely. At α = 1 the model collapses into a naïve forecast. That gives you the search interval and a sanity check: a best alpha sitting at 1 means the model is just copying last period. Run a one-at-a-time sweep of alpha, score each value on a validation window with your chosen KPI, and keep the one with the lowest error. Initialize f_0 without peeking at future periods. A second parameter in the source is the outlier threshold. In the error-based method, errors have mean −0.4 and standard deviation 3.2, and the 99% multiplier 2.33 gives limits of [−8, 7]. Demand 19 against a forecast of 5 is capped at 5+7 = 12. The source leaves the number of standard deviations to experiment, so treat the multiplier as a parameter and tune it the same way.

• If you have one or two parameters with narrow ranges, use a simple one-at-a-time sweep. If you have many parameters or wide ranges, use random search with cross-validation.
• If the best alpha lands at 1, or the forecast just echoes the last observation, then the model is behaving like a naïve forecast. Revisit the model structure and the data, not just the parameter.
• If a configuration scores far better on training data than on validation or holdout data, then reject it as overfitting and prefer a simpler setting.
• If an initialization or feature uses information from periods you are forecasting, then remove it before tuning, because the score is leaking.
• If two KPIs rank configurations differently, then decide which error matters to the business (bias vs. accuracy) before choosing a winner, not after.

• Tuning on the same data you report on. The tell: a great score that collapses on new periods.
• Data leakage in initialization (for example, starting at the average of the first five periods). The tell: unusually low error in the first few periods only.
• Changing the KPI mid-search. The tell: the 'best' configuration changes each time you switch metric, and you cannot say why one was chosen.
• Taking an edge value at face value (alpha = 1). The tell: the tuned model equals a naïve forecast and adds nothing over it.
• Tuning on outlier-contaminated data. The tell: the error standard deviation is inflated, so thresholds are overestimated (the source's 'Go the Extra Mile' point). Recompute without the flagged outliers before trusting the parameters.

• ▸Model and version: which model is being tuned (e.g., simple exponential smoothing, tree, XGBoost).
• ▸Parameters and ranges: each hyperparameter with its allowed range (e.g., 0 < α ≤ 1).
• ▸Scoring KPI: the single metric used (Bias, MAPE, MAE or RMSE) and why.
• ▸Search method: one-at-a-time sweep, or random search with cross-validation and number of trials.
• ▸Data split: the training, validation, and holdout periods used.
• ▸Leakage check: how initialization and features avoid using future periods.
• ▸Trial table: configuration, training score, validation score for every run.
• ▸Selected configuration: the winner, picked on validation score.
• ▸Holdout result: the one-time final score, and whether it matches validation.

1. 1Format the history as a supervised dataset first. Each row of X holds a fixed window of past demand (the book's dummy example uses four quarters) and Y holds the next observation. Split into X_train/Y_train and X_test/Y_test before adding anything, so every later feature is judged on data the model has not seen.
2. 2Stack all products into the same dataset. The model learns from products #2, #3 and #4 as well as #1, so the relationships it finds are shared across the portfolio. Features are what let it tell the products apart.
3. 3Add features in the order the book introduces them: more historical demand horizons, then demand drivers and leading indicators (Ch. 20), then categorical features such as product attributes (Ch. 22, integer or one-hot encoded), then cluster labels from k-means (Ch. 23). Add one family at a time and re-score each time.
4. 4Start with a small set of features, not everything available. At each node a tree chooses a split from the features it is given, so useless inputs are candidate splits that can fit noise.
5. 5Choose each candidate on a business rationale before you test it. The Super Bowl 'leading indicator' for the economy shows that a feature can correlate in-sample with no causal link. If you cannot explain why a feature should move demand, treat any gain it gives as suspect.
6. 6Score every change on data the model was not trained on. Feature Optimization #1 (Ch. 18) covers three levels: training set, validation set, holdout. Feature Optimization #2 (Ch. 24) covers dataset creation and feature selection. Use the holdout only for the final verdict, so repeated feature trials cannot overfit it.
7. 7Compare train error with test error after each addition. In the overfitting example, 10% error on periods 0–20 came with an RMSE nearly doubled afterward. A large gap means the feature set is memorizing.
8. 8Remove features that do not improve out-of-sample accuracy, then re-test. Use a forest's feature importance (Ch. 16) to shortlist candidates for removal. Where several features are redundant, cap how many a tree sees per split (max_features in forests and extremely randomized trees, the colsample_by parameters in XGBoost). This diversifies the trees, but too many constraints lower accuracy.

The book's quarterly dummy dataset shows what a useful feature does. Each row has four past quarters as X and the next quarter as Y: [5,15,10,7]→6, [15,10,7,6]→13, [10,7,6,13]→11, [7,6,13,11]→5, [6,13,11,5]→4, [13,11,5,4]→11, [11,5,4,11]→9, [7,2,3,1]→1. Test the question 'Is the first demand observation >7?'. The four 'Yes' rows (first value 15, 10, 13, 11) have Y of 13, 11, 11, 9, all ≥8. The four 'No' rows (first value 5, 7, 6, 7) have Y of 6, 5, 4, 1, all ≤7. That one input cleanly separates high from low next-quarter demand, so a tree will keep it. The book also shows a 'bad question', one where the answer tells you nothing about Y, and a feature like that should be dropped. The same logic applies to every candidate input. Keep it only if it separates outcomes on data held out from training.

• If accuracy on the training set is very good but test or holdout error is much worse (RMSE nearly doubled in the book's example), then suspect overfitting. Remove recently added features before adding model complexity.
• If a candidate feature has no plausible causal link to demand (the Super Bowl indicator is the cautionary case), then exclude it however well it correlates in-sample.
• If a feature improves training accuracy but not validation accuracy, then drop it. Judge keep/drop on validation, and reserve the holdout for the final check.
• If many features overlap, then restrict how many each tree may use per split (max_features, colsample_by*) and add trees. If accuracy falls, loosen the restriction.
• If one product's history looks too short to learn from, then pool it with other products and add product-identifying features such as categorical attributes or cluster labels.

• Judging features on the training set. The tell: accuracy keeps rising with each added feature while later-period error worsens, as in the 10% error that nearly doubled in RMSE after period 20.
• Adding spurious 'leading indicators' because they correlate historically. The tell: you cannot state a causal reason, and the relationship disappears out of sample.
• Reusing the holdout set while trying features. The tell: validation and holdout both look excellent, then live forecasts disappoint, because the holdout stopped being independent.
• Feeding every available column and relying on the algorithm to sort it out. The tell: feature importance is spread thinly over many inputs and removing several changes test error very little.
• Over-constraining features per split to fight overfitting. The tell: both train and test error rise, because each tree is too weak to add value to the ensemble.

• ▸Feature / family: name the candidate (extra demand horizon, demand driver, leading indicator, categorical attribute, cluster label).
• ▸Business rationale: one sentence on why it should move demand; leave the row empty if you cannot write one.
• ▸Encoding / construction: how it enters X (raw value, integer or one-hot encoded, k-means cluster label).
• ▸Baseline error: validation error (MAE or RMSE) before adding it.
• ▸New validation error: validation error after adding it, with all else unchanged.
• ▸Train vs validation gap: train error minus validation error; a widening gap signals overfitting.
• ▸Importance rank: rank from the forest's feature importance after the addition.
• ▸Decision (keep / drop / retest): choose from the validation change and the gap, not training accuracy.
• ▸Final holdout error: fill once, only for the final feature set.

1. 1Classify the cause first. Encoding mistakes and errors need a process fix so they stop recurring. Exceptional but real demand (promotions, marketing, spot deals, strange customer behavior, destocking) is cleaned out of the forecast history but not treated as a data error.
2. 2Decide whether to automate. Manual flagging is time-intensive, error-prone and unrewarding, and few planners will do it. The larger the dataset, the stronger the case for an automated rule. Build the rule in Python (pandas/NumPy/SciPy) so it scales beyond what Excel can do.
3. 3Classify each series by demand shape: flat, trended, or seasonal. This choice decides which detection idea you can use, because limits built around the historical mean fail on trended or seasonal series.
4. 4For flat series, set the limits at μ ± z·σ of the demand history. Use z = 2.33 to trim roughly the top and bottom 1% (98% probability inside the range), or 2.58 for a 99% range. A chaotic series (large σ) automatically gets wider limits.
5. 5Replace each observation outside the limits with the nearest limit (clip it). Leave every in-range observation unchanged. Check that a clean series comes out untouched.
6. 6For trended or seasonal series, do not apply limits to the raw demand, because the highest or lowest points are no longer the outliers. Apply the standard-deviation logic to the forecast error instead (Idea #3, 'Error Standard Deviation'). Flag points where the forecast misses by more than z times the error σ.
7. 7Use winsorization (clip at the 1st/99th percentile) only as a blunt baseline. It always cuts the tails, so it creates fake outliers on clean data and under-corrects real ones.
8. 8Validate by re-running the forecast on raw and cleaned history and comparing forecast error. The author reports error reductions of a couple of percentage points from cleaning. Review the list of changed points rather than trusting the rule blindly.

Take Table 10.1, three years of monthly demand ranging from 4 to 17, and Table 10.2, the same data with Y2 M7 changed to 100. Winsorizing at the 1st and 99th percentile gives limits of 4.4 and 16.6 on dataset #1. That alters real points (a 17 becomes 16.6, a 4 becomes 4.4), so it creates fake outliers in clean data. On dataset #2 the limits are 4.4 and 70.9, so the 100 is only cut to 70.9, still far too high. Switching to μ ± 2.33σ gives limits of 3.0 to 17.8 on dataset #1, and no observation changes. On dataset #2 the limits are −22.2 to 47.9. Only the outlier moves, from 100 to 47.9, and the low points are untouched. That is better, but 47.9 is still large. The cost of leaving it shows in the Table 2.3 example: one 100 pushes the average from 9.5 to 18.1 while the median stays at 8.5. A flat-demand forecast would be pulled up badly by it. For seasonal data like Table 10.5, whose outlier sits in Y2 M11, a limit around the overall mean would flag the normal seasonal peaks. The thresholds must therefore be built on forecast error.

• If the series is flat and you want an automated rule, then use μ ± 2.33σ rather than percentile winsorization, because it leaves clean series unchanged.
• If the series has a trend or seasonality, then do not threshold raw demand. Threshold the forecast error (σ of the error), because the seasonal highs and lows are not outliers.
• If an outlier is an encoding mistake, then correct the value and fix the process that created it, so the same mistake does not come back.
• If a spike is real but exceptional (promotion, spot deal, destocking at −80%), then smooth it out of the history, because the forecast should not learn from it.
• If a clean dataset gets modified by your rule, then loosen the threshold (higher z) or change method. A good rule changes nothing when there are no outliers.

• Clipping at fixed percentiles (1st/99th) on every series. Tell: clean datasets show changed points (17→16.6, 4→4.4) and a real outlier is only partly reduced (100→70.9).
• Tightening the winsorization percentile to catch the outlier. Tell: the extreme point shrinks but the clean series gets clipped even more, so you trade one problem for another.
• Applying mean-based σ limits to seasonal or trended items. Tell: legitimate seasonal peaks and troughs are flagged while the true outlier (e.g., Y2 M11 in Table 10.5) can sit inside the limits.
• Assuming one cap is enough. Tell: the 100 is reduced to 47.9, but the remaining spike still lifts the forecast, so check the post-cleaning values and not only the flag count.
• Cleaning silently and never fixing sources. Tell: the same encoding errors recur each period, and nobody has reviewed what the rule changed.

• ▸Item/series ID: the product or SKU the rule applies to.
• ▸Demand shape: flat, trended or seasonal; this selects the method.
• ▸Outlier cause: mistake/encoding error or exceptional demand (promotion, spot deal, destocking).
• ▸Method: winsorization (baseline), σ of demand (flat only), or σ of forecast error (trend/seasonal).
• ▸Threshold z: for example 2.33 for ±98% or 2.58 for ±99%.
• ▸Lower and upper limits: computed values (e.g., μ − 2.33σ and μ + 2.33σ).
• ▸Flagged observations: period, original value and replacement value (the limit).
• ▸Clean-data check: confirm that a series with no outliers is left unchanged.
• ▸Forecast error before vs. after cleaning: compare the chosen KPI on raw and cleaned history.
• ▸Process-fix owner: for mistakes, who corrects the source of the error.

1. 1Write down the business question first: do you want a forecast that matches the demand average, the median, or a deliberately biased low value? The definition of this construct says the KPI you optimize decides which of the three the model aims for. Pick the KPI from the answer, not from habit.
2. 2List the candidate KPIs from Chapter 2: bias, MAPE, MAE and RMSE. Use bias to see whether the forecast is systematically high or low. Use MAE and RMSE as optimization objectives. The passages show these two being swapped in the optimization function and compared.
3. 3Compute the forecast error for every historical period where a forecast exists. In the Excel template this is a |error| column (F4 =abs(C4-B4)) and a squared Error² column (G4 =F4^2). Skip the first two demand observations (rows 2:3) because an error there is meaningless. Copy the formulas down to the end of the historical period only. Never compute error on future forecasts, because there is no demand to compare against.
4. 4Aggregate the errors into the KPI cells. MAE is the average of |error|. RMSE is the square root of the average of Error². In the book's template, J3 holds RMSE and J4 holds MAE.
5. 5Set the KPI as the objective of the optimizer: Excel Solver, or the Python optimization function that selects the best model. The passages state the default is MAE, and that you change to RMSE (or another KPI) by editing a few lines of the function.
6. 6Run the optimization once per candidate KPI on the same demand dataset. Record the selected model and parameters for each. The book warns that as you add models (Chapters 7, 9, 11) and use more complex demand, different KPIs will select different models. Where they disagree, the KPI choice is the real decision.
7. 7Constrain the parameter search while you do this. Do not allow alpha and beta to range over the full 0 to 1, which the book calls a beginner mistake that can cause overfitting. A narrower range forces a less reactive model.
8. 8Lock one KPI as the objective and as the reporting metric for the model. Keep bias alongside it as a check, and re-run the comparison whenever you add models or the demand pattern changes.

The book's own case is the forecast engine built in Chapter 6. The engine tries models on a dummy demand dataset and picks the best one using MAE, and Figure 6.6 shows the model it selects. To test KPI sensitivity, you edit the lines of the optimization function so the objective is RMSE instead of MAE. In the Excel version, you add |error| and Error² columns (F and G) starting at row 4, because the first two observations produce no meaningful error. You then add cells for RMSE (J3) and MAE (J4) and point Solver at one of them. You rerun on the same data and compare the winning model and its alpha and beta. The book's point is that on simple data the two objectives may agree, but as more models and more complex demand are added, they select different models. Whichever KPI you set as the objective therefore shapes the forecast you end up using. The book also tells you to keep the alpha/beta range narrow, since a 0-to-1 range is a beginner mistake that invites overfitting.

• If you want the forecast to track the demand average, then optimize on the KPI that fits that goal and check bias. Do not accept a KPI by default just because it is the template's MAE.
• If MAE and RMSE select different models on your data, then treat the disagreement as a business decision. Chapter 2's discussion of outliers and extreme values is the reference for why they diverge.
• If you add a new model to the engine (Chapters 7, 9, 11), then rerun the selection under every candidate KPI before trusting the previous winner.
• If the optimizer picks a very reactive model, then check whether alpha and beta were allowed the full 0 to 1 range. Narrow the range before blaming the KPI.
• If you are filling error columns in a sheet, then stop at the end of the historical period and start after the first observations that lack a prior forecast.

• Defaulting to MAE (or RMSE) without deciding what the forecast should aim at. The tell: nobody can say whether the model targets the average, the median, or a biased value.
• Computing errors on future periods or on the first two observations. The tell: error cells that reference blank or non-existent demand, or an unexpectedly flattering KPI.
• Running the optimization with one KPI only and assuming the chosen model is the best. The tell: switching to RMSE later changes the selected model and nobody expected it.
• Giving the optimizer a full 0 to 1 range for alpha and beta. The tell: an overly reactive model that fits history tightly and does worse in use, which is the overfitting the book warns about.
• Judging the model on a KPI different from the one it was optimized on, or ignoring bias. The tell: the model scores well on its objective but the forecast sits consistently above or below demand.

• ▸Forecast goal: state whether the forecast should target the demand average, the median, or a deliberately biased low value, and why.
• ▸Candidate KPIs: list which of bias, MAPE, MAE and RMSE you will test.
• ▸Error window: record first and last period with a valid forecast (skip the first two observations; stop at the end of history).
• ▸Error columns check: confirm |error| and Error² are computed only inside that window.
• ▸Alpha/beta range: record the allowed range for each and the reason for it (avoid the full 0 to 1).
• ▸Model selected under MAE: name the model and its parameters.
• ▸Model selected under RMSE: name the model and its parameters.
• ▸Disagreement note: if the two models differ, write which KPI you choose and the business reason.
• ▸Bias check: record whether the selected forecast runs systematically high or low, and the date of the next re-test.