Statistical Modeling and Analysis for Database Marketing Effective Techniques for Mining Big Data

1. 1Fix the target type first. The passages split dependency models into binary targets (RESPONSE) and continuous targets (PROFIT), and Chapter 12 assesses each with its own accuracy, Cum Lift and precision measures. Pick the response-model or profit-model measures before you score anything.
2. 2Hold out unseen data before you fit. Predictive power is performance on data the model has not seen, so score a validation file and never judge on the build sample.
3. 3Report accuracy as error rates for the target type: Chapter 12 has separate sections, 'Accuracy for Response Model' and 'Accuracy for Profit Model'. Treat these as the baseline and do not stop there.
4. 4Run a decile analysis and read Cum Lift. Rank records by model score, cut them into deciles, and compare the cumulative performance of the top deciles to the file average. Do this for the response model and the profit model separately.
5. 5Add precision and separability. Chapter 12 introduces both because decile analysis alone is incomplete. Precision is measured for response and profit models (the profit version uses HL/SWMAD, with SWMAD built in section 12.6.1), and separability is checked for both. The guidelines section pairs Cum Lift, HL/SWMAD and CV. The passages name these measures without defining them, so take the formulas from Chapter 12.
6. 6Bootstrap the decile analysis to test stability. Chapter 13 argues that traditional single-sample decile analysis is weak. Resample, rebuild the decile analysis and Cum Lift across resamples, and read the spread as the stability of the predictions.
7. 7Assess implementation performance and efficiency by bootstrap too. Chapter 13 extends the bootstrap beyond validation to model implementation performance and model efficiency. Use it to compare candidate models on uncertainty, not on one point estimate.
8. 8Check visually that predictions track outcomes. Chapter 3 lists the smooth residual by score groups plot and the smooth actual vs. predicted by decile or score groups plots. Look for residuals with no pattern and actual outcomes that follow predicted ones.

The source's running case is a response model for an investment product, built with logistic regression in Chapter 3. Applying the drill-down, you score a holdout file and compute the response-model accuracy and error rates. You then build the decile analysis and read Cum Lift to see how much the top deciles outperform the file average. Next you add precision and separability from Chapter 12. Then you bootstrap the decile analysis as Chapter 13 describes, rather than trusting a single decile table, and record how much Cum Lift varies across resamples. Finally you plot the smooth actual vs. predicted by decile groups and the smooth residual by score groups. The model earns a predictive-power claim only when Cum Lift is high and also stable across the bootstrap, and when residuals show no systematic pattern. The passages give no numeric results for this case, so you supply the figures from your own data.

• If you have a single decile table and nothing else, then do not claim predictive power; add precision, separability and a bootstrap, because Chapter 13 identifies weaknesses in how decile analysis is used.
• If the target is RESPONSE (binary), then use the response-model accuracy, Cum Lift and precision measures; if it is PROFIT (continuous), then use the profit-model versions, including HL/SWMAD.
• If Cum Lift varies widely across bootstrap samples, then treat the model as unstable regardless of how strong its single-sample lift looks.
• If the smooth residual by score groups plot shows structure instead of scatter around zero, then the model is missing something. Revisit the variables or add an interaction variable (Chapter 8), then reassess.
• If two candidate models look similar on Cum Lift, then use the bootstrap assessment of model efficiency to choose between them.

• Judging on the build sample. The tell is a lift that looks excellent but was never computed on a holdout file.
• Relying on decile analysis alone. The tell is that the report has one Cum Lift table and no precision, separability or resampling evidence.
• Using the wrong measure family for the target. The tell is response-style metrics applied to a continuous PROFIT target, or the reverse.
• Reporting a single point estimate of lift. The tell is no interval or spread, so a reader cannot say whether the next sample would give the same answer.
• Skipping validation shortcuts in the analytic cycle. The passages warn that analysts jump to one step and ignore the rest, including the rough comparisons of efficacy. The tell is a final model with no competing technique to compare against.

• ▸Target variable and type: name it and mark binary (RESPONSE) or continuous (PROFIT), which sets the measure family.
• ▸Validation data: describe the unseen file and how it was separated from the build data.
• ▸Accuracy / error rate: record the response or profit accuracy figure on the holdout.
• ▸Cum Lift by decile: record Cum Lift for the top deciles from the decile analysis.
• ▸Precision measure: record precision for the response model, or HL/SWMAD for the profit model, using the Chapter 12 formulas.
• ▸Separability measure: record the separability result (CV per the Chapter 12 guidelines) for the model.
• ▸Bootstrap spread: record the range of Cum Lift across bootstrap resamples (Chapter 13).
• ▸Visual check: note what the smooth residual by score groups and the actual vs. predicted plots show.
• ▸Verdict and next action: state accept, revise or reject, and the one change you will make if revising.

1. 1Name the business target first. The source says the analyst's job is to predict and explain a target variable such as RESPONSE (binary) or PROFIT (continuous). Choose the one that matches the money decision. Response modeling uses logistic regression; profit modeling uses ordinary regression.
2. 2Straighten the predictors before you trust any lift. Plot the smooth trace of the target against each variable and read the test score (TS). In the source, TS 6 is 'almost' at the cutoff of 7, and TS 3 means no noticeable relationship. Re-express the variable (log for dollar-unit variables, or a GenIQ-evolved structure) until the trace has no serious kinks.
3. 3Check the re-expressed variable on a smooth predicted-vs-actual plot. Points scattered closely around the 45-degree line mean the structure is reliable enough to count as an important predictor. Then keep the best subset of variables, not every candidate.
4. 4Assess the finished model on three axes: accuracy, decile analysis with cum lift, and the two added concepts, precision and separability. Do not rely on decile analysis alone. The source argues that decile analysis is commonly misused.
5. 5Validate with the bootstrap, not a single holdout. Run bootstrap decile analysis validation, then bootstrap assessment of model implementation performance and of model efficiency. This puts a confidence band on the lift you expect to realise.
6. 6Locate the key-drivers by decile with the predictive contribution coefficient (PCC). Rank each variable's PCC within each decile, recode the ranks to 0/1 (non-key-driver/key-driver), and compare each decile's key-drivers with the overall-model key-drivers.
7. 7Tailor the campaign message to the deciles you will actually mail. The source treats deciles 1 through 6 as the typical implementation range. Keep the overall key-driver as the core message and add the decile-specific driver where the key-driver table says so.
8. 8Profile your best customers (descriptive, predictive and look-alike profiling) so the targeting effort is well defined before the model is deployed. The source warns that a flawed targeting effort wastes the model.

The source's PCC key-driver table for a three-variable model (zAGE, zGENDER, zINCOME) shows how model output becomes a marketing decision. AGE is the single overall key-driver. It is also the sole key-driver for the top decile and deciles 3, 7, 8, 9 and bottom. Decile 2 is driven by AGE and INCOME. Deciles 4 and 6 are driven only by GENDER. Decile 5 is driven by AGE and GENDER. The strategic reading: build the inclusive strategy around AGE, then adjust it across deciles 1 through 6. Add INCOME to the AGE message for decile 2. Lead with GENDER, with an undertone of AGE, for deciles 4 and 6. Add a GENDER component to the AGE message for decile 5. The same source shows the upstream quality check. In the 10-point mini case study, re-expressing INCOME as gINCOME raised the PROFIT correlation from 0.763 to 0.894 (a 7.24% gain). Re-expressing AGE as gAGE raised TS from 3 to 6 and r from 0.172 to 0.819. The author calls the 376.2% figure meaningless because the original r was meaningless. Treat a big percentage gain off a near-zero baseline the same way.

• If the target is binary RESPONSE, then build and score a logistic response model. If the target is dollars (PROFIT), then use the ordinary regression profit model. Do not mix the assessment metrics.
• If a variable's smooth trace is kinky or its TS is below the cutoff of 7, then re-express it (log for dollar-unit variables, or a GenIQ structure) before you judge its importance.
• If the original correlation is near zero or meaningless, then ignore the percent improvement after re-expression and judge the new r and TS directly.
• If a decile's key-drivers differ from the overall-model key-drivers, then adjust the message for that decile rather than running one uniform creative.
• If you rely only on decile analysis or a single validation sample, then add precision, separability and bootstrap validation before you claim the model delivers its expected lift.

• Judging success by decile analysis alone. The tell is a good-looking cum lift table with no precision, separability or confidence-interval evidence behind it.
• Quoting a huge percentage improvement from a meaningless baseline, such as the 376.2% gAGE figure. The tell is a strong claim whose starting correlation is near zero (r = 0.172, TS = 3).
• Using raw or standardized coefficient magnitudes to declare 'most important' drivers. The source says neither gives a perfect ranking: units differ, and standardized coefficients misbehave when variables are correlated, as they almost always are in database data. The tell is a driver ranking that changes when the units or correlated inputs change.
• Running one message across all deciles. The tell is the key-driver table showing deciles whose drivers (such as GENDER in deciles 4 and 6) differ from the overall driver (AGE), yet the creative is identical.
• Targeting without a defined best-customer profile. The tell is a campaign aimed at a segment that was never defined predictively or by look-alike, the 'flawed targeting effort' the source warns about.

• ▸Target variable (RESPONSE or PROFIT): state which and why it matches the business decision.
• ▸Predictor straightness: list each key variable, its TS before and after re-expression, and its r before and after.
• ▸Predicted-vs-actual check: note whether the smooth plot hugs the 45-degree line for each kept variable.
• ▸Accuracy, cum lift, precision, separability: record the value of each, not decile analysis alone.
• ▸Bootstrap validation: record the confidence interval on decile lift and the implementation-performance result.
• ▸Overall key-driver and decile key-driver table: list the overall driver and the drivers for deciles 1-6.
• ▸Message adjustments by decile: write the tailored message for each decile where drivers differ from the overall driver.
• ▸Best-customer profile: record the descriptive, predictive and look-alike profile used for targeting.
• ▸Business outcome (user-supplied): enter your own campaign ROI and profit figures. The source passages supply no values for these.

1. 1Start with the data, not a model form. Write down only your content-domain experience and the target (e.g., RESPONSE or PROFIT). Defer any choice of model structure until you have looked at the data. The paradigm is 'Let your data be your guide': the analysis suggests the structures and their forms, and the model follows.
2. 2Look at each candidate predictor against the target before fitting anything. Run the correlation coefficient and a scatterplot. Then add a smoothed scatterplot, because a raw scatterplot of a large file hides the trend, and a correlation coefficient alone only reflects straight-line association. Record the shape you see: straight, bulging, or irregular.
3. 3For a binary target such as response, build a logit plot of each predictor, since the logit is the scale a logistic model works on. Judge whether the plot is straight. Straight data is the working criterion, and you check it explicitly rather than assuming it.
4. 4Where the logit plot is curved, re-express the predictor to straighten it. Move along the ladder of powers in the direction the bulging rule indicates for the curve's bulge. Replot after each re-expression and stop when the plot is acceptably straight. Use the measure of straightness you have chosen so the decision is not just visual.
5. 5Where the bulging rule does not apply (the curve is not a simple bulge), switch techniques. Fit a model and examine a fitted logit plot, or a smooth predicted vs. actual plot, and re-express from what the mismatch shows. The case study's MOS_OPEN is handled this way.
6. 6Treat every pattern you recognize as a partial description, as Tukey puts it, and look beneath it for new insight. If a CHAID tree or other exploratory tool suggests structure, such as an interaction, test it as a hypothesis in the model rather than accepting it outright. Use only simple arithmetic and easy-to-draw pictures where they suffice.
7. 7Run the validity check on the resulting model. If it passes, treat it as final and interpret. If it fails, return to the analysis or the data and look for new structure. Repeat until the model is sound and validated. Expect to iterate; the loop is built into the paradigm.
8. 8Extend the visuals past the exploratory phase. Once the model is implemented, use graphics to describe how the final model behaves for prediction (the purpose of Chapter 14's displays). Cap your effort by a stopping rule you set in advance, because EDA breeds the feeling that the work is never done.

The book's logistic-regression case study shows the sequence in miniature. The candidate predictors include FD2_OPEN, INVESTMENT and MOS_OPEN, and the dependent variable is a response target. First the analyst builds logits and logit plots for each predictor, because the model needs straight data. For FD2_OPEN and INVESTMENT, the logit plots show a bulge, so the bulging rule picks the direction to move on the ladder of powers, and the plots are re-checked until straight. For MOS_OPEN the bulging rule does not apply. The analyst instead uses a smooth predicted vs. actual plot, which shows where the fitted line departs from the smoothed actual trend, and re-expresses MOS_OPEN to close that gap. Only after this straightening are variables ranked by importance (e.g., with the G statistic). The passages give the order of operations but not the specific re-expressions or results, so you must read those from your own plots.

• If a smoothed scatterplot or logit plot is curved with a recognizable bulge, then re-express along the ladder of powers per the bulging rule and replot before modeling.
• If the curve has no clean bulge, then stop applying the bulging rule and use a fitted logit plot or smooth predicted vs. actual plot to guide the re-expression.
• If the correlation coefficient looks weak, then check the smoothed scatterplot before dropping the variable, since the correlation only reflects a straight-line relationship.
• If the model fails validation, then return to the analysis and data for new structures rather than adjusting the model's assumptions.
• If you feel the urge to keep exploring, then stop at your predefined criteria (straight data, validated model); do not chase every possibility, and prefer methods like smoothed plots over stem-and-leaf displays, folding or binning on large samples.

• Choosing the model form first (e.g., a regression with the raw variables) and fitting. The tell: no plots exist in the project before the first coefficients.
• Relying on the correlation coefficient alone. The tell: a variable is discarded or kept with no scatterplot or smoothed plot on file.
• Using small-data tools on a large file. The tell: stem-and-leaf plots, folding or binning that break down, or raw scatterplots that are solid blobs, leading to the false conclusion that EDA does not work with big data.
• Forcing the bulging rule on a curve it does not fit. The tell: re-expressed plots remain curved and the fitted vs. actual plot still diverges.
• Endless exploration driven by analyst insecurity. The tell: new plots keep being added with no stopping rule and no decision made from them; or the opposite, unacknowledged bias, where the analyst assumes EDA is bias-free.

• ▸Predictor and target: name the variable and the target it is plotted against.
• ▸Smoothed scatterplot / logit plot shape: straight, bulging (which direction), or irregular.
• ▸Bulging rule applicable? Yes or no; if no, name the alternative technique used (fitted logit plot or smooth predicted vs. actual plot).
• ▸Re-expression tried: the step taken on the ladder of powers and why.
• ▸Straightness check after re-expression: your measure and the result.
• ▸Structures or interactions suggested by exploration (e.g., from CHAID): list and how you will test them.
• ▸Validity check outcome: pass or fail; if fail, what you will revisit (analysis or data).
• ▸Stopping criterion and decision: the rule you set in advance and whether it is met.

1. 1Fix the dependency structure first. Name the left-hand side (target) and the right-hand side (candidate predictors). The target is binary (RESPONSE) or continuous (PROFIT). If management goals are qualitative or the ideal target is unavailable, record which surrogate you are using. Every importance ranking inherits that choice.
2. 2Screen each candidate against the target pairwise. Compute the correlation coefficient and always read it beside a scatterplot. Then run the smoothed scatterplot and the general association test on the variables the correlation under-rates. The passages treat these as the 'two simple data mining methods for variable assessment', meant to catch relationships a single linear number misses.
3. 3Straighten the data before judging importance. For a binary target, build logit plots and check whether each predictor's logit is a straight line. Apply the ladder of powers and the bulging rule to re-express bent variables. Where the bulging rule does not apply, use the fitted logit plot or the smooth predicted-vs-actual plot. Importance measured on bent data is measured on the wrong variable.
4. 4Assess importance with the G statistic. Compute it for a single variable, then for a subset, then compare the G values of different subsets. Rank the relative importance of the variables from these comparisons and select the best subset. This replaces ranking by p-value alone. The source calls the multivariate-normal random-sample assumption 'virtually untenable' and says the effects of violating the model assumptions are hard to determine with any exactitude.
5. 5Do not rank by the standardized regression coefficient alone. The source flags several common misinterpretations of the regression coefficient and offers the Predictive Contribution Coefficient as the alternative with 'greater utility than the standardized coefficient'. Compute it for each variable in the chosen model and compare its ordering to the standardized-coefficient ordering. Treat every disagreement as a finding to investigate.
6. 6Run a competing selector: GenIQ variable selection. GenIQ is a genetic model, so it is assumption-free and nonparametric. Its only requirement is a fitness function, and the best-so-far model across all generations is the result. Set population size and the reproduction, crossover and mutation probabilities by experimentation, because the source says they are data- and problem-dependent. Compare the subset it finds against the G-statistic subset.
7. 7If the final model has no coefficients (GenIQ), interpret it with the quasi-regression coefficient. The source offers it as an assumption-free alternative frame of reference, including a partial quasi-RC for any model and a version for coefficient-free models. Use it to check that the direction and size of each variable's effect make sense.
8. 8Close with Tukey's discipline. Use several techniques, compare their variability roughly, and judge the final choice on several criteria. The source warns that any single criterion is a 'precise (and thereby inadequate)' surrogate. Keep the selection-effect problem in mind too: its reference list cites multiple-comparison and data-mining correction work (Dunn, Ye).

The source's case study builds a response model for an investment product. Candidate predictors include FD2_OPEN, INVESTMENT and MOS_OPEN. The chapter sequence is the worked path, though the passages give no numeric results, so none are shown here. (1) Plot logits for each predictor against RESPONSE. (2) Re-express FD2_OPEN and INVESTMENT with the ladder of powers and the bulging rule. (3) MOS_OPEN does not yield to the bulging rule, so use the fitted logit plot and the smooth predicted-vs-actual plot to find its re-expression. (4) On the straightened variables, compute G for each variable alone, then for subsets, and compare the subsets' G values to pick the best subset. (5) Re-rank the survivors with the Predictive Contribution Coefficient and note any rank that differs from the standardized coefficient. (6) Run GenIQ variable selection as an independent check and compare its subset to the logistic one. If GenIQ's model is coefficient-free, use the quasi-regression coefficient to read each variable's effect.

• If a variable's correlation with the target is weak but its smoothed scatterplot or general association test shows structure, then keep it as a candidate and re-express it before dropping it.
• If the logit plot of a predictor is not straight, then re-express it (ladder of powers, bulging rule, or the fitted/smooth plots) before computing G or ranking it.
• If the standardized-coefficient ranking and the Predictive Contribution Coefficient ranking disagree, then trust the PCC for importance claims and examine the variable for correlation with other predictors or for a bad scale.
• If the final model is coefficient-free (e.g., GenIQ), then do not report importance from a coefficient. Compute the quasi-regression coefficient instead.
• If you cannot state what the fitness function or comparison criterion rewards, then do not run the selection yet. A genetic model is only as good as its fitness function, and the criterion is always a surrogate.

• Ranking variables by p-value or t-statistic as if the classical assumptions held. The tell: the ranking changes after a re-expression or when one correlated predictor is added or removed.
• Computing importance on unstraightened variables. The tell: a variable with a visibly curved logit plot scores weak, then jumps after re-expression.
• Reading the standardized coefficient as 'importance'. The tell: a large standardized coefficient on a variable whose removal barely changes the model's predictive performance.
• Treating the GenIQ run as the optimum. The tell: only one run, default parameter settings, and no comparison with another technique. The source says even correct settings do not guarantee the best solution.
• Using one familiar technique and one criterion. The tell: the analysis skips Tukey's steps 3, 4 and 7 (competing techniques, rough efficacy comparison, several criteria) and goes straight to optimizing a single surrogate.

• ▸Target and surrogate: name the target (RESPONSE/PROFIT) and any surrogate used, with the reason.
• ▸Candidate variable: one row per predictor.
• ▸Pairwise screen: correlation coefficient, scatterplot shape, and smoothed-scatterplot or general-association-test result.
• ▸Straightening action: logit-plot shape and the re-expression applied (ladder of powers, bulging rule, fitted logit or smooth predicted-vs-actual plot).
• ▸G statistic: value for the variable alone, then for each candidate subset it belongs to.
• ▸Standardized coefficient rank: rank by standardized coefficient, kept for comparison only.
• ▸PCC rank: rank by Predictive Contribution Coefficient, with any disagreement noted.
• ▸GenIQ selection: whether GenIQ retained the variable, plus population size and reproduction, crossover and mutation settings used.
• ▸Quasi-RC and decision: quasi-regression coefficient (direction and size) if the model is coefficient-free, then keep, re-express or drop, with the criterion used.

1. 1Fix a baseline before you touch an advanced tool. Build the standard logistic regression model (LRM) on the original variables. In the worked example, nine original variables were scored with five statistics-based variable selection methods. Every later gain is measured against this baseline, not against zero.
2. 2Define the problem and the target before choosing a technique (steps 1 and 2 of the seven-step cycle). Management objectives are often qualitative, and the right dependent variable may be unavailable, so a surrogate is needed. The technique you know best is not necessarily the best one.
3. 3Specify interactions deliberately. Create the interaction variable as the product of two or more component variables, then test its significance with CHAID. The source calls this an unconventional use of CHAID. Keep the interaction only if the test supports it and the model's predictive power improves beyond what its component variables give.
4. 4Impute missing values with CHAID instead of defaulting to a blanket fill. The source uses CHAID as a method for filling in missing values. Run the model with and without the imputation and compare, so the imputation has to earn its place.
5. 5Let a machine-learning method propose variables, then hand them to the traditional model. In the source, GenIQ built new variables from the original nine, and the enhanced best subset reached fifteen variables. Refit the same five LRM selection methods on that subset. This hybrid GenIQ-LRM is still a linear approximation of a possibly nonlinear response function.
6. 6Test the assumption-free model itself. The full GenIQ Model, meaning the entire tree with all its structure, is a nonlinear superstructure. Score it against the hybrid GenIQ-LRM, not only against the original LRM.
7. 7Judge every comparison at the actionable depth-of-file, the top four deciles, as well as across the file. Report the range of gains across methods and the mean gain. Do not report only the best case.
8. 8Give yourself a frame of reference for coefficient-free models. A model with no regression coefficient has nothing to interpret. The source supplies a quasi-regression-coefficient (quasi-RC) as an assumption-free alternative, so you can explore ideas like GenIQ without losing interpretability.

The source's response illustration starts with nine original predictor variables and five statistics-based variable selection methods, each feeding a logistic regression model. The average performance of these five LRMs is AVG. GenIQ then creates new variables. Some correlate with response more strongly than the best original predictor, and two fall between the second- and third-best originals. This yields an enhanced best subset of fifteen variables. The five methods are rerun on that subset, giving AVG-g. Percentage gains of AVG-g over AVG range from 6.4% down to 0.7%, with a mean of 3.9% in the top four deciles. The GenIQ Model itself, the full nonlinear tree, is then compared with the hybrid GenIQ-LRM. Gains range from 7.1% to 1.2%, with a mean of 4.6% in the top four deciles. The sequence is baseline, then ML-enhanced variables inside the familiar model, then the ML model itself. Each stage is measured against the one before it, and the smallest gains are reported alongside the largest.

• If you suspect two variables act jointly, build their product as an interaction variable and test it with CHAID. Keep it only if the test supports it and the model's predictive power improves.
• If a variable has missing values, fill them with CHAID-based imputation and compare against the unimputed model before adopting it.
• If the LRM is a linear approximation of a response you suspect is nonlinear, then test the GenIQ-enhanced variable subset first, and the full GenIQ Model second.
• If an advanced model's gain shows only across the full file, then check the top four deciles. That is the actionable depth, and the gain must hold there.
• If you adopt a coefficient-free model, then compute a quasi-RC so stakeholders have an interpretable frame of reference.

• Choosing the technique you are most comfortable with and stopping there. The tell is that no competing technique was run, so steps 3 and 4 of the cycle were skipped.
• Treating machine-learning output as a replacement for the baseline. The tell is that no AVG versus AVG-g style comparison exists, so you cannot say what the new method added.
• Reporting only the best gain. The tell is a single headline figure with no range. The source's gains run from 6.4% down to 0.7% for variable selection and from 7.1% down to 1.2% for the model.
• Declaring discovered structures significant. The tell is a list of CHAID or GenIQ findings treated as final. The source says structures are not final results and should be checked for sense, natural grouping, and consistency on fresh samples.
• Using a coefficient-free model with no interpretive frame. The tell is that nobody can explain the model's variable effects, so the team distrusts it or cannot act on it.

• ▸Target variable and any surrogate: state the dependent variable and why it was chosen.
• ▸Baseline model and variables: list the LRM and the original variables (the AVG reference).
• ▸Interaction candidates: list the component variables whose product you will form.
• ▸CHAID interaction test result: note whether the test supports keeping each interaction.
• ▸Missing-data variables and imputation method: list each variable with missing values and whether CHAID imputation was applied.
• ▸Enhanced variable subset: record the variables added by GenIQ or another ML method and the new subset size.
• ▸AVG vs AVG-g gain: record the range and the mean percentage gain in the top four deciles.
• ▸Full GenIQ Model vs hybrid GenIQ-LRM gain: record the range and the top-four-decile mean.
• ▸Quasi-RC or interpretation note: record how the coefficient-free model's effects are explained.

1. 1Name the target type first. Response (binary) and profit (continuous) models get different assessment tools: the source pairs Precision with response models and SWMAD with profit models, and applies Separability to both. Choose metrics from that pairing, not from habit.
2. 2Run the traditional decile analysis and read Cum Lift as the baseline. Keep it as one view, not the verdict. The source treats it as the traditional concept and adds two others to sit beside it.
3. 3Add Precision. For a response model, check how closely predicted response matches actual response. For a profit model, build SWMAD, the source's precision measure for continuous targets. Report it next to Cum Lift.
4. 4Add Separability. Check whether the model separates the groups (deciles) from one another, not just whether the top decile looks good. Apply it to both response and profit models.
5. 5Replace the single hold-out with a bootstrap validation. Draw repeated bootstrap samples, recompute the decile analysis on each, and report Cum Lift as an interval rather than a point. The source's Chapter 13 sequence is: the problem with traditional validation, three questions, how to bootstrap, bootstrap decile analysis validation, then bootstrap assessment of implementation performance and model efficiency.
6. 6Use the bootstrap interval to judge stability. A decile whose Cum Lift swings widely across samples is not a reliable rank-ordering, even if one hold-out showed it as strong.
7. 7Apply the guidelines section: use Cum Lift, HL/SWMAD and CV together, not one alone. Settle which metric governs the decision before you look at results.
8. 8Treat every metric as a surrogate. Tukey's steps, quoted in the source, say explicit criteria are hard to define, so precise surrogates stand in. The final step is comparison on several criteria, so make the final call on several criteria, not one.

The source's market-segment case (Chapter 9) shows the single-hold-out baseline this construct improves on. The author draws a fresh hold-out of 5,000 and compares actual and predicted segment counts. Actual: 650 Hassle-free, 1,224 Service, 1,916 Price, 1,210 Features. Predicted: 655, 1,268, 1,870, 1,207. Both columns sum to 5,000. The shares are 13.0% actual vs 13.1% predicted for Hassle-free, 24.5% vs 25.4% for Service, and 38.3% vs 37.4% for Price. Features is 24.2% by actual count (1,210 of 5,000) vs 24.1% predicted (1,207 of 5,000). The author concludes there is no bias. That check covers totals only, and it rests on one hold-out, introduced with the words 'without arguing the pros and cons of validation procedures.' Under the rigorous approach you would add two things. First, bootstrap the check, so you see whether the Service gap (+44 customers) and the Price gap (−46) persist across resamples or are noise from this one draw. Second, add precision and separability metrics, because matching marginal counts does not show that individual customers are placed in the right segment.

• If you have only one hold-out result, then treat it as a single draw, not proof, and bootstrap before you sign off on the model.
• If Cum Lift looks strong but you have no precision or separability figure, then withhold approval. The source introduces those metrics because decile analysis alone is incomplete.
• If the target is continuous (profit), then use the SWMAD-based precision measure; if binary (response), then use the response precision measure. Do not swap them.
• If bootstrap intervals for adjacent deciles overlap heavily, then do not claim the model rank-orders at that granularity.
• If the metrics disagree (good Cum Lift, weak separability), then do not average them away. Follow Tukey's last step and decide explicitly across the several criteria.

• Treating one hold-out as validation. The tell: the report gives a single Cum Lift number with no interval and no repeat draws.
• Reading Cum Lift as accuracy. The tell: top-decile lift is quoted while predicted-versus-actual agreement and decile separation are never shown.
• Using a response-model metric on a profit model, or the reverse. The tell: a continuous PROFIT target is assessed with a binary-style measure and no SWMAD.
• Checking only matching totals. The tell: the actual and predicted marginals agree, as in the segment table, but nothing shows case-level correctness.
• Letting a precise surrogate stand in for the real goal. The tell: one convenient criterion is optimized, with no comparison across several criteria.

• ▸Target type: binary (response) or continuous (profit); this fixes which precision measure applies.
• ▸Decile analysis / Cum Lift by decile: list the baseline figures.
• ▸Precision measure: response precision, or SWMAD for profit, with its value.
• ▸Separability measure: record the value and what it says about decile-to-decile separation.
• ▸Validation method: single hold-out or bootstrap; state the number of bootstrap samples.
• ▸Bootstrap interval for Cum Lift (top deciles): lower and upper bounds.
• ▸Stability verdict: note whether adjacent decile intervals overlap.
• ▸Metric conflicts: list any disagreement among Cum Lift, precision and separability.
• ▸Final decision and criteria used: state go/no-go and the several criteria it rests on.

1. 1Score the data first. Append the predicted logit to every record, then convert it to a predicted probability of response. Every later diagnostic is built on these two columns.
2. 2Compare candidate variable subsets with the G/df rule (Section 3.9.4). Divide each model's G (likelihood-ratio chi-square) by its degrees of freedom and keep the subset with the larger value. A smaller subset that wins on G/df is the better-specified model.
3. 3Build the smooth residual by score groups plot. Define score groups from the unique value combinations of the model's predictors. Plot the mean residual against the mean predicted response for each group. Do not treat variables that nest inside another (MOS_DUM inside MOS_OPEN) as extra dimensions.
4. 4Test the residual plot against the null plot with the General Association Test from Chapter 2. Do not rely on eyeballing, which the source calls subjective because analysts tend to see what they want to see.
5. 5If the plot is declared null, still scan for local patterns. A small wave of smooth points with no general trend marks a weak spot: a set of score groups where predictions are biased even though overall prediction is good.
6. 6Build the smooth actual vs. predicted by decile groups plot. Rank the scored data in descending order of predicted response and cut it into ten equal groups. Plot the mean actual against the mean predicted for each group, and draw the 45-degree line. Points should hug the line in proper order (top to bottom).
7. 7Quantify the hug with the correlation coefficient between the ten smooth actual and predicted points. A larger r means less scatter. Use it only to compare candidate models, because an r from 'big' points is a gross estimate of individual-level fit.
8. 8Handle weak spots at implementation. If the scoring process allows exception rules, assign the biased groups manually: for example, always target new customers (top decile) and place expired customers in the middle deciles. Otherwise revisit the structure, as EDA prescribes, until the model passes validation.

The case study models TXN_ADD with logistic regression. The starter five-variable subset (MOS_OPEN, MOS_DUM, FD1_RCP, FD2_RCP, INVEST_LOG) has G/df = 201.031/5 = 40.21. The three-variable subset (FD2_RCP, MOS_OPEN, MOS_DUM) has G/df = 186.058/3 = 62.02. By the Section 3.9.4 decision rule, the three-variable model wins and becomes Equation 3.16: predicted LGT_TXN = 0.54164 + 1.4942*FD2_RCP – 0.3507*MOS_OPEN + 0.9249*MOS_DUM. Its score groups are 18 (three FD2_RCP values times six MOS_OPEN values; MOS_DUM adds none because it is nested in MOS_OPEN). The smooth residual plot (Figure 3.11) is declared equivalent to the null plot by the General Association Test, so overall prediction is good. Four score groups in the lower right, however, have clearly negative smooth residuals, which means the model over-predicts TXN_ADD there. The smooth residuals range from –0.26 to 0.16 (range 0.42, standard deviation 0.124). The source treats this as a weak spot to be handled with exception rules, and records the statistics for later comparison.

• If two candidate subsets differ in size, then compute G/df for each and keep the larger. The three-variable model (62.02) beat the five-variable model (40.21).
• If the General Association Test says the smooth residual plot is null, then accept overall specification as adequate. Still inspect the plot for local patterns before signing off.
• If a cluster of adjacent score groups shows same-signed residuals, then declare a weak spot. Either add an exception rule at implementation or revisit the structure.
• If the decile points hug the 45-degree line in proper order, then predictions are good on average. If the order breaks, treat the model as mis-specified even if r is high.
• If you are comparing decile-plot correlation coefficients, then use them only to rank models against each other. Do not report them as individual-level accuracy.

• Judging the residual plot by eye. The tell: you 'see' randomness without running the General Association Test.
• Declaring victory on a null residual plot and missing a local pattern. The tell: a cluster of same-signed smooth points, like the four negative score groups, goes unexamined.
• Keeping more variables than needed. The tell: a larger subset has a lower G/df than a smaller one, as the five-variable model (40.21) did against the three-variable (62.02).
• Reading the decile-plot r as individual-level accuracy. The tell: a high r from 'big' points is quoted as if it measured record-level fit.
• Never revisiting the structure after a failed check. The tell: the model is finalized without iterating on structures, contrary to the EDA loop of revisiting analysis and data until a validated model emerges.

• ▸Candidate subsets: list each variable set under consideration.
• ▸G and df per subset: record the likelihood-ratio chi-square and its degrees of freedom.
• ▸G/df per subset: compute G divided by df; the winner is the larger value.
• ▸Chosen model equation: write the final logit equation with its coefficients.
• ▸Score groups: list the unique predictor-value combinations and their count (excluding nested variables).
• ▸General Association Test result: record whether the smooth residual plot is null or not.
• ▸Local pattern / weak spot: note which score groups show same-signed residuals and the direction of bias.
• ▸Decile plot check: record whether points hug the 45-degree line in order, plus the smooth-point correlation r.
• ▸Action: choose an exception rule, a structure revision, or acceptance, with a one-line rationale.