Applied Multivariate Stats Social Sciences Stevens

1. 1Identify the assumptions your chosen technique requires before touching output: for MANOVA these are independence of observations, multivariate normality, and homogeneity of the covariance matrices (Sections 6.3, 6.6, 6.9); for ANCOVA add homogeneity of regression slopes/hyperplanes (Sections 9.5, 9.10); for repeated measures the multivariate approach avoids the sphericity restriction of the univariate test (Sections 13.4, 13.7).
2. 2Check independence first because it is the most damaging violation — correlated observations inflate the effective sample and bias the F test; if observations are clustered (students within classrooms), route to a multilevel formulation rather than a single-level MANOVA (Sections 6.3, 6.4, 15.2).
3. 3Assess univariate normality on each dependent variable as a screen before testing multivariate normality, since univariate non-normality guarantees multivariate non-normality (Sections 6.5, 6.6, 6.7).
4. 4Test homogeneity of the covariance matrices across groups (the multivariate generalization of homogeneity of variance) — recall the test statistics like Wilks' A depend on pooling within-group variability in W, so unequal covariance structures across groups distort that pooling (Sections 6.8, 6.9, 5.13).
5. 5For ANCOVA specifically, test the assumption of homogeneous hyperplanes on SPSS before trusting adjusted posttest means; a group-by-covariate interaction voids the single adjustment (Sections 9.5, 9.10).
6. 6After fitting, run a residual analysis: build the residual file and check for outliers, influential observations, omitted variables, and ignored nonlinear relationships — the HLM manual explicitly instructs this post-fit check (Section 15.12, Chapter 3.14).
7. 7When a violation is found, choose a remedy tied to the technique: switch to a multilevel model for dependence, use the univariate corrected repeated-measures test when the multivariate assumptions fail, or reconsider ANCOVA with intact groups where assumptions are shaky (Sections 6.4, 13.7, 9.6).
8. 8Cross-validate or validate the model on held-out data so that assumption-driven bias in the fitted equation is exposed rather than hidden (Sections 3.11, 14.10).

Take the three-group MANOVA on the sample problem run in the SPSS control lines with DEP1 and DEP2 across GPS(1,3). The multivariate null hypothesis is tested with Wilks' A = |W|/|T|, where W deviates observations about each group's own means (within-group variability) and T deviates them about the grand mean. Because A pools within-group variability in W, its validity rests on the covariance matrices being homogeneous across the three groups; if group 3's DEP1/DEP2 covariances differ sharply from groups 1 and 2, the pooled W misrepresents error and A is biased. So before reading the A value you would request CELLINFO(MEANS) and inspect each group's spread, then formally test homogeneity of the covariance matrices (Section 6.9). Only once independence, multivariate normality, and covariance homogeneity hold do you interpret the Wilks' A significance test. If covariance homogeneity fails, you consider an alternative multivariate test statistic (Olson, 1974) less sensitive to that violation (Section 5.13).

• If observations are nested or clustered, then independence is violated — abandon single-level MANOVA and formulate a multilevel model (Sections 6.3, 15.2).
• If each dependent variable fails univariate normality, then multivariate normality also fails — screen univariately first and address it before proceeding (Sections 6.6, 6.7).
• If the group covariance matrices are heterogeneous, then prefer a test statistic more robust to that violation rather than defaulting to Wilks' A (Sections 6.9, 5.13).
• If the covariate-by-group interaction (homogeneous hyperplanes) is significant, then do not report a single ANCOVA adjustment of posttest means (Sections 9.5, 9.10).
• If the multivariate repeated-measures assumptions are questionable, then decide explicitly between the univariate and multivariate approach rather than assuming equivalence (Section 13.7).

• Reading Wilks' A significance without checking covariance homogeneity — the tell is unequal group spreads in the CELLINFO(MEANS) output that were never formally tested (Sections 6.9, 5.13).
• Treating clustered data as independent — the tell is a design with natural groupings (students in classrooms) analyzed at a single level, producing an F far larger than justified (Sections 6.3, 15.2).
• Checking univariate normality and declaring multivariate normality satisfied — the tell is no separate multivariate normality assessment at all (Section 6.6).
• Applying ANCOVA to intact groups without testing homogeneous hyperplanes — the tell is adjusted means reported despite a group-by-covariate interaction (Sections 9.6, 9.10).
• Skipping the post-fit residual check so outliers and omitted nonlinear terms stay hidden — the tell is no residual file created, contrary to the HLM manual's explicit instruction (Section 15.12).

• ▸Technique chosen: name the multivariate method and its required assumptions
• ▸Independence status: state whether observations are clustered and how independence was verified
• ▸Univariate normality: record the screening result per dependent variable
• ▸Multivariate normality: record the multivariate-level assessment separately
• ▸Covariance homogeneity: note the test of equal group covariance matrices and its result
• ▸Slope/hyperplane homogeneity: for ANCOVA, record the covariate-by-group interaction test
• ▸Post-fit residual check: confirm residual file built and outliers/influential points inspected
• ▸Remedy applied: state the corrective action taken for any violation found
• ▸Validation: note the cross-validation or model-validation step performed

1. 1Before running any maximization procedure (stepwise regression, discriminant analysis, canonical correlation), compute your n/k ratio — subjects divided by predictors. For social science research, target about 15 subjects per predictor for a reliable, cross-validating equation; treat ratios of 5:1 or less as high-danger zones for substantial shrinkage.
2. 2Distrust the sample R². Recognize that R² = SSreg/SStot measures variance accounted for in THIS sample; it is inflated when predictors are selected from a much larger candidate set, because the procedure seizes on sample-specific correlations.
3. 3Reserve a validation strategy at the design stage, not after: either hold out an independent sample to re-apply the derived prediction equation, or obtain a formal estimate of generalizability (shrinkage) before trusting results.
4. 4Consult the shrinkage relationship: expected loss in predictive power depends strongly on the magnitude of the squared multiple correlation AND on whether predictors were culled from a larger set. Larger candidate pools and modest R² mean expect more shrinkage.
5. 5Do not treat the overall F test for multiple correlation (F = [R²/k]/[(1−R²)/(n−k−1)]) as evidence of generalizability. It can be significant while the equation still fails to cross-validate — significance on-sample is the wrong question.
6. 6When you find patterns you 'wanted to find,' stop and ask whether you predicted them a priori. Post hoc explanations of scattered significant results are the signature of capitalization on chance.
7. 7Apply the equation to genuinely independent data and report the new correlation. If it collapses, the on-sample fit was an artifact and the equation has no scientific utility.

Guttman (1941) tested 136 subjects on 84 predictors — an n/k ratio of roughly 1.6:1, far below the 5:1 danger threshold. On the original sample the multiple correlation was a very impressive .73, the kind of result that 'looks great.' But when the same prediction equation was applied to an independent sample, the correlation dropped to .04 — essentially zero. The .73 was almost entirely capitalization on chance: the maximization procedure had fitted the random peculiarities of those 136 people, not any real population relationship. With 84 predictors and so few subjects, the equation had every opportunity to seize on sample noise. The lesson is direct: the equation had no generalizability, and its purpose — predicting on future samples — was unfulfilled, making it of limited scientific value.

• If n/k is 5:1 or less, then expect substantial shrinkage and refuse to trust the sample R² without cross-validation.
• If predictors were selected from a much larger set, then inflate your expectation of shrinkage further and treat the on-sample fit as optimistic by design.
• If the equation does not predict well on an independent sample, then discard or shrink it — good on-sample fit is not scientific utility.
• If you are running 20-30 tests (e.g., many two-way chi-squares or elements of a correlation matrix), then adopt a stringent per-test alpha (e.g., .001 via Bonferroni) because roughly 5% of tests will be significant by chance alone.
• If results were not predicted a priori and are explained only afterward, then treat them as candidate chance findings until replicated.

• Reporting the sample R² (or multiple correlation) as if it estimates population predictive power. Tell: no independent validation sample and no shrinkage estimate anywhere in the write-up.
• Using the overall regression F test as proof the equation is good. Tell: significance is claimed (e.g., R²=.50, n=50, k=10, F=3.9, p<.01) but nothing addresses whether the equation cross-validates.
• Fishing across many tests and explaining the survivors. Tell: 10-12 significant between-correlations out of 150 tested at .05, then a narrative explaining each — when ~8 significant results are expected by chance.
• Stuffing in predictors because SPSS makes it trivial. Tell: k large relative to n, predictors highly intercorrelated, and no theoretical rationale for the set.
• Confusing a large sample's easy significance with a real, generalizable relationship. Tell: huge n, tiny effect, and no separate check that the pattern holds on new data.

• ▸Sample size (n): record total subjects used to derive the equation.
• ▸Number of predictors (k): count every predictor entered or considered.
• ▸n/k ratio: divide n by k; flag if 5:1 or less, aim toward 15:1.
• ▸Candidate pool size: note whether predictors were selected from a much larger set (yes/no) — if yes, expect more shrinkage.
• ▸Sample R²: record the on-sample squared multiple correlation and label it 'optimistic, unvalidated.'
• ▸Validation plan: state the independent sample or shrinkage-estimation method reserved before analysis.
• ▸Cross-validated correlation: record the new correlation when the equation is applied to independent data.
• ▸A priori predictions: list which significant results were predicted in advance versus explained post hoc.
• ▸Number of tests run + alpha: count total statistical tests and set a stringent per-test alpha (e.g., .001) if many.

1. 1Fix the n/k ratio before you fit anything: count your predictors (k) and require roughly 15 subjects per predictor for social science data. If you have fewer than 5 subjects per predictor, expect substantial shrinkage and treat any reported R as inflated.
2. 2Resist the urge to overfit. Do not dump 20-30 predictors into a stepwise run with only 100 subjects; instead carefully select a small set (aim for 3-5 variables) using substantive knowledge or prior literature before running any automatic procedure.
3. 3Treat stepwise F-tests as rough guides only, never as proof you found the best model. Remember the actual statistic is the largest of several F-ratios, so a nominal .05 can correspond to a real error probability near .50 when 10 candidates compete.
4. 4Run at least one of the three formal validation methods from Section 3.11: (1) data splitting — derive the equation on one random half and cross-validate on the other; (2) the PRESS statistic; (3) the Stein formula (via Herzberg, 1969) for average predictive power across many samples.
5. 5Compare the original-sample R against the cross-validated correlation. If predictive power drops off sharply, declare the equation of limited utility — it has no generalizability and hence limited scientific value.
6. 6Fit and compare several candidate models rather than crowning one. Where different selection methods disagree, note that a subset ignored by one method may be more substantively relevant; ignore variables no method selects.
7. 7Screen for outliers and influential points first (Section 3.14), because a single outlier can dramatically distort the fitted equation and destroy generalizability before validation even begins.

Guttman (1941) demonstrates the danger concretely. He had 136 subjects and 84 predictors — an n/k ratio well under 2:1, far below the 5:1 floor. On the original sample the multiple correlation was a striking .73, the kind of number that tempts an investigator to declare success. But when the prediction equation was applied to an independent sample from the same population, the new correlation collapsed to .04. The .73 was almost entirely capitalization on chance: the regression procedure, a mathematical maximization, had wrung out the last ounce of apparent predictive power by seizing on the peculiarities of that specific sample. With no generalizability, the equation was useless for its actual purpose — predicting on future samples. The lesson: a high in-sample R proves nothing until it survives cross-validation.

• If your n/k ratio is 5:1 or less, then expect substantial shrinkage and do not trust the original-sample R — plan for validation before interpreting anything.
• If you want an equation that cross-validates with little loss, then secure about 15 subjects per predictor for social science research.
• If the cross-validated correlation drops off sharply from the derivation-sample value, then reject the equation as having no generalizability.
• If a stepwise procedure reports a 'significant' F, then treat it only as a rough guide, because sequential selection is data snooping and the F-tests are positively biased.
• If several models fit about equally well, then choose on substantive relevance rather than on the single model a stepwise routine happens to output.

• Reporting the original-sample multiple correlation as the study's result. The tell: no independent sample or PRESS/Stein estimate is mentioned anywhere — the R is a maximization artifact.
• Overfitting by including too many predictors. The tell: k is large relative to n and the model performs badly on any new sample from the same population.
• Believing stepwise found 'the best model.' The tell: a single automatically selected subset is presented with no comparison pool and no acknowledgment that other subsets may be equally or more relevant.
• Taking stepwise significance seriously. The tell: p-values from sequential entry are quoted at face value, ignoring that the largest of several F-statistics does not follow the assumed F-distribution.
• Skipping outlier screening before fitting. The tell: no diagnostics for influential points, so a few split-off cases silently inflate the fit and wreck replication.

• ▸Sample size (n): record total subjects used to derive the equation.
• ▸Number of predictors (k): count every predictor in the final equation.
• ▸n/k ratio: divide n by k; flag if 5:1 or less, target ~15:1.
• ▸Original-sample R: enter the multiple correlation from the derivation sample.
• ▸Validation method used: mark data-splitting, PRESS, or Stein/Herzberg formula.
• ▸Cross-validated correlation: enter R from the independent/held-out sample.
• ▸Shrinkage: subtract cross-validated R from original R; note if drop is sharp.
• ▸Outliers/influential points checked: yes/no and action taken per Section 3.14.
• ▸Selection procedure caveat: note if stepwise used and that its F-tests are treated as rough guides only.

1. 1Set your practical-significance threshold BEFORE collecting data. Decide the minimum raw difference that matters in your context — e.g., 'method A must average at least 5 points higher on achievement' — so statistical significance never gets confused with importance.
2. 2Run the significance test, but treat p < .05 as a gate, not a verdict. When group sizes exceed roughly 200, expect nearly everything to reject the null; a significant p tells you almost nothing about magnitude at that sample size.
3. 3Compute the confidence interval for the difference in means and read its full width against your pre-set threshold. Ask whether the ENTIRE interval clears the practically meaningful cutoff, not just the point estimate.
4. 4Calculate an effect size measure so magnitude is expressed on a standardized scale, and calculate a measure of association (variance accounted for) to see how much of the outcome the effect explains.
5. 5Benchmark the effect size against what other researchers have found in the same content area — a large effect relative to the field's typical findings signals practical significance; a small one, even if significant, does not.
6. 6Calibrate the standard to the maturity of the theory: in areas with little developed theory, accept less stringent standards for what counts as meaningful; in areas with well-developed theory, demand more.
7. 7When many tests are involved, tie practical significance to the multiple-testing problem: only associations strong enough to survive a stringent alpha (e.g., .001 under Bonferroni) are worth calling meaningful, since those are typically the only ones of practical consequence anyway.

You compare two teaching methods and decide ahead of time that one method must average at least 5 points higher in achievement to matter practically. The test comes back statistically significant. But the 95% confidence interval for the difference in population means is (1.61, 9.45). You do NOT have practical significance: although the difference could be as large as 9 or slightly more, it could also be less than 2 — below your 5-point threshold. Because the interval spans values on both sides of what matters, the significant p-value is not enough. You then compute an effect size and compare it to what others have found in the same area before drawing any conclusion about importance.

• If group sizes exceed ~200 and a result is significant at .05, then do not treat significance as evidence of importance — go compute magnitude (CI, effect size, association) before concluding anything.
• If the entire confidence interval lies above your pre-set threshold, then you have practical significance; if any part of it falls below, then you do not.
• If the research area lacks well-developed theory, then accept less stringent standards for practical significance; if theory is well-developed, then demand stronger effects.
• If you are running many tests, then require associations to survive a stringent alpha (e.g., .001), since only strong associations of practical consequence will remain — accept the resulting power loss unless n is large.
• If an effect is small relative to what others have found in the same field, then classify it as not practically significant even when p < .05.

• Equating statistical significance with importance. The tell: a paper reports p < .05 with n > 200 and never mentions effect size, CI width, or variance accounted for.
• Judging on the point estimate alone while ignoring interval width. The tell: 'the difference was 5.5 points and significant' with no acknowledgment that the CI runs down to 1.6.
• Using a threshold decided after seeing the data. The tell: the 'meaningful difference' cutoff conveniently matches whatever the observed effect turned out to be.
• Ignoring inflated overall alpha from multiple tests, so weak, spurious associations get promoted to 'findings.' The tell: 10-12 of 150 between-correlations are significant at .05 (where ~8 are expected by chance) and all are treated as real.
• Applying one universal magnitude standard regardless of field maturity. The tell: a modest effect in an atheoretical area is dismissed, or an effect is called large without any comparison to prior studies in the same content area.

• ▸Pre-set meaningful difference: state the minimum raw difference that matters, decided before data collection
• ▸Group sizes (n per group): note if any exceed ~200 (significance becomes near-automatic)
• ▸Observed difference / point estimate: the raw magnitude found
• ▸p-value and alpha used: record and mark whether adjusted for multiple tests
• ▸Confidence interval (95%): lower and upper bounds; does the ENTIRE interval clear the threshold? Y/N
• ▸Effect size measure: computed value on standardized scale
• ▸Measure of association (variance accounted for): proportion of outcome explained
• ▸Field benchmark: typical effect sizes others report in this content area, and how yours compares
• ▸Theory maturity of area: well-developed vs. limited — sets how stringent your standard should be

1. 1Trace the inference chain backward from your conclusion: state the claim, then ask what analytic decision, then what design decision, then what measurement, each supports it — a break anywhere degrades soundness of the whole.
2. 2Count every statistical test actually performed, including implicit ones (each main effect and interaction in a factorial design, each dependent variable tested separately, each item compared). In a four-way ANOVA that is 15 tests; a two-way design with seven dependent variables tested separately compounds further.
3. 3Bound your overall Type I error before interpreting: for k tests at level a', use ka' (Bonferroni) up to about 10 tests at .05, and switch to 1-(1-a')^k beyond that for a tighter bound.
4. 4Interrogate measurement reliability of every variable feeding the analysis, since fallible data attenuate F ratios and confound interpretation — especially when several dependent variables have widely varying reliabilities and intercorrelations.
5. 5Check whether the data's dependency structure was respected: if observations are nested (students in schools), confirm neither naive aggregation nor disaggregation was used, because both compromise the validity of significance tests and inflate Type I error.
6. 6Discount R-squared and multiple-R for capitalization on chance: apply a shrinkage formula and, critically, substitute for k the TOTAL number of predictors from which the final set was selected, not just those retained.
7. 7Confirm the authors validated their prediction equation on fresh data or via cross-validation, checked outliers and influential points (e.g., Mahalanobis distance), before treating the fitted relationship as a sound inference.
8. 8Reduce dimensionality on principle: pare predictors and dependent variables to the smallest defensible set, since redundant terms yield less precise inferences and a smaller p aids robustness and interpretation.

Schutz (1977) tried to empirically validate a contingency theory of leadership among California principals and superintendents by regressing several measures of administrative success on 24 personality predictors within heterogeneous samples. The theory itself is plausible, but the inference drawn from the empirical validation is not sound. With 24 predictors and modest samples, the multiple R capitalizes on chance, so a large observed R does not represent a real relationship. The correct move is to estimate shrinkage using a formula like Herzberg's, and — following Cohen and Cohen — to use k equal to the full 24 predictors from which selection was made, not the smaller retained set. Using only the retained predictors introduces positive bias and makes the relationship look stronger than it is. The lesson: a conclusion can be undermined not by a flawed theory but by an analysis that ignores capitalization on chance, so the soundness of the inference collapses at the analysis stage.

• If you run more than about 10 tests at .05 (or more than ~30 at .01), then abandon the ka' approximation and report 1-(1-a')^k as your overall-alpha bound.
• If two or three significant results were not predicted ahead of time in a design running many tests, then treat them as probable Type I errors, not findings.
• If your data are nested and the intra-class correlation is nonzero, then use multilevel modeling rather than aggregating or disaggregating, because both distort the significance test.
• If you selected j predictors out of a larger pool of m, then compute shrinkage with k=m (or at least a median between j and m), never k=j.
• If dependent variables are numerous, unreliable, and highly intercorrelated, then combine similar measures or run principal components first, because inferences on fallible, redundant data are misleading.

• Doing multiple pairwise t tests instead of ANOVA — the tell: overall alpha silently climbs toward the Bonferroni bound (e.g., 10 tests → up to .50), so reported significance is spurious.
• Interpreting each effect in a complex design at .05 as if protected — the tell: a four-way ANOVA has 15 tests and overall a ≤ .75, yet the author reports no correction.
• Reporting R-squared or multiple R at face value — the tell: no shrinkage estimate, no cross-validation, and k set to the retained rather than the full candidate predictor count.
• Ignoring nesting by aggregating to the group level or disaggregating to individuals — the tell: school-descriptor variability is artificially deflated (disaggregation) or within-group variation and power are lost (aggregation), inflating Type I error.
• Throwing every available variable into the model — the tell: many predictors with near-zero incremental validity and low-reliability criteria, attenuating F ratios and muddying interpretation.

• ▸Central conclusion: state the single claim whose credibility you are assessing.
• ▸Total tests performed: count every effect, dependent variable, and comparison, including implicit ones.
• ▸Overall-alpha bound: enter ka' or 1-(1-a')^k with the chosen method and value.
• ▸Predicted vs. post hoc: mark whether each significant result was hypothesized in advance.
• ▸Variable reliability check: note dependent variables with low reliability or widely varying intercorrelations to combine or drop.
• ▸Dependency structure: record whether data are nested and the ICC; confirm multilevel modeling was used if needed.
• ▸Shrinkage/validation: enter the k used in the shrinkage formula (full candidate pool) and the cross-validation result.
• ▸Outlier/influence check: record Mahalanobis distances or influential-point diagnostics performed.
• ▸Verdict: sound / conditionally sound / unsound, with the weakest link in the chain named.

1. 1Establish the design before you touch a statistical package. The passages state the quality of the research design is crucial (Section 1.7) and treat a strong design as the prerequisite for meaningful analysis — decide random assignment, control groups, and covariates at the planning stage, not after data collection.
2. 2Distinguish the two validity threats you are protecting. Internal validity concerns whether the effect is real given confounds; external validity concerns whether it generalizes. Map each design choice to which threat it addresses before proceeding.
3. 3Use random assignment to attack internal validity. When you cannot randomize (intact groups), plan an alternative: the text points to ANCOVA with intact groups (Section 9.6) as a way to statistically adjust posttest means and reduce error variance for confounds.
4. 4Budget your Type I error against the number of tests you intend to run, because a maximization design with many effects inflates spurious findings. For k tests at alpha a', apply the Bonferroni bound: overall a <= k(a'). A four-way ANOVA runs 15 tests, giving overall a <= 15(.05)=.75.
5. 5Guard external validity against capitalization on chance. Many multivariate procedures are mathematical maximization procedures that seize on sample properties; predict your key effects ahead of time so unpredicted 'significant' results are flagged as possible Type I errors, not treated as discoveries.
6. 6Check the independence assumption of the general linear model at design time. The text warns a small violation causes actual alpha to be several times the nominal level; if observations are correlated (nested, repeated), plan for the Hierarchical Linear Model (Chapter 15) rather than standard GLM.
7. 7Secure adequate sample size and data integrity as design elements. The author's stated goal is checking data, assessing assumptions, and ensuring adequate sample size 'so that the results can be generalized' — treat these as design deliverables, not post-hoc fixes.
8. 8Screen for conflict of interest in the design phase. Section 1.8 notes financial conflict of interest can be a real problem; document funding sources and analytic decisions before running tests.

A researcher wants to compare several groups on a dependent variable but is unaware of ANOVA and instead runs 10 pairwise t tests, each at the .05 level. The Bonferroni inequality gives overall a <= .05 + .05 + ... + .05 = .50, so the probability of at least one false rejection could easily reach 30-35%, far too high. A stronger design uses one-way ANOVA for the omnibus comparison, controlling overall a. Now scale the problem: a four-way ANOVA (A x B x C x D) automatically runs 15 tests — four main effects and eleven interactions — so overall a <= 15(.05) = .75. If two or three effects come back significant and were not predicted ahead of time, they may well be spurious Type I errors. The design fix is to name the hypothesized effects in advance and treat unpredicted 'wins' as chance findings requiring replication.

• If you cannot randomly assign subjects and must use intact groups, then plan ANCOVA (Section 9.6) with covariates chosen in advance to adjust posttest means and reduce error variance.
• If you will run k statistical tests, then compute the Bonferroni bound overall a <= k(a') at the design stage and decide whether to lower per-test alpha or restrict the number of tests.
• If your observations are correlated (nested, clustered, or repeated), then do not use the standard general linear model — its independence assumption, even slightly violated, inflates actual alpha several times; use the Hierarchical Linear Model instead.
• If your procedure is a mathematical maximization technique, then pre-specify the effects you expect, because these procedures capitalize on chance and threaten external validity — investigators 'find nonexistent patterns that they want to find.'
• If effects were not predicted ahead of time and emerge from many tests, then treat them as candidate Type I errors and require replication before reporting them as real.

• Running many pairwise t tests instead of the correct omnibus test. The tell: overall a is uncontrolled — 10 tests at .05 pushes false-rejection probability toward .50.
• Assuming a sophisticated design or complex statistic removes the multiplicity problem. The tell: a four-way ANOVA quietly performs 15 tests with overall a <= .75, yet the researcher reports each at .05.
• Treating unpredicted significant results as discoveries in maximization procedures. The tell: results were data-driven, not hypothesized in advance, and the pattern conveniently matches what the investigator wanted to find.
• Applying the general linear model to correlated observations. The tell: the nominal alpha is set at .05 but the actual alpha is several times higher because the independence assumption is violated.
• Deferring sample size, data checking, and assumption assessment to after analysis. The tell: results cannot be generalized because adequacy of the design was never verified as a prerequisite.

• ▸Assignment method: state random assignment or, if intact groups, the ANCOVA plan and covariates chosen in advance.
• ▸Internal validity threats: list confounds and the specific design element controlling each.
• ▸External validity threats: note whether the procedure is a maximization method prone to capitalizing on chance.
• ▸Predicted effects: name the specific hypotheses/effects committed to before data collection.
• ▸Number of tests (k): count every effect/comparison to be tested.
• ▸Bonferroni bound: compute overall a <= k(a') and record the per-test alpha chosen.
• ▸Independence check: confirm observations are independent, or specify the Hierarchical Linear Model for correlated data.
• ▸Sample size and integrity: record the adequacy guideline used and the data-checking plan.
• ▸Conflict of interest: disclose funding and any financial conflict per Section 1.8.

1. 1Count k correctly for the procedure you run: in multiple regression k is the number of predictors, in factor analysis it is the number of measured variables, in MANOVA it is the number of dependent variables. Do this before touching the data.
2. 2Divide your available N by k to get the current ratio, then compare against the >15:1 benchmark the book applies to many multivariate procedures.
3. 3If the ratio falls short, solve for the N you need: multiply k by your target ratio (e.g., 15) to get minimum required sample size, and treat that as a recruitment target, not a suggestion.
4. 4Do this at the design stage, not after collection — pair the ratio check with the a priori sample-size determination the book covers in section 5.15 (Power Analysis-A Priori Determination of Sample Size).
5. 5Where N cannot be raised, reduce k instead: cut dependent variables in MANOVA (see 5.14, 'How Many Dependent Variables for a Manova?') or use components as a variable-reducing scheme (11.3) so the ratio recovers.
6. 6In factor analysis specifically, judge the ratio against factor reliability (11.7, Sample Size and Reliable Factors): a thin ratio yields factors that will not replicate.
7. 7Report the achieved ratio alongside results so a reader can gauge stability; do not present coefficients or loadings from an underpowered ratio as if they were reliable.
8. 8Cross-validate when the ratio is marginal — the book uses cross-validation (e.g., 14.10) precisely to catch results that a small sample produced by chance.

Take the reading-methods study from the passages: two teaching methods compared on five criterion variables — syllabication, blending, sound discrimination, reading rate, and comprehension. Here k = 5 dependent variables. Applying the >15:1 benchmark, you need at least 5 × 15 = 75 subjects for stable multivariate results. The t-test illustration in the book uses only 15 per group, i.e., N = 30; run against five DVs that gives a 30:5 = 6:1 ratio, well below the guideline. At 6:1 the discriminant weights and any factor structure would be unstable and unlikely to replicate. The fix is either recruit toward 75+ subjects or trim the DV set (section 5.14) until the ratio clears the benchmark. This keeps you from reading noise as an effect.

• If N/k is below 15:1, then either raise N to k×15 or drop variables before analyzing — do not proceed as-is.
• If the treatment is expensive but data on extra DVs is cheap (the book's reason 3), then still cap DVs by the ratio, because cheap variables inflate k and wreck stability.
• If you cannot raise N, then reduce k via component/factor reduction rather than accepting an unstable model.
• If the achieved ratio is only marginally above 15:1, then cross-validate before trusting the coefficients or loadings.
• If sample size is small, then also loosen alpha (the book notes .10 or .15 is reasonable for small N) rather than pretending the underpowered model is precise.

• Adding 'one more' dependent variable because collecting it is cheap — the tell: k creeps up while N stays fixed and nobody rechecks the ratio.
• Counting k wrong for the procedure (using predictors when you should count DVs, or vice versa) — the tell: the required-N figure looks implausibly low.
• Checking the ratio after data collection instead of at design — the tell: you're now stuck rationalizing an underpowered N.
• Trusting factor loadings from a thin sample — the tell: factors that fail to reappear on a second sample (the reliability warning in 11.7).
• Interpreting a couple of significant multivariate results from a small-N study as real — the tell: they were not predicted ahead of time and, like the multiple-test spurious results the book warns about, are likely type I errors.

• ▸Procedure: name the analysis (regression / MANOVA / factor analysis).
• ▸Variable count k: enter predictors, DVs, or measured variables per the procedure.
• ▸Available N: current or expected subjects.
• ▸Achieved ratio: divide N by k, write as X:1.
• ▸Target ratio: default 15:1 unless justified otherwise.
• ▸Required N: multiply k by target ratio.
• ▸Gap: Required N minus Available N (positive = shortfall).
• ▸Remedy chosen: raise N, cut variables, or reduce with components.
• ▸Cross-validation flag: yes if ratio is marginal (near 15:1).

1. 1Start from substantive knowledge, not availability: list only predictors (or dependent variables) with a theoretical or prior-empirical justification for their link to the criterion. Weisberg: the analyst's knowledge of the substantive area is the single most important tool for selecting a subset.
2. 2Set a target subject-to-variable ratio before you touch the data. With k predictors and n subjects, compute n/k; a larger ratio improves cross-validation prospects and reduces shrinkage.
3. 3Cap the count deliberately. Invoke scientific parsimony and the Lord & Novick point that incremental validity of new variables drops sharply after a certain point because tests overlap in content — a fifth or sixth predictor adds little new relevant to the criterion.
4. 4When you genuinely have many candidate variables in a set, run a components (or rotated factor) analysis and carry forward only the components accounting for most of the variance, rather than all raw variables. This also attacks multicollinearity because components are uncorrelated by construction.
5. 5Fix the variable set a priori. Do not screen the data, then keep the variables where 'all the significance occurs' — that is selection bias the reader can't see.
6. 6Document every test implied by your full original set. If you began with 15 dependent variables in a three-way design, record that 15 x 7 = 105 tests are on the table, and apply Bonferroni against that full count, not the surviving subset.
7. 7Consult Table 3.8 logic before trusting your R^2: shrinkage depends strongly on the magnitude of the squared multiple correlation AND on whether predictors were selected from a much larger set. Selection from a large pool inflates expected shrinkage.
8. 8Validate: because these procedures are mathematical maximization procedures, they capitalize on chance. Cross-validate the pared-down equation to confirm the pattern is not a sample artifact.

Consider the insidious selection-bias case from the passages. An investigator runs a three-way design with 15 original dependent variables, producing 105 = 15 x 7 statistical tests. Each is done at the .05 level, so the Bonferroni inequality guarantees overall alpha is less than 105(.05) = 5.25. If the investigator reports only 7 significant results but conceals that 105 tests were performed, Bonferroni shows that most or all of those results could be spurious Type I errors. Judicious variable selection would have required naming and justifying a small dependent set a priori — say, the 3 variables where significance is confined — before analysis, and reporting the total test count. The same logic applies to canonical correlation: rather than analyze 35 variables with 300 subjects, reduce each set via components to about 10 total, improving the subject-to-variable ratio to 30/1 and yielding more interpretable, stable results.

• If a variable has no substantive or prior-empirical justification and is included only because it is available, then drop it.
• If n/k is small (or you have fewer than ~1,000 subjects with a large variable set), then reduce variables — via a priori selection or components analysis — before running the model.
• If a set contains many intercorrelated variables, then run a components/factor analysis and retain only the components accounting for most of the variance instead of all raw variables.
• If you narrowed a large candidate set to the subset where significance appeared, then report the full number of original tests and apply Bonferroni to that total, treating surviving results as possibly spurious.
• If multicollinearity remains high and the subject-to-variable ratio is small, then consider (canonical) ridge regression, accepting biased coefficients in exchange for lower variance and more stable interpretation.

• Throwing everything in the hopper because variables are available. Tell: predictor list matches the dataset's columns, not a theoretical rationale.
• Prodigious numbers of dependent and/or independent variables (Cohen's complaint). Tell: no stated justification for each variable and an implausibly low n/k ratio.
• Hidden selection bias — reporting only the small subset where all significance landed. Tell: methods describe fewer variables than the design implies; the reader can't see the discarded ones.
• Adding a fifth or sixth predictor expecting real gain. Tell: near-zero incremental validity and heavy content overlap with existing predictors.
• Trusting sample R^2 without accounting for selection from a large pool. Tell: no shrinkage estimate, or shrinkage computed as if predictors were pre-specified when they were data-selected.

• ▸Criterion / DV(s): state the outcome(s) the model must explain.
• ▸Candidate predictor list: every variable considered, including any you will drop.
• ▸Substantive justification per variable: one line of theory or prior evidence linking it to the criterion.
• ▸Final retained set (k): the parsimonious count you commit to before analysis.
• ▸Sample size (n) and n/k ratio: compute and record; flag if the ratio is small.
• ▸Reduction method used: none / components (factors retained) / ridge — and why.
• ▸Total tests implied: variables x contrasts (e.g., 15 x 7 = 105) for Bonferroni.
• ▸Selection-source note: were predictors chosen from a much larger set? (affects shrinkage).
• ▸Validation plan: cross-validation or replication step to check for capitalization on chance.

1. 1Convert raw scores to z scores before anything else, so outliers become visible against a common scale and multivariate normality checks are interpretable.
2. 2Run univariate normality diagnostics on every variable first — checking marginal normality per variable is adequate as a proxy for multivariate normality. Use SPSS EXAMINE to generate the distributions.
3. 3When a variable departs from normality, look at whether one or a few unusual observations (outliers) are the cause, since deviations from normality are often due to outliers; only then decide between removing/handling the point and transforming the variable.
4. 4Plot standardized (studentized) residuals against predicted values. If assumptions hold, residuals scatter randomly about the horizontal line r_i = 0; treat any systematic pattern as a signal to stop and diagnose.
5. 5Read the residual pattern specifically: quadratic clustering means nonlinearity; a fan shape widening as predicted values increase means nonconstant variance; a combined curve-and-fan means both violations at once.
6. 6Apply Box's test for homogeneity of covariance matrices only after normality is achieved on all or most variables, so the test is a clean read on variance differences rather than confounded by non-normality.
7. 7When Box's test is significant with sharply unequal n's, compare the determinants (generalized variances) of the group covariance matrices to determine whether your test will run liberal or conservative before trusting p-values.
8. 8Screen the data outside the modeling program when using LISREL: run raw data through PRELIS to compute the covariance matrix, handle missing data, transform variables, and test normality, then read the clean matrix into the main analysis.

Amlung (1996) reanalyzed Champion and Miller's data in which 527 women responded to items measuring four Health Belief Model dimensions: seriousness, susceptibility, benefits, and barriers. Because LISREL can read raw data but has no facilities for data screening or missing-value handling, the raw data were routed through PRELIS first. In the PRELIS command file, variables were named (up to 8 characters), missing values were coded as 0, all variables were declared continuous (co), and a covariance matrix (cm) was requested and written to the file amlung.cov. PRELIS also computed descriptive statistics, handled missing data, and provided tests of normality assumptions during this step. Only after this screening pass produced a clean covariance matrix was that matrix read into LISREL to test the two competing CFA models. The sequence shows the discipline: screen, handle missing data, and check normality in a dedicated tool before the substantive model ever runs.

• If a variable departs from normality, then first check whether outliers are driving it before transforming — deviations from normality are often due to one or more unusual observations.
• If Box's test is not significant, then homogeneity of covariance matrices is fine and you proceed; if it is significant with equal n's, then power is attenuated so hunt for transformations on the offending variables.
• If Box's test is significant with sharply unequal n's and the larger generalized variance is with the smaller group, then T-squared is liberal; if the larger generalized variance is with the larger group, then it is conservative.
• If the standardized residuals show any systematic pattern rather than random scatter about r_i = 0, then a model assumption is violated and you diagnose (nonlinearity vs. nonconstant variance) before interpreting coefficients.
• If the missing-data mechanism cannot be assumed random (usually the case, per Rao), then do not lean on maximum likelihood imputation as if it fixed the problem — treat post-attrition group equivalence as suspect and test it.

• Trusting the ease of output: because SAS and SPSS make results trivial to produce, practitioners run analyses without checking for outliers or assumptions and get 'garbage' that looks authoritative.
• Ignoring attrition bias: assuming randomly assigned groups remain equivalent after dropout. The tell is a study that began with random assignment but never re-tests equivalence with a multivariate test after attrition.
• Boosting sample size to beat a low response rate — e.g., mailing 2,000 to net 400. The tell is a proud citation of the returned-sample majority while 1,600 nonrespondents (possibly 60% opposed) are never accounted for.
• Reading Box's test before achieving normality, so a significant result reflects non-normality rather than genuine covariance-matrix differences. The tell is a significant Box test on visibly skewed, untransformed variables.
• Treating correlated observations as independent: e.g., 88 essay responses from 22 subjects counted as 88 independent cases when there are only 22. The tell is degrees of freedom or n that ignores the nesting structure.

• ▸Variables converted to z scores: (yes/no — makes outliers and normality comparable)
• ▸Univariate normality per variable: (list each variable + EXAMINE result: normal / skewed)
• ▸Outliers identified: (record case IDs and which variable(s) they distort)
• ▸Transformation applied: (variable + transform used, or 'none needed')
• ▸Residual plot pattern: (random / quadratic / fan / curve+fan — from studentized residuals vs predicted)
• ▸Box's test result: (n's equal or unequal + significant / not; note generalized-variance comparison if unequal)
• ▸Missing data mechanism: (random assumed / not tenable + handling method chosen)
• ▸Attrition equivalence check: (multivariate test run after dropout? groups still equivalent yes/no)
• ▸Independence of observations: (true n vs. apparent n — flag any nesting/repeated measures)

1. 1Identify the three levers that determine power before running the study: the alpha level you set, the population effect size you expect, and the sample size per group. Power (1 - β) is the probability of rejecting H0 when it is false.
2. 2Estimate the effect size you care about detecting. In the passages, Stevens works with a difference of 0.5 standard deviations between groups; commit to a specific value rather than assuming a large effect will appear.
3. 3Consult a power table (Stevens cites Cohen, 1977) to read off power at your planned per-group sample size. For a 0.5 SD difference: n=10 gives power .18, n=20 gives .33, n=50 gives .70, n=100 gives .94.
4. 4If projected power is below roughly .80, act before collecting data: raise n, or use one of the improvement routes (Section 4.10 on ways of improving power), or set a more liberal alpha (.10 or .15) when a Type I error carries no serious substantive consequence.
5. 5Set alpha with the error trade-off in mind: alpha and beta are inversely related, so tightening alpha from .10 to .01 raises beta sharply (in the 15-per-group example, from .37 to .78). Choose the balance that fits the cost of each error type.
6. 6When interpreting a completed study that failed to reach significance, first estimate its power. In a small-n study, a non-significant result does not license 'the groups don't differ' — it may reflect a Type II error.
7. 7For MANOVA and repeated-measures designs, do an a priori determination of sample size (Section 5.15) or estimate power on SPSS MANOVA (Section 4.11) rather than reasoning from univariate intuition.

Suppose you plan a two-group study expecting the groups to differ by 0.5 standard deviations, and you can recruit 20 subjects per group. Reading Cohen's (1977) table as reported by Stevens, power at n=20 is only .33 — meaning if the effect is real, you have a 67% chance of missing it. Raising the plan to n=50 lifts power to .70, and n=100 lifts it to .94, at which point Stevens says power 'is not an issue.' Because 20 per group is small, you might also test at alpha = .10 or .15 to gain power, which is reasonable here since the substantive consequences of a Type I error are minor. If instead you had already run the study at n=20 and found no significant difference, you should not conclude the groups are equal — the low power (.33) makes a Type II error the likely explanation. This is exactly why Stevens argues far more attention should be paid to Type II error.

• If per-group n is 100 or more for a moderate effect, then treat power as adequate and stop worrying about it.
• If per-group n is about 20 or fewer, then be highly sensitive to poor power and consider testing at alpha = .10 or .15 to compensate.
• If a Type I error has no serious substantive consequence (or n is small), then a more liberal alpha is defensible; if a Type I error is dangerous (e.g., declaring an unsafe drug safe), then set alpha very small, perhaps .001.
• If a study with small n returns a non-significant result, then interpret it as inconclusive, not as evidence of no difference.
• If projected power falls short of .80, then increase sample size or apply the improvement methods before collecting data rather than after.

• Reading non-significance as proof of no effect. The tell: a small-n study (n≈20) with p>.05 described as 'the groups did not differ' with no power figure reported.
• Reflexively setting alpha at .05 or .01 regardless of context. The tell: no discussion of the cost of Type II error or the inverse relationship between alpha and beta.
• Ignoring that tightening alpha raises beta. The tell: choosing alpha = .01 in a small study and then being surprised by a null result — beta can climb to .78 at 15 per group.
• Assuming a large effect will show up so sample size doesn't matter. The tell: no committed effect-size estimate before consulting a power table.
• Running many pairwise tests instead of the appropriate omnibus analysis, losing control of false rejections. The tell: ten t tests across five groups at .05 each rather than a one-way ANOVA.

• ▸Expected effect size: state it concretely (e.g., 0.5 SD difference).
• ▸Planned n per group: the number you can realistically recruit.
• ▸Alpha level: your chosen significance level and why.
• ▸Projected power: value read from a power table (cite source, e.g., Cohen 1977).
• ▸Type I cost: substantive consequence if you falsely reject H0.
• ▸Type II cost: substantive consequence if you miss a real effect.
• ▸Power gap action: if power < .80, list the fix (raise n / improve design / relax alpha).
• ▸Null-result interpretation rule: how you will read a non-significant finding given this power.