Reliability and Validity Assessment
One book, placed in its field — what Edward G. Carmines and Richard A. Zeller's book gets right, where it goes further than the rest, and what the rest of the field adds.
Mike West, Editor · Leading With Data · peopleanalyst.com/guides/profile-reliability_and_validity_assessment
Arrow keys or swipe to move · N for notes · F for full screen
Reliability and Validity Assessment
One book, placed in its field — what Edward G. Carmines and Richard A. Zeller's book gets right, where it goes further than the rest, and what the rest of the field adds.
Mike West, Editor · Leading With Data · peopleanalyst.com/guides/profile-reliability_and_validity_assessment
The problem
Validity erodes when it is left to instinct instead of method.
You were taught the moves piecemeal, never the whole model.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
The model
- 01Concept-Indicator Linkage Strength
- 02Item and Instrument Design Quality
- 03Number of Items in Scale
- 04Theoretical Network Embedding
- 05Random Measurement Error
- 06Nonrandom (Systematic) Measurement Error
- 07Reliability
- 08Validity
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
Step 1 of 5 · Concept-Indicator Linkage Strength
Linkage strength is an argument, not a coefficient; you defend it with theory plus discriminant evidence, not fit indices alone.
Misconception
Practitioners believe that if an indicator correlates strongly with the outcomes they expect, it must be tapping the intended concept.
Reality
Predictive correlation can arise from confounds, shared method variance, or a neighboring construct entirely; a strong linkage requires a theoretical argument for why THIS indicator captures THIS concept, not just that the numbers behave.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
Step 2 of 5 · Item and Instrument Design Quality
Random and systematic error come from different design failures and need different fixes at the drafting stage.
Misconception
Practitioners assume that poorly designed items merely add noise that large samples will average away.
Reality
Bad design produces two distinct problems: ambiguous or inconsistent wording inflates random error, but leading, double-barreled, or socially loaded items introduce systematic bias that a bigger sample only estimates more precisely.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
Step 3 of 5 · Number of Items in Scale
Reliability rises with item count only when the average interitem correlation is preserved.
Misconception
Practitioners believe that adding items always increases reliability, and that a high alpha therefore proves a good scale.
Reality
Item count only raises reliability if new items maintain the average interitem correlation; padding with weak or redundant items either drops that average or artificially inflates alpha through near-duplicate wording while adding no new content.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
Step 4 of 5 · Theoretical Network Embedding
Before collecting validity data, list at least three directional predictions your construct makes, including one it should fail if the theory is wrong.
Misconception
Practitioners believe construct validity is a property you establish by correlating your measure with a few conceptually similar variables you happen to have on hand.
Reality
Validity evidence is only interpretable against a prior specification of how the construct should relate to others; without the theoretical network, a correlation of 0.4 is neither confirming nor disconfirming — it is uninterpretable.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
Step 5 of 5 · Random Measurement Error
Random error lowers reliability and attenuates observed correlations, so a null finding may reflect noisy measurement rather than a true zero relationship.
Misconception
Practitioners assume random error biases their results in a particular direction and therefore worry it inflates their findings.
Reality
Because random error has an expected value of zero, it does not bias individual scores systematically — but it does inflate variance and systematically attenuates observed correlations, weakening rather than strengthening apparent relationships.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
The cheat sheet
- Concept-Indicator Linkage StrengthLinkage strength is an argument, not a coefficient; you defend it with theory plus discriminant evidence, not fit indices alone.
- Item and Instrument Design QualityRandom and systematic error come from different design failures and need different fixes at the drafting stage.
- Number of Items in ScaleReliability rises with item count only when the average interitem correlation is preserved.
- Theoretical Network EmbeddingBefore collecting validity data, list at least three directional predictions your construct makes, including one it should fail if the theory is wrong.
- Random Measurement ErrorRandom error lowers reliability and attenuates observed correlations, so a null finding may reflect noisy measurement rather than a true zero relationship.
- Nonrandom (Systematic) Measurement ErrorSystematic error moves scores in a fixed direction, so larger samples and repeated trials will not fix it.
- ReliabilityReliability is the ratio of true-score to observed variance and rises as random error falls.
- ValidityValidity depends on absence of systematic error, so eliminating bias is prerequisite, not optional.
Reliability and Validity Assessment · Mike West, Editor · Leading With Data
Sources
- Reliability and Validity Assessment — Edward G. Carmines and Richard A. Zeller
Reliability and Validity Assessment · Mike West, Editor · Leading With Data