Design, Evaluation, and Analysis of Questionnaires for Survey Research

What it’s about

This book transforms questionnaire design from an 'art' into a scientific activity. Saris and Gallhofer present a complete program: a three-step procedure for operationalizing complex concepts into concrete survey requests, a thorough mapping of the design choices researchers face (response scales, item structure, batteries, data collection mode), and a rigorous framework for estimating the reliability, validity, and method effects of survey questions using multitrait-multimethod (MTMM) experiments. The crowning achievement is the SQP (Survey Quality Predictor) program, built on a meta-analysis of thousands of MTMM experiments across dozens of countries and languages, which predicts the quality of any survey question from its coded characteristics—before it is ever fielded. The authors then show how to use these quality estimates to correct for measurement error in substantive analyses and in cross-cultural comparisons, demonstrating that ignoring measurement quality leads to seriously biased conclusions about relationships and means.

The through-line

Who it’s for

A survey researcher or social scientist who wants to design questionnaires that accurately measure the concepts they care about and yield trustworthy results.

The problem

Survey questions contain measurement error that biases estimates of relationships and means, and there is no easy way to know a question's quality before fielding it. The researcher feels uncertain whether their data truly measure what they intend and anxious that their conclusions may be artifacts of poor questions.

The plan

1. Operationalize complex concepts into concrete requests using the three-step procedure.
2. Make informed choices about response scales, item structure, batteries, and data collection mode.
3. Predict the quality of each question with SQP before fielding and improve weak questions.
4. Estimate reliability, validity, and method effects where possible via MTMM designs.
5. Correct correlations and estimates for measurement error in substantive analysis.

The payoff

The researcher fields higher-quality questions, knows their measurement quality in advance, obtains less biased estimates of relationships and means, and can make valid cross-cultural comparisons.

See our guide

• Measure And Steer The Business With Financial Data →
• The Four-S Discipline — One System, Four Capabilities →
• Measure What Matters →
• Study People And Organizations With Scientific Rigor →
• The Behavioral Science Underneath It All →
• Build A People Analytics Function →
• Design The Organization And Change It →
• Do Research That Holds Up →
• Design Surveys People Answer Honestly →
• The Wisdom Of The Room — Survey Science, And Why Subjective Measures Are Rigorous →
• Understand What Drives Individual Performance →

Related profiles we’ve built

• The basic principle of people analytics learn how to use hr data to drive better outcomes for your business and employees →
• Handbook of Regression Modeling in People Analytics →
• An Everyone Culture: Becoming a Deliberately Developmental Organization →
• Apa Handbook Io V2 →
• Armstrongs Handbook of Strategic Human Resource Management Improve Business Performance Through Strategic People Management →
• Beyond Hr Boudreau Ramstad →
• The Differentiated Workforce →
• Drive Pink →

Additional reading

• Causal Inferences in Nonexperimental Research · Blalock, H. M. Jr. (1964)

  Cited as a foundational text for causal modeling, which is the basis for the structural equation models used throughout the book to analyze measurement quality and correct for error.

• Convergent and discriminant validation by the multitrait-multimethod matrices · Campbell, D. T., & Fiske, D. W. (1959)

  The seminal paper that introduced the Multitrait-Multimethod (MTMM) matrix, which is the core experimental design for evaluating question quality in this book.

• Questions and Answers in Attitude Survey: Experiments on Question Form, Wording and Context · Schuman, H., & Presser, S. (1981)

  The classic text on experimental survey methodology, frequently cited for its split-ballot experiments demonstrating the powerful effects of question wording and format on responses.

• Mail and Internet Surveys. The Tailored Design Method · Dillman, D. A. (2000)

  Referenced as the key source for practical guidance on questionnaire layout, ordering, and administration across different survey modes, topics which the book touches on but does not cover in depth.

• Structural Equations with Latent Variables · Bollen, K. A. (1989)

  Cited as a comprehensive textbook for Structural Equation Modeling (SEM), the advanced statistical technique required to implement the MTMM models and error-correction procedures central to the book's methodology.

• The Psychology of Survey Response · Tourangeau, R., Rips, L. J., & Rasinski, K. (2000)

  Frequently referenced for its comprehensive model of the cognitive processes underlying survey responses (comprehension, retrieval, judgment, and reporting), which provides the theoretical explanation for why question characteristics affect data quality.

Featured in these guides

• The Behavioral Science Underneath It All →Cited source
• Build A People Analytics Function →Cited source
• Measure And Steer The Business With Financial Data →Cited source
• Measure What Matters →Cited source
• Design The Organization And Change It →Cited source
• Study People And Organizations With Scientific Rigor →Cited source
• Do Research That Holds Up →Cited source
• Design Surveys People Answer Honestly →Cited source
• The Wisdom Of The Room — Survey Science, And Why Subjective Measures Are Rigorous →Cited source
• Understand What Drives Individual Performance →Cited source