---
title: The three defects a pretest catches before you field
description: Item wording, scale collapse and segment breakage are cheaper to find before the field than after. What a pretest actually looks for.
---

# The three defects a pretest catches before you field

Published 2026-10-10

A fielded survey returns a number. If the instrument was broken, it returns a confident number, and
the cost of finding out is the whole field. That is the argument for a pretest: not to know the
answer earlier, but to know whether the question can produce an answer at all.

Three defects account for most of what a pretest finds. None of them requires a population to
detect, which is why a synthetic panel can catch them.

## 1. The item does not measure what it claims

An item claims a dimension by what it asks. The claim is often wrong, and it is testable: does the
answer move with the dimension it is supposed to measure, or with something else?

A common case is an item that measures a consequence rather than an attitude. "Do you support
building more housing in your city?" measures a local policy position. "Is housing affordability a
problem in your city?" measures perception of a problem. The two correlate, and the correlation is
the trap: a study that intends to measure support and reports agreement with the second item has
measured something else, with a plausible number attached.

The pretest signal is a mismatch between the spread of answers and the dimension the item was meant
to describe. When the spread looks like a different axis, the item is measuring a different axis.

## 2. The wording pushes the answer

Every item carries a direction. A pretest looks for items where the direction is doing more work
than the content.

Four patterns account for most of it:

- **Loaded framing.** "How much do you worry about the rising cost of housing?" presupposes that
  the cost is rising and that worry is the response. A respondent who disagrees with both
  premises has no place to answer.
- **Unequal scale labels.** "Excellent, good, fair, poor" is balanced in count and unbalanced in
  meaning; the middle is not neutral. Every point needs a label that a respondent can defend.
- **Double-barrelled items.** "Is the service fast and reliable?" cannot be answered when the two
  halves disagree. Split it.
- **Negation.** "Do you oppose not increasing the budget?" costs a working-memory step, and the
  error it introduces is not random.

The pretest signal is a shift of the distribution relative to the same item asked without the
framing. If the wording produces the movement, the wording is the finding.

## 3. The scale collapses

A scale collapses when respondents cannot distinguish between points, or when the points do not
describe a single dimension. The symptom is a distribution piled on one or two positions, and it
usually looks like a strong result.

Two checks catch it. First, is the scale balanced around a true midpoint? A five-point agreement
scale with two agreement points and two disagreement points is balanced; one with four agreement
points and one disagreement point is not. Second, does the scale mix dimensions? "How satisfied are
you with the price and the quality?" cannot be answered on one satisfaction scale, because the two
objects can produce opposite answers.

The pretest signal is a distribution that does not survive a second run with the scale reversed. If
reversing the scale does not reverse the answers, the scale is measuring something else.

## What a pretest does not do

A pretest does not tell you the incidence of a behaviour, the share of a market or the margin of
error of a field. Those require a probability sample. The point of a pretest is narrower and
earlier: to establish that the instrument can produce an answer that means what it claims.

That is a small claim. It is also the one that a field cannot make for you, because by the time the
field answers, the instrument is already paid for.