Why consultations fail at honesty

Every professional body runs the same exercise. A proposal lands from a regulator or a department, the deadline is six weeks out, and the membership has to be consulted before a submission can honestly claim to speak for anyone.

So a survey goes out. It asks members how concerned they are, on a scale. It offers a free text box at the end for anything else. And it comes back with a number: 68% concerned, say, with four hundred words of comment from the twelve people who had the energy to write any.

That number then has to do an enormous amount of work. It goes into a submission. It gets quoted in a press release. Someone reads it out in a meeting where a decision is made. And at no point does anyone know why those members were concerned, which is the only part a policy team could actually act on.

The scale is not the problem. The stopping is.

There is nothing wrong with asking members to rate something. Ratings are quick, they compare cleanly across years, and some questions genuinely have a scalar answer.

The problem is what happens next, which is nothing. A member selects "somewhat dissatisfied" and the instrument is finished with them. It has recorded a position and discarded the reasoning that produced it. And the reasoning is where the difference sits between two members who chose the same box for opposite reasons: one because the timetable is too fast, one because it is too slow.

Aggregate those two and you get a finding that recommends nothing.

A researcher sitting with either member would not have stopped there. They would have asked the obvious next question, the one that turns a position into an argument: what is it about the timetable? That single follow-up is the entire difference between data and evidence, and it is the one thing a form has never been able to do.

Why the free text box does not rescue it The usual answer is that the comment box catches the reasoning. It does not, for two reasons.

The first is who writes in it. Free text is answered disproportionately by members with the strongest feelings and the most time, which is not the same population as your membership. You are not reading what members think, you are reading what your most motivated members think, and the two diverge exactly where the question is contested.

The second is that nobody follows up. A comment that says "the costs are the issue for us" is a door that never gets opened. Which costs? Falling on whom? Compared with what? The member would have told you. Nobody asked.

So the box produces material that is vivid, quotable, and unrepresentative. It is worse than useless in a submission, because it looks like evidence.

What this costs at the point of decision Associations are not in the market for insight as a nice-to-have. A submission is a formal document that competes with submissions from other bodies, and it is read by people who are entitled to ask how many members are behind it.

If the answer is that four hundred members responded out of fifteen thousand, and that the substantive comments came from twelve of them, the submission is weak. Not because the position is wrong, but because there is no way to show the position is held, or why.

The uncomfortable version of this: the response rate is not an administrative detail. It is the argument. Everything else in the document rests on it, which is why the first thing worth fixing is not the analysis but the asking.

What actually unlocks it Three things change the outcome, and none of them is a better scale.

Ask in a way that permits a reason. A conversation of five minutes gets further than twenty tick boxes, because the second question can depend on the first answer. That is not a technology claim, it is how interviews have always worked. The only thing that has changed is that it is now affordable to do across a whole membership rather than a focus group of nine.

Report the base size next to every figure. Not in a methodology annex. Next to the number. A finding that says "raised by an estimated 26% of 300 responses, 95% interval 21 to 31%" is a claim someone can check. A finding that says "68% of members are concerned" is a claim about fifteen thousand people you did not speak to.

Say what you did not find. Silence on a theme is not disagreement, and a report that treats it as disagreement is making things up. If a cohort is missing from the sample, the honest move is to name them as missing rather than average them away.

The test Take your last consultation report and find any single percentage in it. Then ask two questions: how many people is that drawn from, and what did they say in their own words that led to it being counted that way.

If you can answer both in under a minute, the method is working.

If you cannot, the number was never evidence. It was a summary of a summary, and the reasoning that made it worth anything was thrown away at the moment the member clicked a box and the form said thank you.