Subversive Ventures
← Uncommon Sense
Field Notes

A Seating Plan for Evidence

30 Sept 2026 • David Ashenden

In 1985 Coca-Cola ran nearly 200,000 taste tests on a new formula, and the new formula won. The research answered the question it was given. The company then used the answer to take the original drink off the shelves, a decision the research had never been asked about, and brought it back 79 days later as Coca-Cola Classic. The focus groups had shown the problem in advance. About one participant in ten was angry at the idea of the formula changing at all, even when they preferred the new taste, and their anger shifted the rest of the room. The researchers saw it and treated it as noise.

Researchers have a well-rehearsed argument about presentation: findings delivered as a data dump get filed, and findings told as a story, with the customer as protagonist and the problem as the obstacle, get acted on. Gerry Duffy made the case well in A List Apart in 2024, and I agree with him. Coca-Cola's difficulty sat elsewhere. The story was told, and told to senior people. What failed was the arrangement around it: the question that was commissioned, who was obliged to take a dissenting signal seriously, the room in which it was weighed, and the absence of any record that someone had decided to set it aside.

Those arrangements are governance. In product organisations they are usually left to whoever turns up to the meeting, and I think they deserve the same care researchers give to a discussion guide.

Write the decision down before the question

Research commissioned "to understand our customers better" has no decision attached, and so it has no audience with a reason to listen. Research commissioned because a product lead must choose between two roadmaps by the end of the quarter has both. Coca-Cola's taste tests measured preference. The decision on the table was replacement, and nobody wrote down that those were different questions.

The remedy is a short brief, written before recruitment, naming the decision, the person who will make it, the date, and the evidence that would change their mind. The last item is the one teams skip and the one I find earns its keep. A stakeholder who admits in advance that nothing would move them has told you something worth knowing before a pound is spent.

Different kinds of research also need different sponsors. Foundational work, the diary studies and field visits that describe how people live and work, is often commissioned by one part of an organisation and read by another, months later, by people who never agreed the premise. It needs a senior sponsor willing to hold the question open across a planning cycle. Evaluative work, the testing of a specific design, is usually commissioned by the team that will act on it and needs an owner with authority to change the build that week. Put both on one budget line with one owner and the foundational study tends to report to nobody in particular.

Put the people who decide where they can see the customer

Jeff Bezos has told the story of a meeting in which Amazon's head of customer service was defending a metric showing that callers waited under sixty seconds. Bezos rang the customer service number from the conference room. After more than ten minutes on hold, the metric was no longer the subject of the meeting. His summary, given on Lex Fridman's podcast in December 2023, was that when the data and the anecdotes disagree, the anecdotes are usually right.

What I take from it concerns who was in the room when the phone rang. The executives at that table heard the wait for themselves, and nobody had to prepare a deck to persuade them.

The UK Government Digital Service turned the same instinct into a rule. Its Service Manual says that "everyone should observe at least 2 hours of research every 6 weeks", an idea it credits to Jared Spool's work on exposure hours, and service teams are assessed against the Service Standard by a panel that includes a user researcher. Intuit ran an earlier version from the late 1980s under the name Follow Me Home, sending staff to watch customers install and use its software in their own homes.

Observation is the cheapest governance mechanism I know of. An engineer who has watched two participants fail at the same step needs far less persuading at the readout, and a finance director who sat through one session asks better questions of the business case. It also changes the researcher's position. A researcher who is the only witness holds a great deal of interpretive power and very little formal authority, and when a stakeholder dislikes the conclusion, the lone witness is the easiest thing to attack. Several witnesses who agree on what they saw are much harder to wave away.

Give the sceptic a job

In 1587 Pope Sixtus V formalised the office of the Promoter of the Faith, better known as the devil's advocate, whose task was to argue against candidates for sainthood. John Paul II reduced the role in 1983 as part of a wider reform. He went on to canonise 482 saints, against the 300 or so that Pew Research counts for the previous six centuries. The reform changed the procedure in other ways as well, so the advocate cannot take sole credit for the earlier restraint. For four hundred years, though, the Church gave opposition an office and a fixed place in the process.

Product teams have sceptics already. They tend to work from outside the research and raise their objection in the steering committee after the deck has gone round. Bringing one inside costs little. Invite them to sessions, let them choose a participant, give them the raw notes, and ask them to write the strongest reading against the findings before the readout. Gary Klein's pre-mortem does something similar for plans, asking a team to assume the project has already failed and write down why. An objection raised while watching a customer struggle is cheaper to resolve than the same objection raised once positions have been taken.

The other half of this is what happens to people who bring bad news. Timo Vuori and Quy Huy interviewed 76 Nokia managers, engineers and consultants for their study of how the company lost the smartphone market. Middle managers were afraid of their superiors, top managers were afraid of competitors and investors, and unwelcome information about the company's own weaknesses was softened on its way up. A research function inherits the same dynamic. When findings are met with silence, researchers learn which findings are welcome, and the next study is framed accordingly.

Choose the room and the format together

Findings travel through a handful of venues, each with its own rules. A board pack read the night before rewards one page and punishes nuance. A steering committee wants a recommendation with a cost next to it. A sprint review wants something the team can pick up on Monday. An all-hands will give its attention to a short clip of a customer struggling and not much else. The Slack thread afterwards is often where a finding is settled in practice, by whoever summarises it first.

Format can carry evidence or bury it. On the eve of the Challenger launch in January 1986, Morton Thiokol's engineers sent NASA some fourteen charts and recommended against launching below 53°F, with overnight temperatures forecast well below that. Edward Tufte later showed that the charts listed O-ring damage flight by flight instead of plotting it against temperature, which hid the relationship the engineers were trying to establish. During a break in the teleconference, Thiokol's senior vice president Jerry Mason asked his engineering vice president to take off his engineering hat and put on his management hat, and the recommendation was reversed. Seventeen years later, the board investigating the loss of Columbia examined a Boeing slide titled "Review of Test Data Indicates Conservatism for Tile Penetration" and concluded that its arrangement "served to highlight what management already believed."

I use these cases with some care. They were fatal engineering decisions made under conditions I have never worked in, and it would be glib to reduce them to a lesson in slide design. I use them because the investigations are unusually well documented and they show a mechanism product teams meet at far lower stakes: the evidence was available, and the format and the room determined what was heard.

Amazon's answer was to change the rules of the room. In June 2004 Bezos stopped PowerPoint at senior team meetings in favour of narrative memos, read in silence at the start. Whatever one thinks of the ritual, it settles in advance how evidence will be presented and ensures everyone has read it before anyone speaks.

The order in which venues hear a finding matters as much as its format. A decision owner who first meets an awkward finding in front of their peers will defend their position. The same person, briefed privately the day before, arrives with a response. This is ordinary stakeholder management, and researchers are seldom given the standing to do it.

Keep a record of what was decided

Software architects dealt with a version of this problem some time ago. In 2011 Michael Nygard proposed the architecture decision record: a page or two, kept with the code, setting out the context, the decision, its status and its consequences, with the instruction that all consequences be listed, "not just the 'positive' ones." Its value falls to the person who arrives two years later wanting to know why the system looks the way it does.

Research recommendations deserve the same treatment. Each goes into a log with the question it answers, the evidence behind it, a named owner and a status: accepted, rejected with a reason, or deferred to a date. Rejection is a legitimate outcome and should be recorded as one. Silence does the damage. A quarterly review of the log shows which findings changed the product and which were ignored. It gives the next research brief a track record to cite when it asks for money, and it saves a new product lead from commissioning a study the organisation has already paid for.

Had Coca-Cola kept such a record, the anger in those focus groups would have appeared as a line with a decision next to it. Someone might still have overruled it, and they would have had to write down why.

Account for the recordings

Governance has a narrower meaning that researchers cannot set aside. Participants consented to be recorded for a purpose, and those recordings now pass through transcription services, summarisation models and, in some teams, synthetic users built from earlier studies. Zoom learned in August 2023 how strongly people feel about this, when wording in its terms of service suggested customer content could be used to train AI. Within a week it had rewritten the terms to say it would not use audio, video or chat content for that purpose.

Nielsen Norman Group tested synthetic users in 2024 and found them over-optimistic and eager to please. One said it would finish an online course that real participants admitted abandoning. The firm's advice was to confine them to preparing discussion guides and forming hypotheses, and to keep real participants for the research itself.

The obligations are dull and necessary. Know which tools the recordings pass through, how long they are kept, and whether participants agreed to that. In the readout, mark which claims rest on direct observation and which on a model's synthesis of transcripts. A model's summary of forty interviews is a reasonable first draft, told by a narrator nobody cast, and the people in the room should know which parts came from the participants.

When all this is too much

The case against is strong. Useful research can be fifteen minutes of asking someone to walk you through yesterday and then keeping quiet, as Erika Hall has long argued. Wrap a decision brief, a named sceptic, a venue plan and a log around that and you risk turning a cheap habit into a programme that needs its own business case. In a company of fifteen people the founder is decision owner, sceptic and audience at once, and the right governance is a shared document and a standing half-hour on Fridays.

I think that objection holds for young companies, and for evaluative testing in teams that already have the authority to ship. It fails where research is commissioned by one group and acted on by another, which is where findings go missing. The amount of governance should grow with the distance between the people who ask the question and the people who must act on the answer.

For the next study I would start with one page, written before a single participant is recruited: the decision, its owner and date, what would change their mind, the named sceptic, the venues in the order they will hear the findings, and where the record will be kept.

Sources

← More Uncommon SenseGet new pieces by email