The Outlier Problem: Why Great Ideas Don't Win Votes
Over years of organizing large-scale data challenges — competitions structured much like Kaggle — I watched the same pattern emerge without exception. The wisdom of the crowd produces a reliable, defensible consensus. And then, somewhere in the long tail of submissions, a single outlier solution quietly outperforms everything else.
This observation stayed with me throughout my career. It stayed with me until my last day at IBM, when I finally wrote it down.
The average of many ideas tends to be solid but rarely exceptional. The breakthrough almost always comes from the one idea that very few people recognized — at first.
Most large organizations, when they try to identify where to invest next, do the sensible thing: they poll their researchers, surface the most popular themes, and fund what rises to the top. It is a rational process. It avoids obvious mistakes. But it has a structural blind spot built into it, and that blind spot is where the most important opportunities tend to hide.
What Competitions Teach Us About Consensus
In open data competitions, participants submit predictions or solutions to a shared problem. The aggregate of all submissions — the crowd's consensus — is nearly always better than any individual amateur entry. This is the wisdom of crowds at work, and it is genuinely powerful for filtering out noise.
But the top of the leaderboard tells a different story. The winning solutions consistently come from individuals or small teams who approached the problem from an angle that almost nobody else tried. Their approaches are not popular by definition — they are rare precisely because they require a perspective that most participants do not share.
💡 Key insight: Popularity is a measure of how many people already understand an idea. Breakthrough potential is often inversely correlated with how many people recognize it early.
This means that any system designed to surface the most popular ideas will, by construction, systematically miss the most important ones. It will fund what is already legible. It will overlook what is not yet.
The Consensus Trap in R&D Strategy
Few organisations have invested more seriously in this problem than IBM. Since 1982, IBM Research's worldwide community of top scientists has produced the Global Technology Outlook — a comprehensive annual analysis that identifies disruptive technology trends expected to reshape industry over a three-to-ten year horizon. It is, by any measure, a serious and well-resourced attempt to see around corners.
And yet. Consider two GTO-era bets that looked correct by consensus at the time: blockchain as enterprise infrastructure and genomics as a near-term clinical revolution. Both attracted significant organisational enthusiasm. Both appeared on the radar of many forward-looking institutions simultaneously — which, in hindsight, was part of the problem.
When a technology is popular enough to win an internal poll, it is usually popular enough that every competitor is already looking at it too. The consensus identifies real trends, but it rarely identifies them early enough to matter strategically. By the time something is the most-voted option in a room of the world's best scientists, the window for differentiated advantage may already be closing.
💡 Key insight: The question is not whether a popular idea is real. It is whether being late to a popular idea is worth the same as being early to an obscure one.
Looking at the Long Tail Deliberately
The practical alternative is not to abandon polling or collective intelligence — those mechanisms serve a real purpose. It is to add a second pass that specifically looks at the other end of the distribution.
In practice, this means paying deliberate attention to proposals that are:
- unusually unique in framing — they do not sound like anything else submitted
- surfaced independently by only a small number of researchers
- difficult to evaluate quickly, because the evaluator lacks the relevant context
Most of these will not be viable. The long tail contains a lot of noise alongside the signal. But a focused second-round review by a small expert committee can filter quickly. The few proposals that survive that filter are worth taking seriously — not because they are popular, but precisely because they are not.
💡 Key insight: It is often easier to recognize a great idea than to generate one from scratch. The talent required for a second-round outlier review is recognition, not invention.
The Selection Criteria Problem
There is a second, quieter issue that compounds the first. When organizations share their evaluation criteria with participants in advance — what qualities they are looking for, what dimensions they will score — something predictable happens. People optimize for the criteria.
This is not cynicism. It is rational behavior. If you are told that proposals will be scored on commercial applicability and time-to-market, you will anchor your thinking there. You will not spend effort on ideas that feel too early, too strange, or too hard to justify in those terms — even if those ideas are the most genuinely original ones you have.
The result is a paradox: the more clearly you define what a good idea looks like, the less likely you are to receive one you have never seen before.
💡 Key insight: Keeping evaluation criteria internal does not lower the quality of submissions. It raises the diversity of them — and diversity is exactly what a consensus-heavy system is already missing.
A Small Change With Asymmetric Returns
None of this requires dismantling how organizations currently evaluate research directions. The consensus process does its job well. The proposal is simply to run a parallel track alongside it — one that is explicitly designed to find what the main track is structurally unable to surface.
The cost is low: a secondary review of low-frequency, high-uniqueness proposals by a small committee of lateral thinkers. The potential return is asymmetric: the occasional identification of a direction that nobody else in the industry is funding yet.
That asymmetry is the whole point. Organizations with extraordinary talent and deep research capacity do not need better consensus. They need better mechanisms for recognizing the ideas that the consensus cannot yet see. The crowd will always find the average answer. The question is whether you have a system for finding the one that the crowd missed.
One final thought. This problem does not disappear when the participants are AI agents rather than human researchers. In multi-agent systems where models deliberate and converge on a recommendation, the same dynamic applies: the consensus answer gets recorded, and the dissenting view gets discarded. In a recently submitted paper to ICML, we explored what we call the minority report — a mechanism for systematically preserving the positions that did not win the vote. The intuition is the same as everything above: the agent that was outvoted may have been the one that was right. More on that in a future post.