Adam Szymański

Problem 09 · Decision-selection bias

Decision markets answer the wrong question.

The point of a decision market is to ask what happens if we do X. Hanson, who designed the mechanism, admits the answer you get is contaminated. The known fix is to decide at random, which no organisation will accept.

Causal inference Futarchy data now exists Gates the revenue problem

Simple assumes nothing

The useful version of a prediction market is conditional. What happens to revenue if we hire this CEO? You run two markets, one assuming you hire them and one assuming you do not. Only the market matching your real decision pays out, and the other is cancelled with everyone's money returned.

The trouble is who trades in each one. If traders believe you will only hire that CEO when things are already going well, then the "we hired them" market is pricing a world that was already going well. It looks like the CEO causes good outcomes. What it really shows is your decision revealing what you already knew.

The fix that works is to make the decision randomly at least some of the time. Almost no board will agree to that, so the flaw stays in place.

Moderate assumes you know what a market is

This is the gap between correlation and causation appearing inside a mechanism built to produce causal answers. Conditional prices are conditional expectations, and a conditional expectation equals the causal effect only when the decision is independent of the information traders hold. In any real organisation the decision is made using that information, which is exactly the violation.

Hanson has been explicit that this is unresolved. The standard remedy is randomisation, which turns the conditional into a genuine intervention. The blocker there is political rather than technical, which is why the problem has sat still for two decades.

A lot depends on it. Futarchy is decision markets applied to governance, and futarchy is shipping now: Jito's fee switch through MetaDAO, Sanctum routing governance through conditional markets. Real money, real decisions, this bias underneath all of it. It also gates the revenue problem, since a causal answer is precisely what a paying principal wants.

Technical state of the art and the gap

Let D be the decision and Y the outcome. The market gives E[Y | D = d, trader info], and the object a decision maker needs is E[Y | do(D = d)]. These coincide when D is independent of unobserved confounders, and organisational decision processes are built to violate that by design.

Where the open work sits. Full randomisation identifies the effect and is politically dead. Partial randomisation, committing in advance to a small probability ε of deciding at random, recovers identification on the randomised subsample at the cost of ε times the decision quality, and the bias-variance tradeoff as a function of ε has never been worked out. That is a small, tractable problem that would turn randomisation from a non-starter into a dial. Instrumental variables using pre-committed decision rules is unexplored. So is bounding the bias: even without identification, an upper bound on how wrong a conditional price can be would let a principal use it with a stated caveat, which is more than exists today.

The empirical opening is new. On-chain futarchy produces a public record of conditional prices sitting next to realised outcomes. That data did not exist when the objection was first raised, and nobody has gone and looked at it.

Where I would start

  1. Collect the on-chain futarchy record. Jito through MetaDAO and Sanctum both have conditional prices next to outcomes. Start by just looking at it, because nobody has.
  2. Work out the bias-variance tradeoff for partial randomisation as a function of ε. Small, self-contained, and it makes the fix negotiable instead of taboo.
  3. Try to bound the bias without identification. A stated worst case is usable by a real organisation in a way that "this is contaminated" never is.
  4. Hunt for pre-committed decision rules that could serve as instruments. Anything mechanical, like a threshold or a scheduled review, is a candidate.

What counts as a result

A defensible recommendation for ε, or a bound on the bias. Either one makes decision markets usable by organisations that will never agree to randomise fully, which is all of them.

Related