multi-level probability elicitation

[gusl]

Eliciting probabilities from people is tricky, even when said people are mathematically-educated experts in their field.

You ask them a series of questions regarding the probability of events, marginally or conditionally on some other events.

At first pass, the probabilities they give will typically be incoherent.

Of course, after you point out this incoherency, the expert will try to refine their estimates so as to avoid it... until you show them another incoherency, and so forth.

I can imagine making software that automatically searches for such incoherencies. I would call it "Dutch bookie".

Does this refinement process cause the expert to gradually give us better information about the domain? I like to think so, but this claim should be empirically testable.

This may be relevant: PAPATHOMAS Michail - Correlated Binary Variables and Multi-level Probability Assessments

Comments

[darius.livejournal.com]

Reminds me of Robin Hanson's work on combinatorial info markets. I'm not sure whether I saw mention of that sort of thing in his papers I read around 5 years ago -- seems likely though.