optimal expert elicitation

[gusl]

During an email debate with someone in the last week about the value of Bayesian inference using elicitation, I concluded that there had to exist an area of research aimed at optimally eliciting expert knowledge.

Today, at ML Tea, I chatted with Jay Kadane for the first time, and decided to elicit his opinion on the matter. This turned out to be pretty optimal (notice that he's a co-author of the second publication below).

These were his recommendations:
O'Hagan - Uncertainty in prior elicitations: a nonparametric approach (I think)
Garthwaite, P. H., Kadane, J. B., and O'Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100, 680-701.
balanced, rather philosophical: Barnett - Comparative Statistical Inference

Then I brought up questions about the rationality of disagreement, referencing Aumann's result. Kadane countered that in the real world, people do disagree. At which point, I stopped arguing.
More seriously, his point-of-view was that he's not willing to call irrational anything short of a deductive error. Do Bayesians tend to be more pluralistic and less judgemental than frequentists? I want more data!

--

A proposed experiment illustrating that cognitive science can inform elicitation (though this was not my original intention when I wrote this):

I'd like to propose an experiment: you have a human Observer, who
observes hundreds of samples from some distribution. he does not take
notes, or attempt to memorize them. then, a Scientist comes in, whose
goal is to do density estimation. for whatever reason, the Scientist
can only afford to observe 5 data points himself. now, the Scientist
has a choice to make: should he ask the Observer some questions? the
Scientist knows that Observer's memory is as imperfect as a typical
human's, that his answers will not conform to the axioms of
probability, and on top of all that, he's biased.

Also, http://www.shef.ac.uk/beep/publications.html lists Daneshkhah, A. (2004). Psychological Aspects Influencing Elicitation of Subjective Probability

Further reading:
John Paul Gosling Jeremy E. Oakley and Anthony O'Hagan - Nonparametric elicitation for heavy-tailed prior
distributions Nonparametric elicitation for heavy-tailed prior distributions

---

back to the debating with the frequentist: why elicit at all?

There are a couple of questions here. #1: do people have
*something*like* probability distributions in their mind? #2: if so,
do they tell us anything about distributions in the real world? #3: is
it possible to export that information into the outside world
(crossing the brain-mouth barrier), without destroying its value?
I answer yes to all of these. but only the second and third questions
have normative implications.

If you answer yes to all of the above, I'd say you can't be against eliciting probability distributions.