![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
guslThis Wednesday before my meeting with William, I was at the CS lounge reading a paper, when I unexpectedly heard the words "I need to read up on scoring rules". I promptly got up, and asked who said that.
In the next 3 minutes, I gave Daniel a mini-lesson on the problem of probability elicitation, whose solution is proper scoring rules. When I wrote down the expression that represents the expert's utility maximization, he recognized that the logarithmic scoring rule can be shown to be proper using an identity about cross-entropy:
Let p be the expert's best guess about the probability
Let x be the reported probability
Let U be the utility of reporting x when believes p
In the case the scoring rule is logarithmic, we have:
U_p(x) = p*log(x)+(1-p)log(1-x)
We want to show that the maximum occurs when x=p.
What he recognized is that there is an identical expression for "cross-entropy", and an identity that:
forall x H_p(x) <= H_p(p). (AFAICT, this is not trivial to prove)
From which it trivially follows that p is a maximum.
---
Today, a certain professor got excited with what I told him about prediction markets. Namely:
* terrorism futures
* death-related biases when interpreting the probabilities of catastrophic events
* Google has one
He mentioned the idea of the department using an internal market to predict who's going to get grants.
---
This excellent paper by Robin Hanson unifies scoring rules and prediction markets: Combinatorial Information Market Design
In "Computational Issues", Hanson provides an algorithm for preventing incoherent estimates, i.e. susceptibility to Dutch book, but ... (from p. 114):
Which is to say that if you had a better computer (I want to say "or an efficient SAT-solver", but can't prove that this is reducible to SAT), you could discover arbitrage opportunities, and make money.
In the next 3 minutes, I gave Daniel a mini-lesson on the problem of probability elicitation, whose solution is proper scoring rules. When I wrote down the expression that represents the expert's utility maximization, he recognized that the logarithmic scoring rule can be shown to be proper using an identity about cross-entropy:
Let p be the expert's best guess about the probability
Let x be the reported probability
Let U be the utility of reporting x when believes p
In the case the scoring rule is logarithmic, we have:
U_p(x) = p*log(x)+(1-p)log(1-x)
We want to show that the maximum occurs when x=p.
What he recognized is that there is an identical expression for "cross-entropy", and an identity that:
forall x H_p(x) <= H_p(p). (AFAICT, this is not trivial to prove)
From which it trivially follows that p is a maximum.
---
Today, a certain professor got excited with what I told him about prediction markets. Namely:
* terrorism futures
* death-related biases when interpreting the probabilities of catastrophic events
* Google has one
He mentioned the idea of the department using an internal market to predict who's going to get grants.
---
This excellent paper by Robin Hanson unifies scoring rules and prediction markets: Combinatorial Information Market Design
In "Computational Issues", Hanson provides an algorithm for preventing incoherent estimates, i.e. susceptibility to Dutch book, but ... (from p. 114):
Beyond a dozen or so variables, each with a handful
of values, the above approach quickly becomes infeasible.
After all, when N variables each have V possible
values, there are V N possible states, and 2V N possible
events. When V N is more than a billion or so (e.g.,
thirty binary variables), the above approach becomes
infeasible on today’s computers. In this case some
other approaches must be used to maintain a consistent
set of current market prices, to maintain a description
of the assets each person holds, and to determine when
a user’s assets can allow them to back up a given
new bet.
Which is to say that if you had a better computer (I want to say "or an efficient SAT-solver", but can't prove that this is reducible to SAT), you could discover arbitrage opportunities, and make money.