"anything you can do, I can do better"
Mar. 12th, 2010 12:50 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
guslToday I had a brief chat with Nima at a café on Heather St. He told me that his research is about algorithms to: (a) look at some scatterplots that have been annotated by biologists, (b) figure out how the annotations are made, and (c) outperform the humans...
(b) is a supervised learning problem, in which we construct a descriptive model of human behavior.
(c) requires a prescriptive model, and is impossible to do from this data alone. In NLP, human annotations are normally taken to be the gold standard, because there's nothing better to compare our predictions against.
However, when you consider human imperfection, and the fact that we can use Occam's razor to zero in on the objective of their annotation, it's not so implausible anymore. For example, if you observed that the human annotations seem to be an attempt at circling the densest cluster in the plot, then that's something that computers can clearly do better on. However, the human behavior will have systematic biases, i.e. it deviates from the prescription, which is why the prescriptive model is unlearnable from this data...
However, when the motive behind the annotations is revealed, i.e. when the annotations are used as input to another problem, then our attempt at (c) can finally be evaluated.
See also: programming by demonstration
(b) is a supervised learning problem, in which we construct a descriptive model of human behavior.
(c) requires a prescriptive model, and is impossible to do from this data alone. In NLP, human annotations are normally taken to be the gold standard, because there's nothing better to compare our predictions against.
However, when you consider human imperfection, and the fact that we can use Occam's razor to zero in on the objective of their annotation, it's not so implausible anymore. For example, if you observed that the human annotations seem to be an attempt at circling the densest cluster in the plot, then that's something that computers can clearly do better on. However, the human behavior will have systematic biases, i.e. it deviates from the prescription, which is why the prescriptive model is unlearnable from this data...
However, when the motive behind the annotations is revealed, i.e. when the annotations are used as input to another problem, then our attempt at (c) can finally be evaluated.
See also: programming by demonstration