torpedo reviewing?

John Langford started an interesting and personally dramatic discussion about peer review and raised the possibility of torpedo reviewing, and implications for Internet discussion. The supposed malicious reviewer (or a troll) leaves a comment.

In any case, people seem to agree that the review process has been failing to discuss high-variance papers. I reviewed a moderately high-variance CogSci paper last year ({2,2,4,4}), and those above us failed to start the discussion (it was explicitly their job to do so). The head reviewer was one of the 4, and probably made the call.

JL says that the prevalent system used by Area Chairs to aggregate reviews is a "vetocracy", in which reviewers need to agree unanimously in order for the paper to be accepted, which incentivizes uncontroversial papers. I have no idea how often this is the case, but I'd like to see more controversial papers!

I think our communities are tight, and word does get back to the PC (if they aren't reading the blogs themselves)... but that doesn't mean they will have the resources to ensure good reviewing.

Fernando Pereira favors the system overload explanation
Michael Nielsen on peer review