Blowing Up the Peer-Review Bubble

[ See also the problem with peer-review]

Noam Nisan perfectly nails it in "The Problem with Journals" and it's not open access.

...I’d like to answer this explicitly and talk about the problem with the journal publication system, even assuming that we’ve completely solved the open-access and cost issues:  Journals are simply not fulfilling their main three functions: dissemination, verification, and allocation of attention.

1. Dissemination:
 While originally the main point of a print journal was so that Prof. A. can see the results of Prof. B. relatively quickly, it is clear that, in the age of the Internet, journals only slow dissemination compared to, say, putting stuff on the arXiv.
2. Verification:
 Despite pretenses, refereeing is not really trust-worthy. Results of some importance become believed not when refereed but rather only after the community has studied them for a while.
3. Allocation of attention
: an important goal of leading journals is to filter the “important” papers out of all the submitted ones, so that readers need not read everything but rather only the important stuff. I am afraid that today so much is published so that most of what one reads in most journals should have been filtered out. Partially this is a problem of the publish-or-perish culture and partially due to the coarseness of the refereeing model as a filtering tool.

All three of these main goals can be improved upon considerably using the right tools (that need to be figured out) on the Internet. At the same time that the journal system has lost its usefulness, it has created a lot of harmful side effects: the writing of countless worthless papers, lack of recognition for surveys, books, or other non-”paper” contributions, blind and silly use of metrics like impact factors for hiring, grants and promotion which lead to wasteful optimization of these rather than of real research. All these harmful side-effects could be tolerated had the system served its main purpose — but now we are just paying the price but not getting the goods.

In our community where results can be reproduced relatively cheaply, the peer-review process is hindering our ability to access knowledge and instantly build on that knowledge. So far, I know of three kind-of important papers that have been significantly delayed by peer-review. None of these papers have been noticeably improved by the process when finally published.

I am also surprised that some papers do not start a bidding war between publishers after having landed on arxiv as preprints.

Because the peer review process acts as a such conservative filter, there is always a temptation to seek out editors and run a new "Journal of Compressive Sensing" (I have seen this in other fields). The result is near immediate: worse papers get to be published in these new publications while increasing the publisher's portfolio who can then increase its very expensive bundling proposition to university libraries. Opposite to this, Nuit Blanche provides little filter outright but provides some allocation of attention. I personally think that eventually there is a virtuous cycle of consistently good or OK papers in the community as a result.

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.