Much like what happens in the Twilight Zone, we often hear that peer review before publication is an index of quality, that it is there to serve Science. I would argue it's a cookbook.
Of course Bob, Arian and anonymous all have valid points about SL0 but the real issue is that in the pre-publication peer review model, the interesting discussion happens between the authors, some random limited number of reviewers at some random paper and most importantly, it remains secret. The pre-publication peer review process has robbed us all from the only thing we really care about: Insight, the one provided by post publication peer reviewers such as Bob, Arian and anonymous.
As I have mentioned earlier, in the post peer review process, reproducible research is a requisite whereas in a pre-publication peer review process, it is only a "nice to have" feature, something that only virtuous people will embrace.
In the prepublication peer review process, publishers are all too happy to create new journals and provide some sort of market segmentation that yields journals of varying quality bundled in one offering to universities at a steep price. In sum, prepublication peer review is the culprit and for some it's a cookbook.
Let us now imagine a new iteration of this peer review process that now happens post publication. Let us imagine that there is something on top of Arxiv. Clearly, we need something like StackOverFlow where each question is replaced by an article featured on Arxiv. The Q&A sites generally make a difference between providing an answer and providing a comment, much in the same way one would expect a difference between substantive cristicism of a paper and lighter ones. The StackOverFlow model provides a mechanism for recognizing members of that community who can then be seen as experts. Some amount of change could be made so that people, members of the site would be provided with updates on new versions of papers, anonymity for reviewers while preserving their scores, etc....
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.
This arxiv-thing could be even bigger than peer-review - the article would have "tail" of question about difficult places from novices in the area, explanations from author/experts, reports on 3rd party implementations - all blurring the edges between paper, book on the subject and textbook. Would be great!
ReplyDeleteabsolutely, I am even all for novices to be exposed to even crackpot arguments. In a university setting we don't see those often, but when we do, we are generally quite unprepared to them.
ReplyDeleteFor ONLY one additional reading on the topic, I suggest Opinion 101: The Newly launched Journal Rejecta Mathematica IS a JOKE (but so are all math journals!) by Doron Zeilberger: "Let me conclude with a revolutionary proposal to save paper, and disk-space, by making all math journals virtual. Only keep the arxiv, but for each paper, just mention what journal it got accepted to."
ReplyDeleteI also add the June 2011 issue of the IT Society newsletter where A. Ephremides (The Historian’s Column, page 7) recalls that, half a century ago "The papers were FIRST published in the Transactions and THEN they were presented at the Symposium!! That is a total reversal of what we do today. The distinct advantage of this arrangement was that the audience had the benefit of studying the papers carefully ahead of time, which enabled an in-depth discussion after the presentation. Not a bad idea!. This would be a "revolution" indeed, in its radical sense: going back to the same place, althought a bit later.
Don't you think a conference (why not video/virtual?) would be a great post-peer review place?
ArXiv is already great, IMO because it's a great place to monitor new research appearing as opposed to trying to keep track of all relevant journals in the field. Plus, the latter has quite a delay. This extra layer would be a huge leap forward. I can't help thinking; why doesn't "somebody" do that.
ReplyDeleteLaurent, I think I may have read Doron before, but he is spot on. His views fits exactly what is suggested in this entry. Thanks for the reference. I also like the ref to how things were done before.
ReplyDeleteThomas, I agree how come nobody has done this before ?
Igor.
This is in some view very similar to the concept of "working paper" in economics and other fields. From what I understand (and I can't say I understand it completely) economists publish a "working paper" in the community, which is circulated, discussed, cited, presented in conferences etc. The paper is in some sense a living document and keeps changing as new comments/suggestions pile in. This is in some sense what Igor calls "post-publication peer review." Once the paper is in a final good state and accepted by the community, a journal publishes it, which is the official validation of the paper. I am not sure about two things: a) If the journal actively contacts the author and picks up the paper or the author submits to the journal and b) how intensive is the blind peer review process by the journal's editorial process once the validation in the community exists for the working paper.
ReplyDeleteThis is a model that I think works quite well, especially for a slow-moving field such as economics. In our fields, arXiv can definitely serve this purpose. A meta-service might be necessary for posting comments/feedback, although this could in principle be embedded in arXiv. The field of economics shows that this is not even necessary since e-mail, conferences, blogs etc. can play that role. Still, a more formal comment/review/discussion process would be nice. (BTW, integrating google scholar citations with Google+ and maybe blogger would be a great platform for this... One Google to rule us all!)
What is necessary, however, is that the community understands and accepts this process. This might be harder than it seems. We have been having arXiv for so long now, posting papers and citing then, and still people don't accept it. For example, a recent paper I submitted to a conference relied and cited on results in my Universal Scalar Quantization paper on arXiv, before it had been accepted by Trans. IT. One of the comments of one reviewer was "The paper depends heavily on an unreviewed open-source published document [...] I cannot trust the cited work is correct so it must be reviewed before it can be used for a serious paper [...]" and proceeded to trash the remaining of the submission. Thankfully the remaining reviewers saw the value of the paper and accepted it. However, this incident demonstrates the mentality of some in the community and the importance of understanding the process. Still, I am positive that we will find a way to work within this context.
This will have a couple of side-effects and possibly some unintended consequences:
a) The bar for pre-publishing will become lower before the community accepts a paper. This means a higher volume of papers will appear and some personal way to sort through them will be necessary (already the volume of CS-related papers is overwhelming me... the "to-read" pile keeps growing.) This might mean more reliance to personal connections and trust, making the community more closed instead of bringing the openness desired.
b) The paper will actually take longer to be validated in a proper journal, although it will be cited in the meantime. In a fast moving community this might be an issue. This might give more power to the "difficult" conferences, such as NIPS, which are basically following the old review process. If you want your paper appearing faster, just submit to NIPS. Given the lack of true response/second review process in these conferences, I find these conferences to have more quirky and biased reviews than journals.
These are unintended consequences we will need to combat and take into account as we move to a system based on a pre-publication/open comments process. However, I still find the advantages worthy. A wide volume of comments and pre-publications suggestions will pressure an editor to reject a comment of an obnoxious reviewer (or one who has a personal grudge against you) instead of rejecting the paper.
Anyway... sorry for the long rant/comment :-) I hope it contributes something.
Petros,
ReplyDeleteThanks for the nice and long feedback.
Igor.
This is a great idea - someboy should point the folks at stackexchange at it!!
ReplyDelete