At the moment, these are just disjointed thoughts and quotes about refereeing for math journals. I hope to flesh it out soon.
For the last couple of years, I’ve refereed an average of 6–8 papers a year for various mathematical journals. This is not a very large number, but it takes a significant amount of my mathematical time and energy, and it is growing over time. What I’d like to do here is explain what refereeing is, why I think it’s important, what I think about the process, and how I personally go about it. I also hope to include a short discussion of the ethics of refereeing (just as soon as I decide whether this particular page is ethical!).
(expected) structure of this essay:
- the point of refereeing
- responsibilities of the referee
- what benefits derive?
- my goals as a referee
- structure of a report
- ethics of refereeing
A very interesting page on “Why I sign my referee reports”.
I’m not planning a real rant here, but if you want to see one, check out “Journals, Conferences, and Referees, or why ‘modern’ ‘professional’ Science is organized in a completely screwed up Victorian amateur way”. Hoo boy.
The correctness of the paper is not my professional responsibility. (link to, or quote from, Editorial by Andy Magid in the Notices of the AMS, back in 1997: Attach:TheoremsThatNeverShouldHaveBeenProven.pdf) I do, however, regard it as my personal responsibility to
- (diligently try to) understand the arguments before accepting an article, and
- point out any falsehoods.
The length of a paper must be justified by its contents.
Another personal responsibility: to improve the quality of papers, if I happen to see how to.
Benefits to me personally:
- keep a finger on the pulse of what’s happening
- learn things
- abstract service to the community
- get ideas for future work (though this is ethically fraught)
From Greg Kuperberg’s article about the arXiv (PDF):
In the idealized journal system, the diligent referee first checks the main results of a submitted paper. If they are correct, the referee and the editor then consider whether the results meet the journal’s standard. In practice the system is far from ideal. Referees have very little accountability (although some do an admirable job anyway). Many papers are accepted on the basis of name recognition or out of guilt – just because the referees sat on them for too long. Authors need not take no for an answer, because they can scout for journals that will publish them. Journal editors should serve as a second line of defense, but in practice they are only slightly more accountable than referees. (Again, some do an admirable job anyway.) In the end, readers do not know who refereed any given paper or why it was accepted. Because papers can only be published once, the system reduces peer review to simple binary approval.
I disagree with the the last sentence. Why? Because a very high percentage of my referee reports are conditional acceptances
From the same article,
While many readers presume that referees check the results of papers, in practice editors would scare away their referees if they actually demanded this. This inconsistency is only tenable because refereeing is anonymous. My best idea to address the problem is to have the reviewers check one of three options:
1. I have checked the main results.
2. I do not doubt the main results.
3. I doubt the main results.
Some reviews might need to be co-signed to support option 1. Presumably option 3 would be rarely used.
From Gábor Fejes Tóth’s review of Hsiang’s proposed proof of Kepler’s Conjecture:
It is true that in cases when referees and editors fail to exercise their control, it is solely the author’s responsibility to decide what is the appropriate amount of detail. However, he has to bear in mind that a mathematical proof is a social process: It is only the acceptance by the mathematical community which affirms the legitimacy of a proof.
This column in Optics and Photonics News (local copy) deals with a very different field than mathematics, but has some relevant things to say. In particular, it says
Authors tend to assume that if they address or refute all of your points, then you will judge the revision acceptable for publication. Do your best to make your review meet this expectation. Thus, if the fundamental reason you are opposed to publication is that the paper is not novel or significant, be sure to say so explicitly, in addition to commenting on any technical concerns. You should also try to be thorough. In my experience a typical review takes about…
which I generally agree with, and will comment on further in a later draft.