Thoughts on

Refereed Journal Publication


Chuck Doswell

Created: 22 August 2001: updated: (not yet). As usual, this is my own opinion, and does not reflect on anyone but myself. Those with contrary opinions are welcome to communicate with me <>, especially if they're willing to see their viewpoint made public, on this page!

The peer review process in refereed journals

Publication in refereed journals is central to being a scientist in today's world. However, as I see it, publication in a refereed journal does not guarantee anything about the validity of the ideas contained within the published paper. The process of peer review of manuscripts is one by which potential contributions are considered for publication ... most papers as submitted need some changes to satisfy the referees (reviewers). Some percentage (say, on the order of 20-30 percent) wind up being rejected, for various reasons. Two or three reviewers are generally chosen by journal editors on the basis of the editor's perception that they are qualifed to offer an opinion about a manuscript's subject matter. They are "experts" in other words, whose opinion is considered valuable guidance for an editor to make a decision about the suitability of a paper for publication.

Note -- it is not widely known, but editors for AMS journals are under no particular obligation to follow the recommendations of their reviewers. It can happen that an editor would reject a manuscript that all the reviewers gave a "thumbs up" to. It doesn't happen often, presumably, but all the contents of a journal are, in effect, published at the sole discretion of the editor. If editors pick referees they believe are qualified, it seems to me that going against one's chosen referees' opinious should be very exceptional! I believe it is, although I have no direct way of knowing. If an editor picked a referee and then chooses to ignore her recommendations, they why not simply pick another referee? If the editor disagrees with consistent (positive or negative) input from the referees, then perhaps the editor needs to reconsider her opinion of the paper!

As I've said elsewhere, referees (and, by implication, editors) seem to be engaged frequently in a process of censoring material. That is, they are basing their assessment of suitability for publication on whether or not they agree with the paper's content. Since referees are often chosen because they have published on the nominal topic of the paper, they often have a vested interest in either seeing to it that the content of the paper is or is not published. If it agrees with them, they tend to give it high marks. If it disagrees with them, they tend to give it a low grade.

Editors, it seems to me, often fail to recognize the very real possibility of vested interests of the referees in their decision whether or not to publish a paper. A classic case of this occurs when the authors of a submitted manuscript challenge the ideas of an existing publication and the editor sends it to an author of the publication being challenged. It is altogether reasonable for the editor to send it to such an author -- surely that author's opinion of the critical work is somethng an editor would want to know. But some authors seem automatically to reject any paper they review that turns out to be critical of their work. Some authors take offense at the mildest forms of "criticism". Surely an editor must take this into account when considering that input. Alas, in my experience, editors often seem quite reluctant to come to a decision contrary to the recommendations of author(s) who review a manuscript critical of their existing publication(s).

By the same token, some potential referees are very likely to be eager to see a particular paper published when it agrees with their work. In some cases, papers by former students are being reviewed (or edited) by their ex-professors. In cases where the paper was submitted by a former student, an editor should not accept responsibility that paper -- it seems like an obvious conflict of interest to me. Editors also should be ready to account for positive as well as negative bias in weighing the input from their chosen referees. Again, it's not always obvious to me that editors are doing this.

Most editors don't have time to read all the papers submitted, and usually go along with the recommendations of their referees, if a consensus emerges. When, as is often the case, there are conflicting reviews, the process followed by the editor can change markedly, in effect depending on the editor's whims. This is one of the editor's "perks" by which s/he can influence the content of the journals.

If there is a divergence among the reviews, editors can choose to follow the recommendations of one of the reviewers, or they can follow the recommendations of the majority, or they can follow their own counsel. It seems to me that when editors obtain conflicting reviews, they should accept a greater share of the responsibility for guiding an author in how to respond to their satisfaction. In effect, the editor is whom the author has to please, and the content of the reviews is of secondary import. Editors making decisions in the case of divergent reviews are inserting their own views rather strongly but, at present, they don't have to follow the same requirement that they impose on referees, to provide clear and specific information to the authors about how to repair the situation.

Of course, if the editor ultimately rejects the paper, this seems to close the door for further correspondence. Anything an author might submit will be a new submission, and may not go to the same referees or even the same editor. Rejection is, ultimately, final.

A flawed process?

To say that the peer review process is a flawed system might be taken by some to imply that there is some objective measure of "truth" or "falsehood" associated with scientific manuscripts. Thus, I believe the statement that the peer review process is "flawed" is, in fact, an erroneous statement. How can it be known whether or not it is "flawed" if there is no simple, objective way to determine "truth" of journal manuscripts? Editors and their chosen referees are not attempting to determine the truth or falsehood of the ideas within the manuscript under review or, rather, they should not be so doing. Instead, I believe the idea is to gauge how strong a scientific case the author(s) have made to support their interpretations of the information presented. The idea should be to help the author make his/her case more effectively. Many referees consider their roles to be that of censor, not helper, however.

Sometimes, papers are rejected because the analysis of the information was flawed in some obvious way (e.g. an algebra mistake in a mathematical analysis, or using the wrong analysis technique). Sometimes, rejection is based on an assessment that the interpretation of the information uses a fallacious or inapplicable concept (e.g., problems with the assumptions in a mathematical analysis or misapplying an analysis tool somehow). Sometimes rejection is due to the perception that a weak case was made for the interpretation of the information, when a better methodology is known or a more comprehensive analysis is needed to support the interpretations within the manuscript. Sometimes rejection is simply the result of the paper being badly written. And, of course, sometimes rejection is based on a disagreement between the author(s) and the referee(s).

It is this last case that most often troubles me, of course, given the finality of rejection. The only positive alternatives in the case of rejection are to: (a) try to fix the identified problems with the paper [and hope that the changes will placate the reviewers -- either the old set or a new set, depending on the choices of the editor for the resubmission], (b) wait until the editorship changes hands [and hope that a new set of referees will be chosen, some or all of who might be more amenable to the contents of the paper], or (c) submit to another journal [and hope that a new set of referees will be chosen, some or all of who might be more willing accept the paper]. A negative alternative, of course, is to give up altogether on trying to get the idea published.

It is also troubling when reviews provide a consensus for acceptance, but one referee is apparently strongly opposed to publication. From where I sit, I prefer that the default be for publication in such cases, and allow the opposing referee to offer "comments" ... see below ... thereby allowing the resulting exchange to present the contrary arguments to the scientific community. In other words, diverging reviews should result in a two-sided exchange in the refereed literature, not the censorship of one viewpoint. Since I take this up again, I'll defer any further discussion of this option. What I will point out here is that a dissenting reviewer in such a case has to have enough courage and confidence in her scientific convictions to step out from behind the mask of anonymity!

Everyone agrees, in principle, that the review process does not work perfectly. Flawed results and faulty interpretations of results appear in refereed journals all the time, even as good ideas are shot down before they even have a chance to be presented to the scientific community. Any scientist who has had a paper rejected, and I am certainly a member of that group, feels frustrated when they see articles appearing in the formal literature that are (in their opinion) no better and, perhaps, even inferior to theirs that were rejected.

All scientists have opinions about the content of journal articles in their field, whether they express them or not. Those opinions range from absolute agreement to absolute rejection and everywhere in between. Some articles contain a majority of worthwhile material (in the opinion of the reader) and have only a small problem or two. Others contain a majority of perceived nonsense (in the opinion of the reader) and have only the odd nugget of useful content.

Since getting a paper published depends so strongly on a process driven by the opinions of 2-3 people, it's quite likely that "good" material is inappropriately rejected from time to time, and "bad" material is inappropriately accepted. An assessment by 2-3 readers is a pretty small sample. It certainly would not meet the standards we impose on scientific arguments! In my experience, papers that went through a hellish review process have often wound up being the ones most cited. This is not simply a reflection of positive changes forced by the reviewers, although that certainly can be a factor contributing to better papers. For me, results have been mixed, and while I generally feel I have nearly always benefitted from the critical comments of the referees, I have also been acutely aware of what I call the "crap shoot" aspect of the review process. A lot depends on the 3-4 people that got chosen (including the editor!).

Publication as imprimatur?

If an idea is rejected, it's difficult to recover from that finality. For many scientific purposes, rejected ideas don't exist. Their appearance in the non-refereed ("gray") literature is the only way they can even be discussed, but many formal journals frown on citations to the gray literature. In fact, in my experience as a reviewer, it is very difficult to challenge accepted ideas (i.e., those that have been published in a refereed journal) if a contrary viewpoint cannot be found somewhere in the journals. Editors usually regard arguments against accepted ideas and practices with little favor if they cannot be found in a refereed journal. Hence, although we all agree that appearance in refereed journals does not, in principle, mean the contents are guaranteed to be "absolute truth," the debates that go on behind the scenes to determine the acceptability of further publications are based almost totally on what has already appeared in the formal literature. Hence, the truth of the matter is that the approval implied by appearance in refereed journals is essential for an idea even to be considered as a component in a scientific discussion.

This means, that in practice, appearance of an idea in a refereed journal is virtually essential to its acceptance by the scientific commnunity. If it does not appear in a refereed journal, it mostly is doomed to float about the borders of the scientific community, living in a sort of scientific Purgatory. Its non-appearance removes it from the marketplace of ideas, for most purposes. On the other side of the coin, the appearance of some bad idea in a journal means that a flawed interpretation or faulty analysis or fallacious argument now can be cited as an excuse for following that line of reasoning in yet another publication. Bad ideas can proliferate simply as a consequence of having been "approved fo scientific consunption" by being published in a refereed journal. Remember, some fraction of submitted manuscripts will have benefitted from the "crap shoot" in that they will not have been given the sort of critical reviews they might have gotten from a different set of referees. We all know this goes on, even though we like to believe that the process generally works. The reality is that a lot of crap gets published, sometimes in spite of critical reviews, at the discretion of editors.

I have relatively few problems with this reality if we view journals as communication media, and not archives of established truth. The sad fact is that, for various reasons, journals have come to be viewed by many as houses of holy writ, which they most certainly are not!

Furthermore, scientists often find their work being cited by other authors in such a way that their ideas have been misinterpreted. There is no way to anticipate all the possible ways in which one's work can be misinterpreted, so this sort of reaction to being cited is far more common than meets the eye. Why is it common, but not visible? To answer that, I need to move on to the next topic.

What are Comments/Replies for?

A way already exists to counteract the proliferation of perceived bad work and to attempt to clarify minsiterpretations of one's work. Ideas that are seen by someone to be fallacious or otherwise flawed can be subjected to criticism via the submission of critical comments on the published work. Alas, this vehicle is falling into disuse. See the stimulating and challenging discussion by Dr. Ron Errico in:

Errico, R.M., 2000: On the lack of accountability in meteorological research. Bull. Amer. Meteor. Soc., 81, 1333-1337.

in which the decline in comments is documented.

If a journal is viewed as a medium of scientific communication, rather than an archive of sacred truths, then from time to time, scientists should be moved to submit comments on papers that contain one or more elements with which they disagree. If for no other reason, they need to do this in order to have a contrary viewpoint appear in the scientific literature. In the past, and up to the current date, the AMS says that the appearance or non-appearance of Comments and a Reply [optionally offered by the author(s) of the paper in response to the comments that were submitted] appear at the discretion of the editor. Generally, this used to mean that the author(s) of the comments [hereafter, the commentator(s)] would send in their comments, which would be passed on to the author(s) of the original paper. The author(s) could respond to the comments, and that response would be sent to the commentators, to determine whether or not they wished to (a) withdraw their comment, or (b) revise their comment, or (c) let their comment stand. In the case of (b), the revised comments would be sent back to the author(s) for another round of response. Having done this many times in the past, in my experience, this process converged quickly and the whole exchange would then appear in the "Correspondence" section of the refereed journal.

The point of this process should be that criticism of published work could appear in a timely fashion, not years after the appearance of the original publication. The commentator(s) and the author(s) could have their say, and the whole community could read the exchange and decide for themselves which side had the stronger case. Many times, in my opinion, the exchange represents a very pure form of scientific communication ... perhaps the purest form of comnunication within the journals. At one point in their history, journals were the primary form of scientific communication, apart from face-to-face discussions (which are generally undocumented) and the exchange of personal letters (not usually published). In the journals, comments and replies have traditionally represented "cutting edge" discussions between highly-respected participants, and everyone got a chance to see in black and white that scientific ideas are not just "sacred truths" but rather temporary hypotheses along the path we are all walking, trying to make some sense of the natural world. Even relatively acrimonious exchanges were helpful, to all of us. Criticism of scientific ideas is not something that always can be done with all the niceties of true diplomacy. It's not necessarily pleasant to be criticised and there are not always nice ways of saying that someone has done something wrong or inadequate, at least in the opinion of the critic.

It often turned out, as I saw it in comment/ reply exchanges of the past, that both parties in a comment/reply exchange had some part of the discussion "right" (at least in the context of subsequent scientific understanding, which is also not sacred truth), and they also were both partially "wrong" (in the same retrospective sense). Having the exchange was healthy and, in spite of the possiblity of hurt feelings, the whole community benefitted from the process. Ideas could coexist with their counterparts within the same medium

What's wrong with the "comment" process as it now stands?

As Dr. Errico has noted, the ratio of Comments to Articles in the journals is steadily decreasing. It seems implausible to me that the vast outpouring of "content" in the refereed literaure is all essentially sacred truth. Rather, it seems to me, the meteorological community has shied away from the "confrontation" associated with the publication of critical comments and their associated replies. Where scientists used to get up at conferences and openly criticize the work being presented, now most sessions pass with little or no controversy. In the same way, the refereed literature is now filled with more and more pages of material, but with a waning presence of comments and associated reples. Errico has asserted that as much as 50% of the content of articles (in his own areas of expertise) is misleading or even fundamentally wrong. It seems plausible to me that whatever that percentage is, it probably applies more or less to the whole spectrum of papers in any journal. I'm not prepared to debate the specifics of the percentages, but I endorse fully the spirit of the essay contributed by Dr. Errico!

This looks to me like a "cultural shift" that emphasizes diplomacy and courtesy in preference to a fearless and honest search for understanding. Let me publish my opinions and I'll only criticize yours from the safety of referee anonymity. Once your paper appears in the literature, we'll treat it as if the material is now a dead issue, no longer a matter of discussion. This reluctance to take on material we view as flawed is apparently just to avoid making someone uncomfortable.

Now, it seems, "Comments" are being treated in the same way as regular journal submissions. The de facto policy to send them out for review, having the effect of holding the comments hostage to the opinion of the anonymous referees, who may well have their own axes to grind in the debate. I think this is wrong!

I'm not asking for a complete carte blanche for comments. Surely comments must meet some basic standards. There is no need for the refereed literature to provide "equal time" for any idea, no matter how contrary it might be to existing scientific concepts -- e.g., someone who believes that the Earth is flat, or that tornadic windspeeds are supersonic, or whatever. No, this is not what I'm proposing we turn loose on the journals. Rather, assuming that most scientists would prefer not to engage in a confrontational dialog except when they have strong feelings about some point, I believe it would do simply to have an Associate Editor (again, someone chosen by the editor on the basis of perceived competence) give the Comments a quick look and, unless they violate some reasonable standards for any scientific disussion, give the Edtor a "thumbs up" to pass the Comments on to the author(s) for a response.

Treating comment/reply exchanges like submitted manucripts subjects the commentator(s) and the author(s) to the whims of the editor and the "crap shoot" associated with sending the paper out for review. Why not let the comment/reply exchange be treated differently from routine manuscript submissions? The current de facto process of sending comment/reply exchanges out for review, in my opinion, has a pronounced chilling effect on the expression of contrary viewpoints and the opportunity for authors to clarify their papers in the light of criticism.

I see little value in obtaining what amounts to a tiny sample of the range of possible opinions about such exchanges, upon which to base a publish/not publish decision regarding those exchanges. In other words, unless there are compelling reasons not to go forward with it, the default decision regarding comments and their associated reply should be to publish them. Even if the exchange is characterized by some reasonable amount of acrimony, it nevertheless should go forward. Only if it degenerates into ad hominem attacks and personal insults, rather than staying focused on the science, should there be a mandatory effort to tone down the intensity of the discourse. Scientists often believe passionately in their work, and I, for one, have no problem with that being evident in my communications. I make no claims to being right all the time, but I do claim the right to pursue my profession with conviction, and I gladly give my "opposition" the same right. Let us have the debate and keep the editors and referees out of the business of censoring free scientific discourse in the refereed literature.