These results suggest that one can get reviewer agreement on controversial empirical
papers. Moreover, most of these papers are published without high levels of reviewer agreement.
Apparently, editors do not rely solely on reviewer agreement.
It is interesting that our survey found only two instances of unanimous reviewer
agreement for empirical papers with controversial findings. In one case, the recommendation was
to reject. In the other, it was to accept. In the case of the accepted manuscript, it should be noted
that the editor had invited this submission and had selected reviewers who, he said, were
sympathetic to its content.
Our survey indicates that some controversial empirical papers do get published, even
when there is disagreement among the reviewers. The willingness of editors to publish such
papers is encouraging. On the other hand, 7 of 16 editors cou1d recall no instances of publishing
controversial empirical findings. Consequently, we consider some strategies to increase the odds
of publishing this type of paper in the next section.
Possible solutions. Some methods that are currently used by journals should help.
1. Some journals’ editorial policies allow the author to submit a list of possible
referees, one of whom would be selected.
2. Items can be included on structured rating sheets so that reviewers rate the extent
to which the findings are controversial. Editors can then give such ratings more
weight.
3. Additional reviews can be sought when papers are judged to contain controversial
findings. (This strategy was used for only one of the nine published papers and for
only one of the six rejected papers in our survey.)
4. Special appeal procedures may help for controversial papers. This might involve
other members of the editorial board.
5. Controversial papers can be reviewed initially without revealing the findings. This
procedure is currently used by the International Journal of Forecasting. It has not
been used frequently but, when used, it has been beneficial.
6. Provide a section of the journal for “Controversial Findings.” The selection of an
editor for such a section would indicate the journal’s willingness to provide space
for such studies. Unfortunately, the one application of this approach that we know
(Armstrong 1982b) has produced only one submission, and the findings reported
in that submission were not controversial, only the methods were.
Rather than looking for agreement, it might be useful to seek reviewers to act as advocates. This
advocacy system would be used for papers that are designated as containing controversial results.
A paper could be so designated by the author, the editor, or a reviewer, after which special
advocacy procedures would be used. This might include some of the above mentioned
suggestions. In addition, one could use more reviewers in an effort to find an advocate. An