findings. They rejected these papers on the basis of poor methodology while accepting papers
with confirmatory outcomes that used the identical methodology.
Given the above results, one might expect that if editors rely on consensus among
reviewers for their publication decisions, few controversia1 findings will be published. This
problem could be especially serious in social science journals. These journals generally have low
acceptance rates and their editors may decide to publish only manuscripts with high agreement
among reviewers.
A survey of journal editors. To assess how journals treat empirical papers that present
controversial findings, we conducted a survey of 20 current or recent editors of American
Psychological Association (APA) journals. The two-page questionnaire, together with a stamped,
self-addressed return envelope, was mailed out in March 1990. We followed up with phone calls
10 days after the mailing.
Replies were received from 16 of the 20 editors. One question asked: “To the best of your
memory, during the last two years of your tenure as editor of an APA journal, did your journal
publish one or more papers that were considered to be both controversial and empirical? (That is,
papers that presented empirical evidence contradicting the prevailing wisdom.)” Seven editors
could recall none.2 Four said “yes” and indicated that there was one paper. Three editors replied
that there was at least one. Two said that they published several such papers. It seems that
controversial empirical papers do get published, but infrequently. Almost half the editors could
not recall publishing such papers in the past two years.
We then asked about the peer review for the one published controversial empirical paper
that they remembered most clearly. The question was worded: “How did the reviewers respond
to this paper?” A five-point scale from “unanimously accepted” to “unanimously rejected” was
provided, as well as a “don’t recall” option. One of the nine respondents to this question reported
unanimous acceptance, three reported “majority in favor,” four reported “even split,” and one
answered “don’t recall.” In response to a question on this published paper’s contribution to the
discipline, one editor said “not important,” four said “somewhat important,” and four selected the
highest rating, “important.”
The editors were also asked if they had rejected any papers that were controversial and
empirical. Six of the editors stated that they did not receive such papers, and four said they could
not recall any. The six editors who rejected papers with controversial findings did so, they said,
because of poor methodology and poor supporting arguments. Of the rejected papers that the
editors “remembered most clearly,” only one was “unanimously rejected;” a “majority not in
favor” was reported for two, an “even split” for two, and a “majority in favor” for one. Three
papers were rated as “not important,” and three as “somewhat important.”
2 “Unfortunately,” according to one respondent. Also, in follow-up phone calls, several editors
expressed the desire that more such work be submitted.