Can Peer Review Be Better Focused?
Paul Ginsparg
SUMMARY. If we were to start from scratch today to design a quality-controlled archive and distribution system for scientific and technical information, it could take a very different form from what has evolved in the past decade from pre-existing print infrastructure. Recent technological advances could provide not only more efficient means of accessing and navigating the information, but also more cost-effective means of authentication and quality control. This article discusses relevant experiences of the past decade from open electronic distribution of research materials in
physics and related disciplines, and describes their implications for proposals to improve the implementation of peer review.
[Article copies available for a fee from The Haworth Document Delivery Service: 1-800-HAWORTH. E-mail address: <[email protected]> Website: <http://www.HaworthPress.com>.] KEYWORDS. arXiv, peer review, scholarly publishing, scientific publishing
FREE ACCESS MODELS
There has been much recent discussion of free access to the online scholarly literature. It is argued that this material becomes that much more valuable when freely accessible,1 and moreover that it is in public policy interests to make the results of publicly funded research freely available as a public good.2 It is also suggested that this could ultimately lead to a more cost-efficient scholarly publication system. The response of the publishing community has been that their editorial processes provide an essential service to the research community, that these are labor-intensive and hence costly, and that even if delayed, free access could impair their ability to support these operations. (Or, in the case of commercial publishers, reduce revenues to below the profit level necessary to satisfy their shareholders or investors.) Informal surveys (e.g.,3) of medium-to large-scale publishing operations suggest a wide range in revenues per article published, from the order of $1000/article to more than $10,000/article. The smaller numbers typically come from non-profit operations that provide a roughly equivalent level of service, and hence are more likely representative of actual cost associated to peer reviewed publication. Even some of these latter operations are more costly than might ultimately be necessary, due to the continued need to support legacy print distribution, but the savings from eliminating print and going to an all-electronic in-house work-flow are estimated for a large non-profit publisher to be at most on the order of 30%.4 The majority of the expenses are for the non-automatable editorial oversight and production staff: labor expenses that are not only unaffected by the new technology but that also increase faster than the overall inflation rate in developed countries.
A given journal could conceivably reduce its costs by choosing to consider fewer articles, but this would not reduce costs in the system as a whole, presuming the same articles would be considered elsewhere. If a journal instead considers the same number of articles, but publishes fewer by reducing its acceptance rate, this results not only in an increased cost per published article for that journal, but also in an increased cost for the system as a whole, since the rejected articles resubmitted elsewhere will typically generate editorial costs at other journals. Moreover, in this case there is yet an additional hidden cost to the research community, in the form of redundant time spent by referees, time typically neither compensated nor accounted.
One proposal to continue funding the current peer-review editorial system is to move entirely from the subscription model to an āauthor-subsidyā model, in which authors or their institutions pay for the material, either when submitted or when accepted for publication, and the material is then made freely available to readers. While such a system may prove workable in the long-run, it is difficult to impress upon authors the near-term advantages of moving in that direction. From the institutional standpoint, it would also mean that institutions that produce a disproportionate amount of quality research would pay a greater percentage of the costs. Some could consider this unfair, though in the long-term a fully reformed and less expensive scholarly publication system should nonetheless offer real savings to those institutions, since they already carry the highest costs in the subscription model. Another short-term difficulty with implementing such a system is the global nature of the research enterprise, in which special dispensation might be needed to accommodate researchers in developing countries, operating on lower funding scales. Correcting this problem could entail some form of progressive charging scheme and a proportionate increase in the charges to authors in developed countries, increasing the psychological barrier to moving towards an author-subsidy system. (The other resolution to the problem of unequal resourcesāmoving editorial operations to developing countries to take advantage of reduced labor costsāis probably not feasible, though it is conceivable that some of the production could be handled remotely.) A system in which editorial costs are truly compensated equitably would also involve a charge for manuscripts that are rejected (sometimes these require even more editorial time than those accepted), but implementing that is also logistically problematic.
The question for our scholarly research communications infrastructure is: if we were not burdened with the legacy print system and associated methodology, what system would we design for our scholarly communications infrastructure? Do the technological advances of the past decade suggest a new methodology that provides greater utility to the research enterprise at the same or lower cost?
CURRENT ROLES AND PERCEPTIONS
My own experience as a reader, author, and referee in Physics suggests that current peer review methodology in this field strives to fulfill roles for two different timescales: to provide a guide to expert readers (those well-versed in the discipline) in the short-term, and to provide a certification imprimatur for the long-term. But as Iāll argue further below, the attempt to perform both functions in one step necessarily falls short on both timescales: too slow for the former, and not stringent enough for the latter. The considerations that follow here apply primarily to those many fields of research publication in which the author, reader, and referee communities essentially coincide. A slightly different discussion would apply for journal publication in which the reader community greatly outnumbers the author community, or vice versa.
Before considering modifications to the current peer review system, itās important to clarify its current role in providing publicity, prestige, and readership to authors. Outsiders to the system are sometimes surprised to learn that peer-reviewed journals do not certify correctness of research results. Their somewhat weaker evaluation is that an article is (a) not obviously wrong or incomplete, and (b) is potentially of interest to readers in the field. The peer review process is also not designed to detect fraud, or plagiarism, nor a number of associated problemsāthose are all left to posterity to correct. In many fields, journal publication dates are also used to stake intellectual property rights (indeed their original defining function).5 But since the journals are not truly certifying correctness, alternate forms of public distribution that provide a trustworthy datestamp can equally serve this role.
When faculty members are polled formally or informally regarding peer review, the response is frequently along the lines āYes, of course, we need it precisely as currently constituted because it provides a quality control system for the literature, signalling important contributions, and hence necessary for deciding job and grant allocations.ā But this conclusion relies on two very strong implicit assumptions: (a) that the necessary signal results directly from the peer review process itself, and (b) that the signal in question could only result from this process. The question is not whether we still need to facilitate some form of quality control on the literature; it is instead whether given the emergence of new technology and dissemination methods in the past decade, is the current implementation of peer review still the most effective and efficient means to provide the desired signal?
Appearance in the peer-reviewed journal literature certainly does not provide sufficient signal: otherwise there would be no need to supplement the publication record with detailed letters of recommendation and other measures of importance and influence. On the other hand, the detailed letters and citation analyses would be sufficient for the above purposes, even if applied to a literature that had not undergone that systematic first editorial pass through a peer review system. This exposes one of the hidden assumptions in the above: namely that peer-reviewed publication is a prerequisite to entry into a system that supports archival availability and other functions such as citation analysis. That situation is no longer necessarily the case. (Another historical argument for journal publication is that funding agencies require publication as a tangible result of research progress, but once again there are now alternate distribution mechanisms to make the results available, with other potential supplemental means of measuring impact.)
There is much concern about tampering with a system that has evolved over much of the past century, during which time it has served a variety of essential purposes. But the cat is already out of the bag: alternate electronic archive and distribution systems are already in operation, and others are under development. Moreover, library acquisition budgets are unable to keep pace even with the price increases from the non-profit sector. It is therefore both critical and timely to consider whether modifications of existing methodology can lead to a more functional or less costly system for research communication.
It is also useful to bear in mind that much of the current entrenched methodology is largely a post World War II construct, including both the large-scale entry of commercial publishers and the widespread use of peer review for mass production quality control. It is estimated that there are well over $8 billion/year in revenues in STM (Scientific, Technical, and Medical) primary publishing, for somewhere on the order of 1.5-2 million articles published/year. If non-profit operations had the capacity to handle the entirety, and if they could continue to operate in the $500-$1500 revenue per published article range, then with no other change in methodology there might be an immediate 75% savings in the system, releasing well over $5 billion globally. (While it is not likely that institutions supporting the current scholarly communications system would suddenly opt to reduce their overhead rates, at least their rate of increase might be slowed for a while, as the surplus is absorbed to support other necessary functions.) The commercial publishers stepped in to fulfill an essential role during the post World War II period, precisely because the non-profits did not have the requisite capacity to handle the dramatic increase in STM publishing with then-available technology. An altered methodology based on the electronic communications networks that evolved through the 1990s could prove better scalable to larger capacity. In this case, the technology of the 21st century would allow the traditional players from a century ago, namely the professional societies and institutional libraries, to return to their dominant role in support of the research enterprise.
arXiv ROLE AND LESSONS
The arXiv6 is an automated distribution system for research articles, without the editorial ope...