Scientists, funders and journal editors have thrown their weight behind the San Francisco Declaration on Research Assessment
Campaigners against the use of journal impact factors as a proxy for research excellence received a shot in the arm last night with the launch of the San Francisco Declaration on Research Assessment (DORA).
With an impressive line-up of founding signatories, including individual scientists, research funders and journal editors, DORA states in no uncertain terms that journal impact factors (which rank journals by the average number of citations their articles receive over a given period) should not be used "as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contribution, or in hiring, promotion or funding decisions."
In an accompanying editorial in Science, Bruce Alberts, its editor-in-chief, condemns the misuse of journal impact factors as "highly destructive". He argues that they encourage gaming that can bias journals against publishing certain types of papers; waste time by overloading some journals with inappropriate submissions; and encourage "me-too" science in favour of more risky, potentially groundbreaking work.
This is by no means the first time that these arguments have been made. A number of UK-based scientists, including Stephen Curry, Dorothy Bishop and Athene Donald, have been prominent among those calling for an end to the crude application of impact factors. Similar arguments have long been made by scientometricians (who generate the metrics) and science policy researchers.
And yet, the rise of the impact factor has continued, propelled by a seemingly unstoppable managerial logic. Across UK universities, anyone involved in preparations for the forthcoming Research Excellence Framework (REF) cannot fail to be concerned by the fierce pressures that researchers now face to publish in particular disciplinary journals, and the way impact factors are being applied uncritically to determine who will and won't be submitted to the exercise (which in turn carries paramount weight in recruitment and promotion decisions). Claims by ministers and others that this isn't the case just don't stack up against the reality of what those of us in universities are now experiencing.
DORA has come too late to halt the REF juggernaut for 2014. But it should prompt policymakers and the funding councils to pause and take stock before the next assessment cycle gets underway.
More broadly, the practice of journal-based evaluation has become deeply institutionalised, and it remains to be seen whether a declaration like this will be enough to change behaviour. Reflecting on his blog yesterday, Stephen Curry expressed the hope that DORA becomes a "landmark document".
We share this hope, but like any well-meaning statement of intent, it's unclear how institutions, funders or individuals that continue to use metrics in assessment will be corrected or disciplined. Reputation is a crucial reward system in science and fraud is severely castigated: will it now become shameful to boast of the impact factors of one's publications? How will the organisations that supply metrics respond to the declaration, as they are torn between clients' demands for journal-based assessment and the rigorous handling of scientometric data?
DORA argues that articles and researchers should be judged on "their own merits" and emphasises that the "value of all research outputs should be considered", not just publications. One way to achieve this may be through greater use of altmetrics, which offer new insights into the impact of research. But even here we need to be conscious of the dangers of gaming and the difficulties of capturing some channels of impact.
For science to thrive and respond to societal challenges, diversity is key. Current practices based on journal-based metrics are a serious threat to this. One example of this loss of diversity, which alarms us as science policy researchers, is the suppression of interdisciplinarity. But there are many others, such as a lack of consideration being paid to certain research topics. This is why DORA is important and why it deserves widespread support. You can add your name to the declaration here.
By: Ismael Rafols
Ismael Rafols is a fellow at Ingenio (CSIC-UPV), Universitat Politècnica de València and SPRU, University of Sussex. James Wilsdon is professor of science and democracy at the University of Sussex (@jameswilsdon)
Source: http://www.guardian.co.uk/science/political-science/2013/may/17/science-policy