Innovations in Responsible Research Assessment
Innovations in Responsible Research Assessment
Research culture and career incentives in many disciplines are mostly aligned toward rewarding a narrow range of research(er) activities: the quantity of articles published in “top-tier” journals, and sometimes the citation they get. We hear researchers easily grasp that such measures do not reflect scientific contributions well, and can feel trapped by a system that is hyper-focused on these metrics.
A variety of innovations, however, have been developed to help researchers and institutions change research culture and incentives to better reflect scientific values. Many of these are currently being shared by DORA under Reformscape, a comprehensive repository designed to inspire users and help them explore opportunities for local implementation. It collates career-related policies and practices within research performing organizations, including standards for hiring, review, promotion, and tenure, drawn from academic institutions in more than 20 countries.
Below, we share two of innovations discussed in Reformscape, including using annotated or narrative CVs, and removing journal-level metrics in researcher evaluation contexts.
Annotated or narrative CVs
In the Procedures for Promotion to Associate and Full Professor of the Department of Psychology, the University of Maryland, they stress the importance of recognizing diverse types of activities, in particular leadership, and encourage the use of annotated CVs to better contextualize achievements. The annotation process is intended to provide additional information that will enable evaluators to better assess the substance of the research and overall contribution of the candidate to individual research products.
“The annotated CV enables candidates to more fully document authorship roles; the scope of individual research products; efforts to ensure reproducibility and transparency (open science); the contribution of the research product to science or society; contribution to diversity, equity, and inclusion; and other considerations that provide context for understanding the specific work product as it relates to review criteria.”
Removing journal-level metrics
The Protocols for Faculty Hiring and for Search Committees for the Geisel School of Medicine at Dartmouth College specifies that CVs submitted by applicants for faculty positions should not include journal impact factors (JIFs). This initiative responds to a number of well-documented deficiencies of the JIF as a tool for researcher assessment:
- Highly skewed citation distributions: Citation distributions within journals are highly skewed, meaning a few highly cited articles can disproportionately inflate a journal’s impact factor.
- Manipulation by editorial policy: Journal Impact Factors can be manipulated, or “gamed,” by editorial policies.
- Lack of transparency: The data used to calculate Journal Impact Factors are neither transparent nor openly available to the public.
- Not a measure of individual article quality: It is a measure of the “average citation performance” of papers in a particular journal over the past two years, not a direct measure of the value or quality of an individual paper or researcher. Claims that JIF signifies the value or quality of individual papers are not supported by evidence.
Call to action for the ASCB Community
We encourage ASCB members to be active participants in these conversations. You can:
- Explore Reformscape and reflect on what small step you can take. As researchers, supervisors, mentors, editors, and colleagues, you can reflect on how you are valuing research and researchers in diverse assessment moments like annual reviews, hiring and promoting procedures, reviewing grants and papers, among others.
- Engage and collaborate with others in your organizations. Having conversations about what small and big changes you have seen help improve research quality and rigour, and if and how your assessment approaches are well aligned with your research values and objectives.
Connect with DORA
The San Francisco Declaration on Research Assessment (DORA) is an initiative that operates under ASCB. You can sign DORA at sfdora.org/sign and follow us on LinkedIn and Bluesky for more updates. If you have improved assessment practices at your institution, please share your story with us at info@sfdora.org.