DORA and Responsible Research Assessment: Moving from Awareness to Action

DORA-01

Evolving DORA

By calling on organizations to stop using journal-based metrics for decisions that impact academic careers, the San Francisco Declaration on Research Assessment (DORA) quickly became a successful tool to raise awareness for responsible evaluation of researchers. The principles in DORA resonated not only with the cell biology community, but also with other researchers looking to assess scholarly work on its own merits. 

Since its release in 2013, more than 20,000 individuals and organizations in 153 countries have voluntarily committed to the principles in the declaration. But awareness does not always translate into action, especially when it requires organizational change. A 2019 study from the Scholarly Communications Lab found 40% of research-intensive institutions in the United States and Canada mention the Journal Impact Factor (JIF) in review, promotion, and tenure materials.1 Recognizing the gap between awareness and implementation of research assessment reform, nine organizations—including the ASCB—banded together in 2017 to transition DORA from a statement of intent to an active initiative supporting sustained culture change for research assessment. 

Research Assessment Reform Trends

Over the last four years, DORA has observed a general trend in assessment reform at universities and research-performing organizations: signing DORA is often part of a larger action plan. For example, after signing the declaration in 2018, the European Molecular Biology Laboratory created a “DORA working group” to examine existing practices and identify areas of improvement for the assessment of graduate students, postdoctoral fellows, and research faculty.2

Of the organizations that have signed DORA, a majority are located in Europe (nearly 170), in particular the United Kingdom (85). Institutions in the United States have been slower to embrace the declaration—six research-performing organizations have signed so far. This comes as no surprise given the different policy environments in Europe and the United States. Many European national research funders have joined Coalition S, which aims to make research publications freely available and asks organizations to reduce the emphasis on the JIF in research assessment. The Wellcome Trust in the United Kingdom has taken this a step further by requiring the organizations it funds to publicly commit to assessing research outputs for academic career advancement based on their intrinsic merit and discouraging the inappropriate use of proxies or metrics, such as the title of the journal where the work was published. 

Even so, organizations in the United States have started making moves to reevaluate assessment. In 2020, the Yale Molecular Biophysics and Biochemistry Department tested anonymized applications for tenure-track assistant professor positions to minimize cognitive biases.3 The process resulted in a diverse pool of candidates selected for interviews: seven women and seven people of color were interviewed out of the 14 selected.4 Indiana University–Purdue University Indianapolis created a path to promotion and tenure recognizing contributions to diversity, equity, and inclusion, showing how research assessment can encapsulate an organization’s values.5

Notable progress is also being made in other parts of the world, highlighting the global nature of this movement. The Latin American Forum for Research Assessment Reform brings together research institutions and regional policy makers to facilitate collaboration and knowledge sharing for research assessment reform.6 In China, a subset of universities and research institutes, hospitals, societies, industries, and other representative institutions has been selected to pilot new research assessment practices that deemphasize the use of the JIF in researcher evaluation.7

Supporting Culture Change

As an initiative, DORA supports individuals and organizations working to improve the ways researchers are evaluated for hiring, promotion, and funding decisions. We have found a major barrier is faculty and staff capacity. The development of policies and practices takes time, and so does implementation. Both tasks add to the number of responsibilities vying for faculty and staff attention. 

That is why DORA partnered with Ruth Schmidt, associate professor at the Institute of Design at the Illinois Institute of Technology, to develop a series of tools to reduce administrative burden.8 SPACE to Evolve Academic Assessment, a rubric for analyzing institutional conditions and progress indicators, was released in 2021 to help academic institutions take stock of what conditions are needed to support fair and responsible research assessment.9 Because organizations are naturally at different stages of readiness for change, what works for one department or university might not be effective at another. SPACE provides flexibility by separating reform into three phases (i.e., foundation, expansion, and scaling) and allowing users to create a roadmap for implementation based on their own institutional needs. 

Additional resources are available on the DORA website. Case studies document key elements of change at academic institutions.10 The resource library offers a collection of tools, position papers, examples of good practice, and more.11

In addition to resource development, DORA uses knowledge sharing to promote collaboration and facilitate action across research funders. The DORA funder discussion group meets quarterly to exchange ideas and provide updates about new policies and practices.12 Any staff member from a public or private funding organization is welcome to join (email info@sfdora.org). The group is currently exploring how to optimize narrative CV formats as a tool for responsible research assessment.13

Looking Ahead

In 2021, DORA was awarded a three-year, $1.2M grant from Arcadia, a charitable fund of Lisbet Rausing and Peter Baldwin. The grant will support Tools to Advance Research Assessment (TARA), a project to facilitate the development of new policies and practices for academic career assessment. A major output is the creation of a dashboard to track good practice in academic career assessment. The dashboard can be used to support research assessment reform in a variety of ways. For example, a professor may use the dashboard to monitor the number of universities assessing contributions to open science and research integrity to advocate for change in their department. Whereas a department chair may find the dashboard useful in identifying examples of how contributions to equity and inclusion can be evaluated for an upcoming faculty search. The first version of the dashboard is expected to be released in 2022. More information about TARA can be found on the DORA website (www.sfdora.org/project-tara).

References

1McKeirnan EC, Schimanski LA, Nieves CM, Matthias L, Niles MT, Alperin JP (2019). Meta-Research: Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. eLife 8, e47338 DOI:10.7554/eLife.47338.

2https://sfdora.org/case-study/the-european-molecular-biology-laboratory.

3https://mbb.yale.edu/news/mbb-radically-changing-how-we-search-new-colleagues.

4Horowich R (April 22, 2021). The future of faculty searches? MB&B Department implements new reforms that aim to increase diversity. Yale News https://bit.ly/3rjwxVK.

5Flaherty C (May 14, 2021). The DEI pathway to promotion. Inside Higher Ed. https://bit.ly/3si87eG.

6https://sfdora.org/case-study/the-latin-american-forum-for-research-assessment.

7https://bit.ly/3s8rjLz.

8https://sfdora.org/resource-library/?_resource_type=tools&_dora_produced=1.

9https://sfdora.org/resource/space-to-evolve-academic-assessment-a-rubric-for-analyzing-institutional-conditions-and-progress-indicators.

10https://sfdora.org/dora-case-studies.

11https://sfdora.org/resource-library.

12https://sfdora.org/funder-discussion-groups.

13https://sfdora.org/resource/using-narrative-cvs-process-optimization-and-bias-mitigation.

About the Author:


Anna Hatch is the Program Director for the San Francisco Declaration on Research Assessment (DORA).