The organizers of DORA are soliciting examples of good practices in research assessment including approaches to grant review, hiring and promotion, awarding prizes, and mentoring that emphasize research itself and not where it is published. If you know of any positive examples that showcase the use of diversified research assessment methods and may provide inspiration for research institutes, funders, journals, professional societies, or researchers to abandon their reliance on Journal Impact factors (JIFs) and adopt more enlightened approaches, please contact the DORA organizers.
The University of Texas Southwestern Medical Center
As Chair of Cell Biology at the University of Texas Southwestern Medical Center, Sandra Schmid’s efforts to recruit assistant professors for her program explicitly stated that Journal Impact Factors would not be considered in assessing candidates. Schmid reports that she received ~320 responses. She and two or three faculty members interviewed 20 candidates by Skype and chose to fly in five candidates for interviews. One position has successfully been filled.
This is the advertisement she used for recruitment:
Multiple Tenure-track Assistant Professor Positions
Department of Cell Biology
UT Southwestern Medical Center We have recently moved into brand-new, state-of-the-art facilities and are expanding to take advantage of the rich environment and outstanding biomedically-relevant research opportunities at UT Southwestern. Our vision is to build a highly collaborative and interdisciplinary department that provides outstanding scientists with sufficient resources and a synergistic and supportive environment to enable them to develop as leaders in their fields. As cell biologists, we embrace the complexity of cellular function and behavior. We are interested in the genetic and non-genetic origins of heterogeneous cell behaviors (normal and disease) that emerge as a consequence of the coordinated spatial and temporal regulation (or dysregulation) of cellular pathways. We seek colleagues who share these interests, and who approach cell biological questions from any of multiple perspectives (e.g. imaging, cryo-EM structure, biochemistry, computational modeling, proteomics, genomics, genetics, etc.). The ideal candidate will be evaluated only on the significance of the discoveries the candidate has made–not on the impact factors of the journals where his or her results were published–and on whether he or she is the best fit to complement and augment the intellectual creativity, skills, and innovation potential of our department. To help our assessment, applicants should provide a succinct (2 page) cover letter describing in separate sections:
- Your most significant scientific accomplishment as a graduate student
- What is (or will be) your most significant scientific accomplishment as a postdoc
- The overall goals/vision for your research program at UT Southwestern
- Your experience and qualifications that make you particularly well-suited to achieve your goals
The best candidates will also have earned the respect of their mentors and colleagues. Thus, the quality/content of your cover letter and recommendations will be our principal criteria for your further consideration. Candidates of interest will first be interviewed by videoconferencing as their applications are received. Finalists will be invited to present their current and future research to our department and to meet our faculty. Candidates must have a Ph.D. or M.D. degree. UT Southwestern is an Equal Opportunity/Affirmative Action Employer. Please submit your cover letter, 3 letters of reference and your CV to the attention of the Department Chair, Sandra Schmid at firstname.lastname@example.org. Deadline for Applications was November 1, 2014.
Max Planck Society, Germany
The Max Planck Society recognizes the need to evaluate the actual content of applicants’ research in its recruiting efforts because excellent research does not necessarily appear in journals with high impact factors.
Stefan Fabry of the Max Planck Society describes how the organization assesses candidates for the position of Max Planck Research Group Leader (MPRGL): “We usually want both a complete standard summary of all scientific papers which a candidate has published so far (that is, the publication list) and full copies of the three papers which the candidate considers as her/his own best ones. We do not count ‘top journals’ in the first place, although in practical terms I cannot deny that this does play a certain role when estimating a candidate’s performance. However, we are well aware that an ‘exotic,’ or in other words, an interdisciplinary, approach is often most promising in the long run but easily happens to fail showing up in classical high-impact journals. So we strongly encourage our committees to focus on content rather than on formal aspects.”
Gulbenkian Institute Faculty Application
The Gulbenkian Institute faculty application process explicitly asks that impact factors be excluded from the application materials:
Applicants should provide a curriculum vitae with a well-argued summary of research planned for the next quinquennium (2-4 pages) and up to three references. The CV with publication list should not present numerical indicators such as impact factors, numbers of citations or Hirsch (H) factors, and such indices will not be taken into consideration throughout the selection procedure. Applicants are, however, invited to propose up to three of their papers that they consider of special merit, with a brief explanation of why each was chosen.
Some highly successful scientists explain why Journal Impact Factors (JIFs) have played little role in their decisions about where to publish their work:
1. John Pringle, Stanford University
I decided about 40 years ago that I wanted to publish only in journals at which practicing scientists did the editing and editorial decision-making and at which the overall goal was to serve other scientists and science itself and not a corporate bottom line. I have stuck faithfully to this philosophy except in some cases of collaborative papers for which I did not have the deciding vote. I have never once regretted this decision, having always felt that if my lab and I did work that had genuine impact (i.e., moved our field forward in truly significant ways), it would not need to be justified/marketed using some nonsensical and counterproductive gimmick like the “journal impact factor.”
2. Peter Lawrence, MRC, Cambridge University
When Gary Struhl and I decided to work on planar polarity 20 years ago, we decided for a variety of reasons to avoid publishing in some of the journals that happen to have the highest impact factors. We have found that publication in such journals is not the essential prerequisite to survival that many think. Our postdocs have got good jobs and an EMBO fellowship without any such papers and, so far, we have won grants. These postdocs did well because their work is of high quality and they discovered things that matter; there are still enough scientists out there who recognise that these things are more important than the journal. The problem is that more and more of what should be scientific decisions are being annexed by, or given to, bureaucrats and they prefer adding up to evaluation.
3. Jasper Rine, University of California, Berkeley
The high glamour journals, which sport the highest journal impact factors, are driven by incentives that run counter to the needs and interests of professional scientists. In their need for an uber headline, the editors of these magazines are incentivized to hold out for ever-more data supporting ever greater claims, regardless of the cost of those decisions to the careers of the students and post docs endeavoring to get their work published. Although some postdocs think that publications in such venues are critical to their success, I do not see any data to support that view. Granted, good postdocs with papers in those journals tend to do well. But so do good post-docs whose papers are all in the respected peer-edited, peer-reviewed journals of their field. Here are some relevant data. To date, I have trained 29 post-doctoral fellows, of whom 20 have pursued academic careers. Eighteen of these had no publications in the big three journals during their post-doc, yet were hired as Assistant Professors at the following Institutions: University of Washington, University of Zurich, Rutgers University, Fred Hutchinson Cancer Research Center, UCSD, Ohio State University, University of Wisconsin, NIH, Max Planck Institute in Berlin, University of Stockholm, UC Davis, VanAndel Institute, Purdue, Duke, University of British Columbia, and University of West Virginia. Of these, 13 are already tenured, and to the best of my knowledge, only one of the thirteen has published in one of the big three journals since being hired. In my view, the process of publishing in those journals with the highest journal impact factors can harm a post-doc’s career by forcing one to relinquish control of your valuable time to satisfy the idiosyncratic obsessions of the editors.
4. Mike Eisen, University of California, Berkeley
When I started my lab at Berkeley in 2001, I made a commitment to publish every single one of our papers in open access journals. This meant that I have, for over a decade, actively shunned those journals with the highest impact factors. Although I was told by many, many senior colleagues that shunning the most selective journals would torpedo my career, I secured funding, I got tenure, and was named an HHMI Investigator. Meanwhile my trainees have thrived. Without papers in those journals with the highest journal impact factors, postdocs in my lab secured assistant professor positions at Harvard, Wash U., University of Wisconsin-Madison, Berkeley and other top institutions, and my graduate students have done equally well. The key for me and my lab was not just to occasionally avoid the high-impact-factor journals, but to always avoid them – so that people looking at my CV would realize the absence of papers in those journals was not a failure, but rather the result of a principled stand against their destructive influence on modern science.
5. Peter Agre, Nobel Laureate, Johns Hopkins School of Medicine
Like most scientists, we try very hard to submit our work to the most appropriate journal. Journal prestige may play a role, however publication without unnecessary disappointments, delays, or overhype is essential. In my entire career we have rarely submitted our work to high impact factor journals.
It might be worth mentioning that of our 12 most cited scientific reports (review articles excluded) our publications appeared in SCIENCE (1), NATURE (2), PNAS (4 – all direct submissions), J Biol Chem (2), J Cell Biol (1), J Neurosci (1), and Biochemistry (1).
EMBO Long-term Fellowships
The application process for EMBO Long-Term Fellowships emphasizes the most important outcomes and impact of the applicant’s work rather than where it is published and specifically states that journal impact factors should not be provided:
In the ‘achievements of PhD’ section please provide your own summary of what you consider the most important outcome(s) of your PhD work and the impact of your work on and beyond the respective scientific field.
In your publication list, you should indicate your three most important publications, i.e. the three primary research papers that in your view provided the most important and original contributions to scientific knowledge irrespective of journal name and impact factor. Do NOT add the journal impact factor. Citations to the article or other article level metrics with source may be listed, but are not essential.
Are journal impact factors or journal name taken into account when evaluating applications?
Journal impact factors or name are not used in the evaluation and selection of applications. EMBO encourages evaluation of the quality of the scientific work and its impact on the field, rather than the Impact Factor of the journal in which it was published.
U.S. National Science Foundation
The U.S. National Science Foundation has modified its instructions to grant applicants to recognize that the outputs of scientific research include more than just publications, an idea endorsed by DORA:Instructions for preparation of the Biographical Sketch have been revised to rename the “Publications” section to “Products” and amend terminology and instructions accordingly. This change makes clear that products may include, but are not limited to, publications, data sets, software, patents, and copyrights.
U.S. National Institutes of Health
The U.S. National Institutes of Health has revised the format of the CV or “biosketch” in grant applications. The addition of a short section into the biosketch where applicants concisely describe their most significant scientific accomplishments may help discourage the grant reviewers from focusing on the journal in which previous research was published.
American Society For Cell Biology Policy On Use Of Journal Impact Factors In Advertisements
Reflecting its deep concern about the widespread misuse of journal impact factors to evaluate the outputs of scientific research, ASCB will not permit advertisement of journal impact factors in any of its publications or in promotional materials that it distributes on behalf of exhibitors at the ASCB Annual Meeting.
ASCB initiated the San Francisco Declaration on Research Assessment (DORA; www.ascb.org/SFdeclaration.html). DORA calls upon funding agencies, institutions, publishers, and researchers to cease using the journal impact factor as a surrogate metric for the quality of science and to develop better methods to evaluate research.
When used to evaluate an individual journal article or the work of scientist, the journal impact is a highly flawed metric. And yet it is often used for that purpose, and in emphasizing a journal’s impact factor when communicating with potential authors, publishers may inadvertently perpetuate this unfortunate practice. In particular, DORA encourages publishers to “greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics (e.g., 5-year impact factor, EigenFactor , SCImago , h-index, editorial and publication times, etc.) that provide a richer view of journal performance.”
We hope that our fellow publishers will join us in our efforts to encourage the use of appropriate methods for the assessment of scientific research.
In awarding the Annual ASCB Kaluza Prizes Supported by Beckman Coulter to honor academic excellence in graduate student research, the American Society for Cell Biology adheres to the following policy:
In keeping with guidelines from the Declaration on Research Assessment (DORA), candidates will be evaluated on the significance of discoveries they have made, not on the impact factor of the journals where results have been published. Candidates are strongly encouraged to articulate clearly and succinctly, in one page or less, what scholarly achievements they are most proud of and why those results could have not been achieved without their specific contribution.