Note on Bias Against Novelty

ASCB Executive Director Erika Shugart

ASCB Executive Director Erika Shugart

The use of the Journal Impact Factor as a measure of the quality of scientific research may undervalue precisely the type of novel, “high risk/high gain” research that we need to advance our field. This is one conclusion to be drawn from a recent article by a team of three economists who analyzed the novelty of scientific research and found that “home” fields and bibliometric indicators are biased against novel research.

Jian Wang, Reinhilde Veugelers, and Paula Stephan of the National Bureau of Economic Research set out to create a way to measure novelty in science and how it affects the field. They define novelty as “research that draws on new combinations of knowledge components.” The authors measured novelty by looking at all of the references in a paper and calculating the number of journal combinations that had not been seen in prior papers. They then calculated how related these journals were to each other, by looking at co-citation profiles. More new combinations from less-related journals lead to higher novelty score. An example of novel research is a 2004 paper by Denk and Horstmann entitled “Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure.” This article contains references from Nature Materials, which focuses on chemistry, material science, and physics, along with articles from journals from very disparate fields including neuroscience, cell biology, and physiology.

Not surprisingly Wang, Veugelers, and Stephan found that most papers (89%) contained no new combinations of reference journals so were not considered novel. Of the papers that were considered moderately or highly novel by their metric the authors made the following observations:

  • Novel papers are more likely to be either high impact or low impact, reflecting a high risk profile.
  • There is delayed recognition of these high impact papers, which gain most of their citations after three to four years.
  • Novel articles are more likely to be cited in “foreign” fields than “home” fields.
  • Novel papers are also “less likely to be published in high Impact Factor Journals,” which suggests that there may be obstacles to their acceptance in journals that are central in their field.

The combination of publication in lower Impact Factor journals with the delay in citations might look like a poison pill to a scientist eager to get recognition for career advancement or funding. But this paper defines a new metric for identifying research that is novel and could have a significant long-term impact.

It can be very useful for one area of science to turn its lens on another scientific discipline to reveal larger truths or to lend evidence to confirm assumptions. I have an appreciation for social science due to my work in science communication, a field that is informed and improved by a growing body of social science literature. Now social science has shown what ASCB has been saying for years—Impact Factors are not the best way to measure the worth of research.

About the Author:


Erika Shugart is the Chief Executive Officer of ASCB.