home page

CSR's Nakamura: Streamlining NIH Peer Review in Stormy Times

Richard Nakamura, Photo from NIH, Center for Scientific Review Richard Nakamura

It’s been almost non-stop stormy weather at NIH’s Center for Scientific Review (CSR) for nearly a decade since the doubling of the NIH budget at the millennium ran into the headwinds of war, the Great Recession, and the fiscal stalemate. Richard Nakamura had been serving as the CSR’s temporary head since September 2011 but with his confirmation last December as permanent director, the CSR has cautiously released the latest suggestions from its Peer Review Advisory Committee on protecting an institution that for academic researchers renders life-or-death decisions, the NIH study section. See the NIH Peer Review Notes.

The latest from the Advisory Council includes the idea of abolishing all resubmissions of rejected grant applications—what are called A1 amended proposals—for an A0,  all-or-nothing, single-tier application system. This would effectively leave it to PIs themselves to decide whether a given proposal is really new or if submitting it is a futile gesture.

But before a cold front of objections sweeps in, Nakamura is at pains to insist that what is new here is not the elimination of A1 amendments, which is only one suggestion from the Advisory Council, but CSR’s determination to listen to the research community. "I would say that it’s less about the specifics of the proposed system as our approach to thinking about the future of the peer review system," Nakamura explained in a recent interview. "The reason we are talking about some of these ideas now is not because they are about to be implemented. We’re looking for advice and input from the scientific community about the directions they would like to see us going in."

Certainly CSR is well aware that the history of peer review reform has been tempestuous. Nakamura traced it to the recent surge of grant applications that began in 2002 at 55,000, plateaued around 76,000 in 2005 and 2006, sprang up to 112,000 in response to the American Recovery and Reinvestment Act (ARRA), and still remain in the 80,000 range, long after ARRA funds are gone. Success rates on R01 proposals have dropped to historic lows in recent budget years. The original NIH doubling did increase the number of PIs seeking grants, said Nakamura, and the ARRA spurred many PIs to write more grant applications, which together led to a slight increase in the number of grants per PI. But in Nakamura’s analysis, arithmetic is still the dilemma. "There are more PIs seeking what is essentially a flat amount of money. And that is essentially the core of our problem right now. With success rates at historic lows, our scientific constituency is a lot less happy than it was 10 years ago for obvious reasons."

The unhappiness coincided with CSR’s attempts to address longstanding criticism of the peer review process—that it took too long, that it took far too much time from researchers both in rewriting amended applications and serving on study sections. Others complained that NIH peer review favored an "old boys" network of insiders and that in particular, younger investigators were being frozen out before they even get started. In 2007, CSR began tightening up its internal policies on recruiting, training, and rotating peer reviewers.

Weather Balloon_Courtesy U.S. Army, Army Test and Evaluation Command, Dugway Proving Ground

And in 2008, NIH decided to end A2 resubmissions. They were a waste of valuable time for applicants and reviewers, according to Nakamura. "The A0 and A2 had essentially switched places and committees were making PIs wait in line for their awards. The order in line was not significantly shifting as you went from A0 to A2. All that was happening by having the A2 was lengthening the time to award." A commitment to not let the ending of the A2 disadvantage any subgroup such as new or early-stage investigators sealed its fate, says Nakamura. "The change was relatively fair. The main consequence we have seen as a result of the change is that awards are made sooner. Yes, we recognize as real the pain of PIs who get great scores but no award."

These changes in peer review came just as the bottom seemed to drop out of the world economy and NIH success rates fell to historic lows. Big changes amid big uncertainty made the research community doubly unhappy, says Nakamura. "There has been a tendency for applicants to blame the peer review changes when they fail to get funding though the fall in success rates and inflation cut their chances for funding."

Nakamura admits that CSR reform was not an unalloyed success. "We don’t claim that changing peer review has fulfilled all the goals we had for it. We recognize that some of our expectations have not been met. Among other expectations was that changing the length of our applications would have two great effects: reduce the overall workload which would allow our reviewers to focus on the significance of applications rather than the details of the experiments." The workload did not decrease, said Nakamura, nor did "significance" overtake "approach" in its impact on final scoring and funding, according to CSR studies. Still Nakamura believes that the greater emphasis on significance, innovation, and overall impact improved peer review by forcing applicants and reviewers to focus on these criteria.

With the new suggestions about eliminating amended resubmissions entirely, Nakamura says that the Advisory Council and CSR are wary of unforeseen effects. If NIH considered all applications as new, PIs could respond with endless variations of the same proposal. "This issue of how many applications a PI can send in especially if it significantly increased the number of application would be a major problem and a major flaw of this strategy," he conceded. "It would only work if we kept at the present level or fewer applications each year. We wouldn’t want to harm the success rate and we wouldn’t want further workload at very low success rate." Still Nakamura said, "We think it’s better if PIs decide when futility has been reached in the application process." 

Again, it’s only one proposal, says Nakamura, but it’s an example of the kind of thinking CSR is doing about the future. "The goal here is to look at new ideas to see if we can come up with ideas that can solve current problems. Test them. We’re very aware that what we proposed could cause harm."

Created on Friday, February 15, 2013

facebook twitter1 youtube linkedin