Editor’s Note: What follows is the complete transcript of an interview conducted by ASCB President Jodi Nunnari with Jon Lorsch, director of the National Institute of General Medical Services (NIGMS) at the National Institutes of Health (NIH) on February 22, 2018. The text was edited for clarity.
Nunnari: Thank you for doing this. We have a couple of warm-up questions for you. We want to know who you are. A person with the kind of power that you have, people want to know. I guess the first question is what inspires you to come in every day?
Lorsch: The main thing is to get the most out of the taxpayer’s money in terms of high-quality science. That’s what we do here. And that’s what drives me. That’s pretty much it.
Nunnari That’s a fair answer. What do you worry about it? Professionally but I’m also curious on a personal level.
Lorsch: Professionally, what I worry about is lost talent. I worry about all the tremendous researchers who are struggling to stay funded, are struggling to get funded, who could be the person that makes the next great discovery or 10 years from now, contributes in some important way to a medical breakthrough down the road. We may lose those advances if we don’t find a way to get them funded.
Nunnari: You’re worried about it? You don’t think the system is sufficient to prevent it?
Lorsch: Not now, no. Certainly not. I worry about the next generation even farther upstream of that, the kids who, because of where they’re growing up, don’t have access to scientific research as a career. These are what the New York Times called Lost Einsteins. We may be losing tremendous talent because of the way the system works, the way the funds are distributed. And that could really be limiting scientific and medical progress.
Nunnari: Do you think that’s a bigger problem now or has it always been a problem?
Lorsch: I think it’s always been a problem. I think it could be a growing problem, with disparities across our country, economic disparities, and educational disparities both in say inner city regions and in rural regions, in the middle of the country versus the coasts. I think those are all serious issues that we need to be focusing on and wrestle with.
Nunnari: And you consider that part of your job here?
Lorsch: Oh, absolutely. Yeah.
Nunnari: That’s a good transition into the next set of questions. What are the problems you’re really trying to address here at NIGMS and the biggest issues that you feel that need to be tackled? The corollary to that is what are the tangible results?
Lorsch: The two things I just mentioned: getting the most and best science for the tax payers’ money and ensuring that we are funding all the great ideas and all the great scientists that we can. These are closely related to each other because the way we’re going to get the best science done, the way we’re going to maximize chances for breakthroughs is to ensure that we have a broad and diverse portfolio of researchers.
Nunnari: We’ll get to that later.
Lorsch: If you look at the history of discovery, discoveries come from unexpected places. If I could predict them in advance, my job would be really easy. But we can’t, we know we can’t. The history of science tells you you can’t. It’s like investing in the stock market or in a business in order to maximize the upside potential which, in the case of science, is breakthroughs or the science that leads eventually to breakthroughs. You want to have a broad and distributed portfolio because you don’t know in advance where they’re going to come from.
Nunnari: What’s the tangible result of having a broad and distributed portfolio?
Lorsch: In order to get the best scientific returns on investment for the taxpayers.
Nunnari I’m going to jump a little forward. What evidence is there that a diverse portfolio really does produce the best science? The associated criticism to diverse portfolio that I’ve heard many times in the community is that if you take that strategy of distributing the wealth to more investigators, to more grants, maybe what will result is a shift towards mediocrity.
Lorsch: History shows that’s not true, that the big discoveries come from all sorts of unexpected places. You couldn’t have predicted them in advance. You can come up with plenty of examples of great scientists struggling to be funded.
Nunnari: Just being the devil’s advocate, how many discoveries come from expected sources versus unexpected? You can swing the pendulum a little too far.
Lorsch: That’s always possible that you could go too far one direction. But another way to look at this is through the study that came out of the UCSF [University of California – San Francisco] and the Gladstone Institutes where they used big data analytics to trace the history of medical breakthroughs. They looked at a couple of drugs and asked what the science was and who the scientists were that led up to that breakthrough. They found that literally thousands of scientists over decades, in one case almost a hundred years, were required to build the scientific base that led to those discoveries. There were people along the way who came in and made bigger contributions, but those contributions were entirely based on a foundation of all these other people’s work.
That’s the other aspect of this. People tend to think about discoveries as a point, that Eureka moment, but that’s actually not how they work. Discoveries are predicated on strong foundations of all kinds of other work. This study showed it’s extremely difficult to dissect out each of the strands and say which was the most important. They’re all contributing in a synergistic way. I think that gives a lie to this idea that you can just pick out the few best people and have all the discoveries. Historically, you can pick out individual cases of discovery that indicate that it’s not true, but analysis really shows that it’s true.
Nunnari: I came up with a sports analogy that would be that people need to be trained how to throw more strikes instead of just making the home plate wider.
Lorsch: If we were in a situation where we were funding all the science that peer review thought was worthwhile, you might be able to make an argument that maybe you’re going too far. Even at GM [NIGMS], where we’ve increased the success rate pretty significantly in the last five years, we still have studies that peer review said were highly meritorious, that we don’t have enough money to reach for. And when you’re in that scenario, I think you still have to push as much as you can to get more of those studies in because that next one may either be the breakthrough or maybe a strand of knowledge that 10, 20, 30 years from now ends up being critical for the eventual breakthrough.
Nunnari: What would Utopia look like here at the NIH?
Lorsch: My Utopia?
Nunnari: You get to have a Utopia.
Lorsch: It’d be nice to have a Utopia. I think the key for us, one of the driving principles, is this broad and diverse portfolio. If we had a situation where we really could fund all the science and all the scientists who were meritorious, both in peer reviews’ view and in our review and fund them at a level that was reasonable for them to really achieve significant results. That would be sort of scientific utopia.
Nunnari: I know you don’t like to talk about percentiles, but that’s the way we all think. What do you think that cut off is?
Lorsch: NIGMS doesn’t have a pay line so we don’t [calculate] by percentiles.
Nunnari: I know you don’t have an official payline but everybody follows.
Lorsch: We think in terms of success rates. What is the number of grants funded divided by the number of submitted applications. Right now we’re around 30 percent. I think that’s good. Really good.
Nunnari: That’s good.
Lorsch: There are still things left on the table with a 30 percent success rate, but we would be starting to get into a ballpark where we feel a little more comfortable. When you get down to the 20s or below, you’re really leaving a lot of tremendous science on the table. The other thing I should point out is that’s a grant success rate. In the current system, people can submit lots of grants. That underestimates the true nature of the problem because there are people who have multiple grants. What we started to do is to focus on how many investigators we are funding. That’s another metric that really drives us. We want to make sure that we’re funding enough meritorious investigators –not just enough good grants– because the investigators are the ones with the ideas, they’re the ones who are going to make the big contributions or breakthroughs.
Nunnari: There is some data that suggests the way we peer review is not necessarily a perfect indicator of its impact.
Lorsch: Both in terms of how NIH or other funding agencies do it and in terms of how it’s peer reviewed for publication. I think both of those things are true to some extent for peer review in general, as a human endeavor. In the case of reviewing grant applications, we’re asking people to make predictions about the future. You just know a priori that can’t possibly be a perfect process, right? No one is a soothsayer who can say with complete accuracy, what’s going to happen in the future and yet that is what we’re asking them to do. And, of course, it’s not going to be perfect. In fact, the data from people like Mike Lauer [NIH Deputy Director for Extramural Research] and a number of other studies bear out the fact that it’s not perfect. That’s why when we think about funding decisions, one of the reasons we don’t use a percentage cutoff is because of that fact. We know that if you take an arbitrary percentile, there are a lot of good studies above that arbitrary percentile, most likely. We look at the whole envelope of good scores and treat them more or less all as meritorious grants. And then we started looking at other factors in addition to that kind of holistic review.
Nunnari: That’s a little controversial because it’s kind of placing the power in the hands of the program officers relative to the study sections.
Lorsch: Well, it’s giving them both input into the final decision. Certainly, we take the reviews extremely seriously, but not just the scores. I think that’s really, really important. In fact, the reviews themselves– the summary statements and the individual critiques– are, to some extent, more important for us than the score itself. Because if you think of a priori thinking about it, viewing the results of a study section as a true rank order that tells you that number three is really statistically significantly more likely to predict a good outcome then number seven on the list can’t be true because of the way the system works. There’s twenty some people in a room, three of them read the application and the others are just listening to what they’re saying. The difference between a 10th percentile and a fifteenth could be one or two people who didn’t read the application voting one way or the other.
Nunnari: What about a one percentile?
Lorsch: There’s some evidence that below five percentile is somewhat more predictive than the rest. But again, when you look at the aggregate of the data about peer review– at least NIH peer review– it’s not highly predictive if you’re a fifth percentile that it’s going to be a more significant study then if you’re at a 15 or 20 percentile even.
Nunnari: When you first started it was a different story here?
Lorsch: Well, NIGMS has not, in recent history, used a cut off for making the grant decisions.
Nunnari: Maybe you’re being modest, but I think, in my opinion, one of the great things that you’ve done is to really move from away from fifteen more towards thirty.
Lorsch: In terms of success? Right. Absolutely. That was a driving goal of our strategic plan that we put out in 2015.
Nunnari: You first came in 2013, did you? What were the first kinds of “low hanging fruit” changes you made?
Lorsch: One of our central tenets and the driving goals of our strategic plan is that we wanted to increase the Institute’s focus and support for investigator-initiated basic or fundamental research. Where are we going to find the funds to do that. One of the first places we looked at was the big programmatic or top-down initiatives. These things had been going on for 15 or more years. They were fairly substantial center-based programs targeted at a specific area of science.
Nunnari: Structural biology, structure initiative, chemical biology.
Lorsch: Systems biology centers, for example, the Pharmacogenomics Research Network, MIDAS [Modeling of Infectious Disease Agent Study Centers of Excellence]. We took a very hard look at this. And we evaluated them. That was part of this process, but based on evaluations and based on the shift in emphasis with the end of the [NIH] budget doubling, that really became necessary. We moved those funds back into what we call the research project grant pool. The investigator-initiated bread and butter grant pool. That was one of the ways in which we moved the success rate up. We put over $100,000,000 or so back into R01’s. Then we’ve had other initiatives since then, like the MIRA program [The Maximizing Investigators’ Research Award], which I know you’re interested in talking about, which has cost savings associated with, part of it. We’ve taken the money from those cost savings and put them back into the pool as well.
Lorsch: There’s a dual goal there. One is, as I said, to support more investigators and more investigator-initiated research. The other is to support the sort of rank and file researcher at a somewhat higher level because the size of a modular grant hasn’t gone up in years.
Nunnari: It’s decreased.
Lorsch: Well, it has relative to inflation. One of our goals with MIRA is to make the median award size $250,000, which is about $50,000 more than an average NIGMS R01.
Nunnari: I’ve held one for years and I’m still not there.
Lorsch: I think that’s important but it’s for the people who are surviving on one NIGMS R01, which is a substantial number of people–70% or so of our grantees, something like that.
Nunnari: That’s an impressive number. When you go to make a change like that–“Okay, I’m going to take this money and reroute it”– What is the process here?
Lorsch: Decisions generally are discussed with our Council. We have to discuss starting a new program and get Council’s concurrence. Ending a program doesn’t require Council’s concurrence, although we generally discuss these things. To start a new program there is a formal vote but that’s using new funds in a different way. If we just want to end the program and redirect the funding back into the RPG [Research Project Grant] pool, we consult with Council but that doesn’t require a formal vote.
Nunnari: What about your team? Who’s on your team?
Lorsch: We have a senior staff and we make decisions carefully together with a lot of discussion as a group. That’s the division directors as well as some of the other office chiefs that review office budget, etc. It’s a team of probably about 15 people. Stephanie [Stephanie Older, Chief, NIGMS Office of Communication and Public Liaison who attended the interview] is one of them. That’s the main high-level decision making and vetting group. We also have lots of discussions with program staff. For many things we reached out to the community in various ways. We have Requests for Information frequently, which I know ASCB is often a very useful commenter. We have one coming up — NIH-wide– for the Strategic Plan for Data Science.
Lorsch: We have the open session of our Council where we have a public comment period. Kevin Wilson, [ASCB Director of Public Policy and Media Relations] and others have participated and made comments.
Nunnari: Can anyone come?
Nunnari: Do you generally get a good response from those requests for information?
Lorsch: It varies a lot. It’s always a very, very small fraction of the overall scientific community. I think one has to be careful not to treat it as a vote because it’s certainly not. A lot of responses–a really tremendous turnout– would be a thousand responses. If you think about how many researchers there are, it’s a tiny fraction of all the researchers.
Nunnari: That’s unfortunate. It’s such an opportunity.
Lorsch: It’s one of the values that a society like ASCB gives, which is that we at least hope we’re tapping into a larger swath of researchers by getting your input.
Nunnari: I would say our process is pretty… it’s a good bet that you are.
Lorsch: ASCB, FASEB [Federation of American Societies for Experimental Biology], ASBMB [American Society for Biochemistry and Molecular Biology], those kinds of groups do provide that, but individual researchers comment, and universities and other institutions also to send in their comments.
Nunnari: Was MIRA your baby?
Lorsch: Certainly, I guess you could say that. Others here had been thinking along similar lines. Peter Preusch [Biophysics Branch Chief, NIGMS Division of Biophysics, Biomedical Technology, and Computational Biosciences] certainly was one. He was the initial program director for MIRA. Something I had been thinking about before I even came here, before I even thought that coming here was something that would even be a possibility, was the idea of having a single grant to provide support for the research program of a PI. Now go do your research for five years and we’ll talk about a renewal then. When I came here and started studying the problems, changing the paradigm from this project-based funding model to a program-based funding model–in other words, to a single PI to support a program of research rather than individual projects in a lab. This could have really profound positive implications for the system. It was at a time where the Alberts et al paper [http://www.pnas.org/content/111/16/5773] had come out and they were talking about similar problems, with some similar ideas for things that might be changed. It seemed like the time was right to try an experiment that was relatively radical and see if we could actually change the paradigm for the better.
Nunnari: The MIRA program for the more established investigators started there. It has generated some controversy. Typically, if I’m talking to anybody about MIRA, there’s lots of misinformation, potential misinformation that gets shared, just propagated and so on. This is an opportunity for you to tell us, from the horse’s mouth, how it’s implemented, and what the scope of it is right now.
Lorsch: We just expanded the program. The initial pilot had two parts. One was the early stage investigators because we were determined we were going to let early stage investigators come into this funding stream because we thought that the benefits would be just as good for them as for an established investigator. Then there was the established investigator portion of MIRA. Because it was a pilot, because we didn’t yet have everything worked out, we didn’t want to open it to everyone where we would get potentially a thousand applications or more. In addition, the second goal was to help improve the distribution of funding for the reasons I told you; to find more people and to fund the average person better. What we did was to let in investigators who had a lot of funding; either two or more NIGMS R01s or one for $400,000 direct costs or more.
We told them that in exchange for the benefits of the program– an extra year of funding, not being tied to specific aims so you have much more freedom to change directions as ideas and opportunities arise, and more stability of funding– they were going to have to give up a little bit of their funding. On average, the cuts were about 12% relative to their recent historical NIGMS funding. That’s where some of the myths and misinformation have arisen as what is somebody’s recent historical funding from NIGMS. I can’t speak of any specific cases, but I can kind of give you some, aggregate examples.
For example, let’s say an investigator, in the past five years before they applied for the MIRA, had two years where they had three R01s and three years where they had two R01s. In the fiscal year before they apply for the MIRA, they only had two. When we decide they got a good [MIRA] score, when we look at their budget, they often wanted us to consider that they were a three R01 person. But we can’t consider them a full three R01 person. One, because they didn’t have the money, it wasn’t there, for one thing. This is a renewal and we would be giving them an extra grant so we had to be very careful about that. Also, in fact, only in two of the past five years did they actually have three grants.
We would look at the applicant as a two a grant person. That’s where some of this disparity came. What we considered their historical funding and what someone who three years ago had three grants considered their historical funding was often different. Other things that cropped up as confusions have been that some investigators thought that their grants from other institutes should be counted in the funding calculation and that we would be picking up that funding as well.
Nunari: And they can still hold those grants?
Lorsch: They can.
Nunnari: Just not anymore at NIGMS?
Lorsch: That’s right. There have been people who were unhappy because they thought that their grant or grants at another institute were going to be counted in their funding history and folded into their NIGMS MIRA award.
Those are the kinds of issues. People would very much like us to say here is the exact rule and show them the algorithm to calculate their budget. The trouble is that there are dozens and dozens of different situations. There were almost as many different situations as there are people with multiple grants. It’s not possible to say, here’s the algorithm, you cut it 12% and there’s the number. We really have to look at each case individually and decide.
Nunnari: Can people negotiate? Are some people more persuasive than others?
Lorsch: We are very firm to not let that happen. I think it’s made us unpopular. But we have been pretty strict that once we make an offer of how much money we’re prepared to offer you for this MIRA award, that’s what it is. We’re also very clear with people, if you don’t want that, if you don’t like that, then you can resubmit your R01 and go back into the R01 stream. There’s a tradeoff here. We are asking you to take somewhat less money.
Nunnari: As a corollary to the MIRA program, do you feel like your program officers have gotten tougher about multiple R01s.
Lorsch: Yes. And we’re headed even more so in that direction.
Nunnari: When you say they can go back into the multiple R01 stream they have to do so knowing that it’s even more difficult.
Lorsch: Absolutely. But again, it’s part of the calculus you have to do.
Nunnari: Are other institutes as equally tough on the issue of multiple R01s?
Lorsch: It varies across the NIH. Some take a very hard look at it like we do. Others, maybe, less so.
Nunnari: What’s your target there? Is it going to be a uniform application like the MIRA program? Are the second R01s going to have to hit a certain percentile?
Lorsch: We have discussed whether we want to actually put a hard cap on the number of R01s someone could hold from NIGMS and we haven’t moved there yet. The direction we’re moving is that it is becoming increasingly difficult to hold more than two R01s and the bar is getting higher and higher for getting a third or fourth R01 from NIGMS. In addition to that, we’ve always looked at people’s other funding when we’re making funding decisions.
Nunnari: Right, at Council.
Lorsch: There’s, for instance, our 750K policy.
Nunnari: That’s not a hard line, is it?
Lorsch: Well it’s a hard line if someone gets, will get, with this grant that we’re talking about, over $750,000 in direct costs from all sources, not just from NIH.
Nunnari: Including private? Including HHMI [Howard Hughes Medical Institute]?
Lorsch: Yes. All sources, including from all sources.
Nunnari: Gee, how do HHMI investigators hold NIGMS grants?
Lorsch: I’ll get back to that in a second.
For us to fund that grant requires Council to take a special look at the application and actually say yes, this grant in particular you can consider for funding. Quite often they say no. If they say no, we can’t fund it because they decided that when you do a cost benefit analysis, that’s not the best investment of taxpayer money relative to all the other meritorious grants that you could fund, and sometimes those are grants that scored in the one percentile.
Nunnari: That’s a hard one. I’m more of the meritocracy kind of person. That one’s tough for me.
Lorsch: This idea of merit as A) something that one can measure in a precise way, and B) in a rank order way, I think is flawed. I think merit is something that you can say here is a group of applications that have a high likelihood of producing something valuable.
Nunnari: You’re actively saying this grant is really quite good and we’re just choosing not to fund it.
Lorsch: But remember, what we’re saying is we’re choosing not to fund that application so we can fund another meritorious application and that other meritorious application quite literally could be an early stage investigator who’s just started her lab. You can’t make a one-for-one exchange because you don’t necessarily know that it goes back into the pool and something else gets funded. But that literally is what is going on in those cases that we’re finding. And that’s also part of our funding philosophy, that we prioritize grants to meritorious investigators, as judged by peer review, who otherwise would not have funding if they didn’t get that grant. That is an explicit part of our funding strategy. We would prefer to give a grant to someone who won’t be funded, a talented promising investigator doing good work than give a second or third grant to someone who will still be funded if they don’t get this grant and will still be able to do good work.
Nunnari: Just not the work that they proposed.
Lorsch: Well, that’s part of the problem with the project-based funding system. You get locked in, this grant is for this thing and this grant is for that thing and this grant is for something else and then there’s the other thing I’m most interested in. We run into that situation a lot in Council. The person says the grant I’m actually most interested and excited about is the one that you’re talking about not funding right now. I’d actually like if you’re taking one of the other ones away, I’d be happy. So we do negotiate those things and say, okay, well maybe we’ll give you this one if you relinquish that one. If you don’t have project-based funding it’s much easier.
Nunnari: But running a lab, there are times in your career where you just need more money. You have a burst of innovation. I’ve never found personally R01s to be that restrictive in terms of what gets done versus what is proposed. In practical application, I think when people look at a grant, they look at the investigator quite a bit.
Lorsch: Certainly the way it’s supposed to work is that you promote specific aims and then when you come in for review, we say, did you do this stuff or not? Now if you did everything in specific aim plus more, no one’s going to complain. It’s risky as a strategy. I can tell you firsthand because this is what happened to my first grant. If you do something totally different, even if that totally different thing is better than what you proposed, you can still get beat up on in study section because you didn’t do what you were supposed to do. If you go to MIRA, you don’t have that problem. If something more exciting comes along, you can do it.
Nunnari: Coming back to MIRA, I understand the rationale for the senior investigators based on everything you’ve said. I don’t understand it so much as a mechanism for juniors. I think that there are a lot of good practical things that have come out; for example, the way those grants get peer reviewed. I don’t quite understand why you’d want to limit a junior investigator right out of the blocks.
Lorsch: One of the things we did when we started the ESI [Early Stage Investigator] program was to split the review off for the early stage investigators from the established investigators so they were reviewed in separate study sections and the early stage investigators have their own review criteria.
Nunnari: That’s wonderful.
Lorsch: I think that is a great thing that we’re now comparing apples to apples, oranges to oranges. In terms of the funding, this is another kind of misinterpretation or myth. The maximum they can request as an early stage investigator is $250,000. Almost all of them that we funded got $250,000 in direct costs. That is almost $60,000 more than they would’ve gotten if they had gotten an R01 as an early stage investigator from NIGMS. They’re getting more money. Now when they come in for renewal, they’re not locked in to $250,000 anymore. If they did spectacularly well, they can request an increase in the budget and we can give it to them.
Nunnari: Have you been through that round yet?
Lorsch: No. This is only two years old.
Nunnari: And for the seniors too?
Lorsch: We’re not even up to renewal this year. We’ve got a couple of years.
Nunnari: How many people are involved in this experiment?
Lorsch: In terms of how many grants that we awarded?
Nunnari: In terms of investigators.
Lorsch: There’s several hundred, 400 something. We’d have to get you the exact number, but there have been about 200. [NIGMS later reported that there are 231 Established MIRA grantees and 192 ESI MIRA grantees]
Nunnari: Is that study powered enough? [laughter]
Lorsch: I didn’t do the power calculation. [laughter]
Nunnari: I think you need to. [laughter] Do you have outcomes built in?
Lorsch: Yes, we set up an evaluation program that is built into the program with short, medium, and long-term measures; Do people keep applying? Do people want to do it? Some of it is looking at the productivity and impact of the research. That obviously takes somewhat longer to look at. What are the impressions of reviewers of the applications? We have a nice experiment which is that we have the R01 pool going on at the same time so we can compare the things to one another.
Nunnari: Is the MIRA program capped?
Lorsch: How many people? No.
Nunnari: The juniors could all go there.
Nunnari: The word on the street is the success rate is much greater for the MIRA junior program than if they were to stay in the R01 pool.
Lorsch: It’s about the same.
Nunnari: That’s not the word.
Lorsch: Yeah. Well that’s good. I mean it was a little higher one year, but the same the other. We are making it about the same as ESI R01s.
Nunnari: Is it the same for junior people? In other words, when you go into the pool and pluck out the juniors, they only look at their success rate.
Lorsch: It’s roughly the same, but we’re very focused on it as a pool. So we try to fund as deeply into it as we feel is reasonable.
Nunnari: Can you just comment briefly on the peer review of those?
Lorsch: It still looks at track record just as the established investigator does, but of course the track record is shorter and is focused more on graduate student and post doc time. They have to say, as do the established investigators generally, what are the research questions they’re proposing to study, why is it really important to answer those questions and why are they the right person to answer them. They’re not getting bound to specific aims. That’s similar. The trick, which is also true in the established pool, is that there are investigators from different parts of the ESI career stage. There are people who just started their labs as an assistant professor, and there are people who have had labs for five years. One of the difficult things reviewers have to struggle with is normalizing those two things. They have to say, okay, this person doesn’t have three papers, but they just started their lab and I think their work is really cool so I’m going to score them as well as the person who was in their fifth year and has three papers.
Nunnari: These are broad panels too?
Lorsch: They are broader than people are used to for an R01 in terms of expertise because they’re looking at a wider swath. We’re looking for reviewers who have the ability to step back and think more broadly about a scientific area. Not just their own specific area but who think, oh, that sounds really neat. Even though it’s not my field, I can see why that’s really important and someone should solve that.
Nunnari: You don’t think the quality of the technical review is compromised by that?
Lorsch: Well, again, we’re not asking them in any applications. There is not information about the details of the experiments. There are people on the panel generally who can assess, did this person get the right training to do this work, do their papers look high quality and rigorous, and, generally, does what they’re talking about seem like it’s headed in a good direction. However, did they propose the right controls, that kind of stuff, that’s not part of it. It’s an experiment to see if it really matters if they propose all the right controls in their actual grant if the reviewers are able to say that this is someone who knows how to do good science and does good science.
Nunnari: The numbers say that, we’re talking about juniors, that we’re funding a senior investigators at the cost of early career scientists, which is very concerning I think for our endeavor. MIRA is one program I think, of a number of measures that uniquely addresses that problem, including the review process. There are other programs like that at the NIH. Why are there so many? For the MIRA junior, this idea of comparing apples and apples, oranges and pears, has some merit, In fact it’s been used by the ERC [European Research Council] with good outcomes. Can you just comment on why there are so many programs like this?
Lorsch: It’s important to remember that there are 27 Institutes and Centers at the NIH. Each one has a specific mission and those missions are distinct. What works well for one institute’s mission may not be appropriate for others. For instance, an institute that is very focused on a certain clinical area or certain organ system may really be working in a different space than NIGMS, with its focus on fundamental research. Different programs do make sense in different contexts. That’s one thing to remember. In addition, there is actually piloting and experimentation going on. The hope would be that the models that are successful will broaden and expand. Certainly, I would like to see the MIRA approach adopted more widely if it proves to be ultimately successful here.
Nunnari: Specifically for juniors or for both?
Lorsch: Both. There are programs that are trying to achieve distinct things. The DP2 [NIH Director’s New Innovator Award Program] award for example, has some similarities to MIRA in that they also do their own review. It’s not focused on specific aims but broader, more holistic, forward-thinking things. They have more of an emphasis on high-risk, high-reward, highly innovative work. That’s fine. But it’s not an essential component of MIRA. Someone can be doing a series of experiments that are the next logical steps but are the only way the field’s going to move forward. That’s great for MIRA too. Just as great as if someone is doing some very out of the box high-risk research. DP2 is really for out-of-the-box, high-risk research for junior people. It’s a certain niche that it’s filling. I think it’s a very important niche, but a DP2 by itself is not going to fund all the ESIs that are out there who are doing great work. Because, let’s face it, most of them are not doing the crazy out-of-the-box research, which isn’t to say they’re not doing fantastic, amazing, important science.
Nunnari: Something like the ERC, they have a very diverse mission. Having peer review organized by career stage just makes a lot of sense and that should go across the board at the NIH.
Lorsch: I think you’re right. I think that is a very important thing to think about. Certainly, that’s what we are doing in MIRA. The issue is how well does it scale and what will be the implications for review workload for example. If you had to split every study section in order to review all the ESIs, just think, CSR [NIH Center for Scientific Review] would have to double the number of study sections. That’d be a huge increase in workload and costs. We’d need probably considerably more additional reviewers from the community. Those are the balances that one needs to consider.
Nunnari: Going back to Utopia, if we could just wave a magic wand and have peer review organized in that kind of way, another consideration for the NIH would be what revenue would be put into each of those groups. What would you do?
Lorsch: It’s a very good question. It’s one we’ve wrestled with internally a lot. What is the right number of ESIs coming into the system, what is the right pool of mid-career people, more senior people, et cetera? It’s a very complicated question because of demographic shifts, right? Society changes, science can change. I think it’s hard to say, ‘here’s exactly the right number that will be true 20 years from now too.’ That complicates it.
Nunnari: You could tune the size of the grants even right to the stage and probably run some numbers and control the pipeline that way.
Lorsch: People have wanted to do that for a long time, but when you talk to economists and demographers about the possibility of making predictive models of workforce, they tell you that that’s naive. In your utopian world, [laughter] you can do that. I think in the real word, there are far too many variables, too many known unknowns and unknown unknowns to accurately do that in a way that would make sense.
Nunnari: NIGMS and NIH have made stabs, though, at tuning? A little bit. One of them was the GSI [Grants Support Index] and that failed. What did you learn from that failure?
Lorsch: I don’t want to preempt the NGRI Working Group [Next Generation Research Initiative] that’s hard at work right now and is thinking about what to do in this space. The goals of GSI were, in my opinion, very laudable. They’re all the goals I told you about; improving the distribution of funding, making sure we can support more investigators, and getting more ideas in the system. Change is difficult and hopefully the NGRI process is going to come up with something that meets these goals in a way that is more palatable to the community.
Nunnari: I think there’s a lot of concern in the community around this process, mostly to do with unintended consequences.
Nunnari: We all are aware that we’re in a fixed pie kind of situation, so where’s the money coming from to fund the program?
Lorsch: Larry Tabak [Principal Deputy Director, NIH] talked about this at the ACD [NIH Advisory Committee to the NIH Director], and gave an update on the NGRI Working Group. One of the concerns the working group had was this issue of unintended consequences. NIGMS did a modeling study that raised significant concerns that if we just implemented NGRI the way it initially been proposed, we would’ve ended up defunding as many meritorious mid-career and senior people and taking their only grant away–as ESIs and this new category EEIs [Early Established Investigators] that we funded. Very quickly the working group– and they had already honed in on this issue even before we did that study– raised concerns that it wasn’t a good approach. Unless there was some way to at least direct the flow of the money from one place to another, you could do harm when you were trying to do good. Something the committee is trying to figure out is a better approach.
Nunnari: Do you think that GSI was a better solution for this problem?
Lorsch: I think it identified a problem in the system, which is a maldistribution of funding. Interestingly, it was a very similar approach to things FASEB had proposed in their Sustaining Discovery report, and that Alberts et al proposed. It wasn’t that the NIH pulled this thing out of thin air. There was a lot of thought that went into this and a lot of input came from these places. The Madison Report that was put out around the same time proposed similar ideas. These ideas have actually all been proposed in various ways by the community and NIH was trying to be responsive to the community.
Nunnari: Speaking of Albert’s et al, it was a very influential opinion piece that was written in PNAS [Proceedings of the National Academy of Science] and it focused on the problems associated with the biomedical work force pipeline. In my mind, it primarily focused on addressing the flow into the pipeline.
Lorsch: And how to support people once they are there.
Nunnari: Right, and not addressing what we just were talking about, which is that senior investigators are increasingly taking a bigger piece of the pie. The analogy is that there’s a clog in the pipeline. In looking at the flow into the pipeline, how influential was this article in some of the decisions that have been made and are currently being made at NIGMS, specifically about training the workforce like graduate students and postdocs?
Lorsch: Those questions have been under intense consideration for a long time. Certainly, they started before Alberts et al, with the biomedical workforce report that Shirley Tilghman and Sally Rocky chaired. There are a lot of real issues and problems that have been identified through those kinds of studies but many of them are extremely difficult and [solutions] may be intractable to actually implement. It’s not that NIH doesn’t recognize those problems and hasn’t worked hard on solutions to them. But to think that they’re easily solved is probably incorrect.
Nunnari: Let me be more specific. I understand that there are some pretty big changes coming down the pipeline for training grants.
Lorsch: From NIGMS. We have a whole new training grant funding opportunity announcement.
Nunnari: It’s causing a lot of anxiety in the community. It’s viewed like another hurdle to jump over. Is that related to the workforce issues?
Lorsch: There are elements of it that certainly are. Things such as reporting outcomes of graduate training programs are something that Shirley Tilghman, for instance, has called for repeatedly and is now codified in the FOA [Funding Opportunity Announcement]. Programs are expected to report their outcomes on a publicly available website so that students–both applicants and current trainees–can make reasonable judgments on whether they should go to a certain program and what they can expect as an outcome. Similarly, training in a career development program or programs within training grants is another area of emphasis that we’ve put into this new FOA.
Nunnari: Nontraditional, not academic careers?
Lorsch: Nontraditional, exactly. [There are a] multitude of different pathways that someone can go down and still be contributing in important ways to the scientific enterprise, like this guy here. [points to ASCB’s Kevin Wilson] We’re focused on improving the diversity of training, both in terms of the people doing the science, and the trainees themselves. Changing the focus of the didactic portion of the curriculum away from a kind of fact-based teaching model to one in which we are focusing on the range of skills that are needed to be an outstanding scientist. These could be technical skills; they can be operational skills like experimental design, strong interpretation of data, things of that nature. And then there are the professional skills; communication, education, teamwork.
Nunnari: Those are big changes.
Lorsch: We’re asking for big, big changes. Is this just another hurdle? No, we’re looking for transformation here, not just adding new bumps in the road. We are looking for a new road. We are not asking programs to just add stuff in. It specifically says this in the FOA. We’re not asking programs to just tack new stuff on in the way they might have done for some requirements in the past. We want them to relook at everything they’re doing and rework it. If someone wants to teach skills instead of facts, don’t just put a skills course in. Get rid of your facts course and replace it with a skills course or get rid of most of the facts courses.
These are the kinds of things we’re really hoping to see. Big changes. I do totally recognize that everyone’s taxed. Everyone’s busy. They’re trying to get their grants funded. Hopefully if they apply for MIRA and their funding is more stable, that’ll free up some time for them. But this is taxpayer money that’s for a purpose and it’s a privilege that institutions get to use it, to improve their training and to produce well-trained outstanding scientists–PhD’s. If someone feels like it’s too much work and they don’t want to do it, I’ve got plenty of people who want it to it and will put the effort and time into it. I don’t mean to sound harsh, but that is the reality.
Nunnari: Is there going to be some latitude? They’re such big changes. You can’t expect a program to just turn on a dime.
Lorsch: There’s definitely going to be a ramp-in phase. I was talking to my review staff the other day about this. The way they’re looking at it is what’s going to be viewed as an excellent application in the first round probably wouldn’t be reviewed as an excellent application two years from the first round. There’s going to be an evolution of expectations. What we’re looking for is real commitment, real innovation, and good ideas.
Nunnari: Whose brainchild was this?
Lorsch: This was something that came from a lot of us. Certainly, I’ve been thinking about this for years.
Nunnari: Was this something NIGMS’ Council voted on?
Lorsch: Yes, absolutely. This was approved by Council after discussion with Council over the years.
Nunnari: What if you look at the distribution of training grants now instead of investigators, if we put our MIRA hat on?
Lorsch: We’re talking about how we can improve the distribution of training support so that we reach more outstanding potential researchers in the country and potentially outstanding institutions. We actually have a new area that institutions can apply for a training grant in Transdisciplinary Basic Biomedical Sciences. It’s only open to institutions that don’t have one of our basic science training grants already or institutions that have multiple ones and want to merge some of their programs.
Nunnari: I wonder how many will take you up on it.
Lorsch: It’s an experiment. This idea came from community input. Allison [Allison Gammie Director, NIGMS Division of Training, Workforce Development, and Diversity] was talking to different institutions and multiple times deans or program directors asked if they could merge two or three of their programs. This is a mechanism to allow them to do it and we’ll see if they take us up on it or not.
Nunnari: Interesting. What projects are you currently working on?
Lorsch: This is one of them.
Nunnari: Which one like really excites you the most?
Lorsch: Certainly, I would have to mention data science. I’m the co-chair of the NH Scientific Data Council [https://datascience.nih.gov/bd2k/about/org/SDC], which is an NIH-wide body that’s responsible for overseeing, organizing, and making recommendations to NIH leadership on issues related to data science on behalf of the ICs [NIH Institutes and Centers] and the Office of the Director. We were charged with developing a data science strategic plan to help guide NIH’s efforts in enhancing the ability of the scientific community and other stakeholders to engage modern data science approaches in advancing biomedical discovery. We’ve been working on that for several months now. I presented to NIGMS’s Council, I presented it at the NIAMS [National Institute of Arthritis and Musculoskeletal and Skin Diseases] Council and at the National Library of Medicine and other meetings. And, we’re about to put out a draft version of the plan, which I’m actually pretty excited about, for public comment in the next couple of weeks. You should see an RFI [Request for Information] out— 26 pages—soon.
Nunnari: Why are you so excited about it?
Lorsch: I think it really focuses NIH and hopefully the rest of the ecosystem, the rest of the community, on what we need to do to make use of the opportunities that data science and big data are presenting us to overcome the challenges that we’re going to need to overcome. It’s something that’s overdue. When we started looking at the data science ecosystem, especially the resource part of it (the databases), [we noted] these are all models that grew up 30, 40 years ago really. Some of them started as books, literally books. And then they became databases on punch cards and then on tapes. Those models just slowly evolved. There’s never been a careful, top down, holistic look to say, ‘is this the best way to arrange this both for the research community to make it as useful as possible and for the taxpayers in terms of getting the most for the money. The answer to both of those I think is pretty clearly, no. This is a great opportunity for us to try to help the community to improve that.
Nunnari: Would there be money that comes with that?
Lorsch: We’ll see. There would be a need for an infusion of money. Where that comes from, whether it’s individual ICs [Institutes and Centers] or something else, has to be seen. I think there’s a lot of commitment at NIH to truly meet the challenges and the opportunities posed by data science and big data. Take an individual example like the model organism databases, right? Right now there are all these separate databases. If you want to compare the fly sequence to the worm sequence to the yeast sequence, you’re basically going to each of those databases and pulling them out and then putting them in a document comparing them. That doesn’t make any sense in this day and age, right? A graduate student, postdoc or you, Jodi, should be able to go to one place and compare all the sequences instantly and get all the information about the gene products, et cetera, without having to jump around.
Nunnari: I spend a lot of time doing that.
Lorsch: Absolutely. So we need to de-silo this system. When we do those things, when we improve the integration, de-silo it, when we focus on the right things, when we focus on user service and efficiency and support for the community, when we’re evaluating data resources, we will actually find some economies. We’ll be able to actually get more for our money because I think there’re inefficiencies in the system now that we can actually fix.
Nunnari: It does make sense for NIH to do this; this is one arena where top down seems necessary.
Lorsch: At least as a lead. It’s going to take the community to help. It’s another case of change and change will be painful for some segments of the community. Our goal is to make it easier and better for you as researchers to use data using data analytical techniques. Right now, as you were saying, there are some challenges to doing that, [which] shouldn’t be in place in 2018.
Nunnari: Is this what’s exciting you?
Lorsch: That’s one [thing]. Certainly, another area that we’re working on internally at NIGMS has to do with diversity. If you look at the diversity of PhDs being produced in the biomedical sciences, the number of under-represented minority students getting PhD’s has actually increased about nine-fold in the last 25 years. It’s quite impressive. But if you look at the diversity of the professoriate, at least at academic medical schools, it hasn’t changed at all in the same period; it’s flat. That plus some other analyses that have come out recently have really indicated that a failure point is the transition from postdoc to faculty. People just aren’t making that transition for whatever reason.
We don’t know if they’re choosing not to or if they’re trying and not being successful or some combination. We’re working on interventions, a new program that would help bridge that gap and allow a more diverse population of postdocs to become successful academic faculty members running their own independent labs. That’s another thing we’re excited about.
Nunnari: That would excite me. Maybe this is a good place where I can ask how the ASCB can help.
Lorsch: I think I already mentioned the important role ASCB provides in giving the NIH advice and feedback, and [I] urge you to keep doing that. It would be great to see ASCB in terms of it’s the leadership, embracing this idea of diversity as well.
Nunnari: Absolutely I think that’s one of our main goals. That would be an area where it would be great to partner.
Lorsch: I think those are important things. Really supporting the early stage investigators, the trainees, and students. They’re becoming very dispirited, quite understandably. All those papers and reports had a galvanizing effect and therefore a helpful effect. They also did have the effect of further dispiriting people because they pointed out a lot of the problems in the system. That’s not to criticize them in any way, but I think we do have to begin to really make it known that there is positive change going on at NIH and elsewhere and that things are actually moving in a more positive direction. If those younger people, and older people as well, get involved, we can keep the momentum and really fix the system.
Nunnari: When do you think we’ll hear about this NGRI Taskforce?
Lorsch: June is when it’s supposed to be reported to the ACD.
Nunnari: Is there a commitment or is there a no-go button in there somewhere so it meets a commitment to see the program through?
Lorsch: In June they’re supposed to report to the ACD. The committee is supposed to report to the full ACD on its recommendations. We will see what those recommendations are. I don’t know yet what will come out of it, but I’m optimistic.
Nunnari: Just backing up a little bit, you have MIRA young investigators and senior ones. They’re separate. You said the programs aren’t capped at this point. Is that a consideration in terms of the money flow, in a similar way?
Lorsch: Yes, these are all research project grants, right? It’s trading one kind of research project grant for another. An investigator who now has one or two R01s decides she wants instead to get a MIRA. And she gets a MIRA. If that works better for her, then that’s good. That’s the way we want to see it. It’s not that we took funding away from someone to give it to someone else in that situation, for example with the ESI’s. If an ESI gets a MIRA instead of getting an R01, they get actually more money. It’s a good thing.
Nunnari: That’s what I’m getting at.
Lorsch: We are looking at it as a total market economy. We’re letting people vote with their feet. If they want MIRAs, they apply to MIRA. It’s not detracting from the net pool in that it’s not reducing the number of RPGs [Research Project Grants] or people who have RPGs. Again, we’re managing towards funded investigators.
Nunnari: And at some point we’ll have metrics out on that?
Lorsch: We put them out frequently. You’ve probably seen our Feedback Loop posts [https://loop.nigms.nih.gov/] on the status of the MIRA and we’re getting ready to put a funding trends post out that you’ll see soon, which will have information about success rates and funded investigators [the post published on 2/27/18: https://loop.nigms.nih.gov/2018/02/application-and-funding-trends-in-fiscal-year-2017/] . Sometime, hopefully not too far from now, we’ll have another MIRA Feedback Loop post. It gives more information.
Two things about MIRA that might be useful for you. We’ve seen the number of ESI applicants to NIGMS almost double since we started the MIRA program. That’s just fascinating to me. They, by definition, are all doing science relevant to our mission.
Nunnari: Where do they come from?
Lorsch: I don’t know. I’m not sure.
Nunnari: They would have applied to another institute?
Lorsch: Some of that. However, these are people who are really doing work that’s related to the NIGMS mission. If they were trained to do something totally different, it’d be a little hard for them to switch. Maybe there’s been an increase to some extent in hiring. I don’t know. That would be very nice if that were true. It’s a fairly short period of time for that to have happened. Maybe these were people who had been on the verge of giving up and came back into the system. That would also be nice if we were bringing some people back. We need to study that. It’s an interesting question.
Nunnari: Even senior investigators don’t need to have two R01s anymore.
Lorsch: Not anymore. I was going to mention that we’ve now opened it up. If you have one NIGMS R01, you can apply for a MIRA when you’re grant comes up for renewal.
Nunnari: That’s a no-brainer.
Lorsch: I would think so. If you just have one R01 and its, say $210,000, we’re going to try to give you 250 [$250,000] and you get an extra year of funding. Plus, you get all the flexibility and the stability. To me, what’s not to love?
Nunnari: If you’ve never been a two, three [R01’s]
Lorsch: Exactly. The only reason you would not do it, say, is if you somehow think you’re going to get more R01s. The numbers aren’t on your side, just even historically the numbers on not your side. The other thing I was going to mention is when we look at the people awarded ESI MIRAs, they are about a year and a half, almost two years younger than the people awarded ESI R01s from NIGMS. Somehow, we are funding people earlier, which I think is really great. It’s only two years of data. Let’s see if the trend continues. We’ve got to study it, but I’m optimistic. I think that’s a pretty neat thing because one of the issues we’re concerned about that we didn’t discuss is how long people are taking before they get their first grant. 39 is the average age at NIGMS.
Nunnari: They’re getting their faculty jobs, I know because I mentor all of our junior faculty, and sometimes [they are] waiting all the way up until they can, existing on their startups, which are enormous. Burning their startups up and then applying. They wait, they delay getting into the system. My anecdotal experience with our juniors, [is] this last crop has immediately gone for the MIRAs. They feel like they have a better shot because they’re getting separately reviewed, they’re feeling more confident that they’re getting assessed on their skills and not in comparison to somebody way more senior with a lot more proven productivity. This was huge, in my mind. MIRA, no MIRA, whatever. Having them separately peer reviewed is just enormous.
Lorsch: What you’re saying is consistent with what we’re seeing, which is that we’re funding people earlier in the MIRA program.
Nunnari: Good unintended consequence perhaps.
Lorsch: I’d like to push it earlier. I mean, 37 is still kind of late to me. I got my first grant; I think I was 30 or 31.
Nunnari: That was fun. That was great. Thank you very much.