DEB Numbers: Per-Person Success Rate in DEB


Our friends at Dynamic Ecology posted a little while back about the NSF-wide trends in per-person success rate based on this 2014 report to the National Science Board that provided merit review process statistics across the whole agency[i]. There were several questions in the comments to that post regarding the context for the numbers and how they would look for DEB or IOS, especially since preliminary proposals were explicitly excluded from the calculations in the report to the NSB[ii].

So, we’ve put something together with DEB data to follow-up on that discussion. Our analysis sticks to the general approach of the NSF-wide report with modifications to allow inclusion of preliminary proposal data.

Part 1: Inclusion Criteria

First, let’s be clear about what we’re counting here. The NSB report’s Figure 14 illustrated a per-PI success rate based on counts of Competitive Research Grant Actions leading to Award or Decline decisions. That institutional jargon terminology specifies 3 different filters to define what was counted,

A context filter: Competitive (a stand-alone grant request) versus Non-competitive (changes to an existing grant such as a supplement or a PI move to a new institution) decision-making;

A content filter: Research (just what it sounds like, both Core and Special programs) versus Non-research (e.g., fellowships, dissertation support, travel support, conferences) activities;

An outcome filter: Awarded or Declined versus Any Other Outcome (e.g., invite, not invite, still pending a decision, returned without review, or withdrawn before a decision)

This is actually a really good set of filters for narrowing down the universe of “stuff NSF does” to questions about “bread and butter” grants. Ignoring the Any Other Outcome proposals is a good thing since those categories of proposals were never actually part of the competition in most cases across NSF. On the other hand, it complicates measurement of programs where large numbers of preliminary proposals are involved, as is our case.

Part 2: The Proposal Data

Our first table presents the big picture of proposal submissions for DEB for a period of 2006-2014 (chosen mainly because that was the span of complete years beyond which the server was getting angry with us for requesting too many records, #overlyhonestmethods). We’ve divided them up following each of the filters mentioned above and also split out the DEB-relevant sub-units. (Note: for consistency across all of the different proposal types and with the NSF-wide data, this table counts separately all proposals with unique identification numbers in FastLane. This differs from the way DEBrief usually combines separate proposals from collaborative group into a single “project” unit for counts.)

PerPI_Success_2014_1(click image to make legible)

We have discussed some of these trends before, but to quickly review the basic points:

1) Total actions spiked with the launch of the preliminary proposal system in 2012 but have since come down a bit. This was preceded by another spike in 2010 that was in part a reaction to stimulus funding in 2009 (evidenced by upward jumps in DDIGs, and Core programs from 2009-2010) and also a major spike in special programs that reflects the launch of Dimensions of Biodiversity and some other redistribution of special program responsibilities between Divisions in BIO.

2) Economic stimulus (ARRA) money in 2009 and the wiggle-room gained by clearing out some of the backlog of requests and paying down future commitments resulted in significantly elevated award counts in 2009 and 2010 that distort the longer-term pattern.

3) Incoming preliminary proposal numbers (2012-2014) have been nearly flat, as have the number of research grant award actions, especially when considering both core and special program components over the entire period.

We’re not adding per-proposal success rates to this table specifically because the preliminary proposal process crosses fiscal years and the corrections needed to account for the complexities of the process make that number very different from the straight-forward single-year data above (see endnote i). Per-proposal success rates are shown in our FY2014 wrap-up post.

Part 3: Per-Person Calculations

Each action in the table from Part 2 links to a record of between 1 and 5 persons (PIs and CoPIs) on the proposal cover page.

[Contextual tangent: we are not differentiating between core and special programs in DEB for the per-person success rate. Could it be done? Sure, but the special programs and core programs are both funding research grants and we see that applicants to one or the other quite often switch targets depending on the convenience of deadline or opportunity. Ultimately getting one or the other provides the same result, research funding.]

In total, there were 11,789 unique PIs/CoPIs associated with the 20,724 Competitive Research Grant actions in DEB between 2006 and 2014. During the same time frame, DEB made 2,671 Competitive Research Grant Awards that included a total of 2,970 unique PIs or CoPIs. Most individuals (75% of unique PI/CoPIs) who applied to DEB for funding never received a Competitive Research Award during this entire 9-year period.

The NSB report calculated PI success rate in a 3-year moving window, we’ll do that in a moment. First, we want to split it a different way to account for the stimulus (ARRA) funding in 2009; when combined with the smoothing of the window, that spike in awards winds up distorting some details we’d like to explore.

Annual Per-Person Success Rate

Fiscal Year 2006 2007 2008 2009 2010 2011 2012 2013 2014
Total Unique PIs and CoPIs Applied 2052 2057 2060 2274 3266 2733 4310 4546 3950
Unique Women Applied 503 476 488 559 791 676 1127 1211 1093
Total Unique PIs and CoPIs Awarded 352 449 424 620 601 392 424 455 414
Unique Women Awarded 93 105 114 166 163 100 110 126 110
Per-Person
Success Rate
17.2% 21.8% 20.6% 27.3% 18.4% 14.3% 9.8% 10.0% 10.5%
Per-Woman Success Rate 18.5% 22.1% 23.4% 29.7% 20.6% 14.8% 9.8% 10.4% 10.1%
First Recordings of a PI 2052 1149 909 1006 1553 943 1645 1542 990
Last Recordings of a PI 526 542 471 595 1085 780 1600 2240 3950

The notable patterns here:

1) The preliminary proposal system brought in a huge increase in persons applying each year, double pre-stimulus levels.

2) 2010 was a big year, matching what we saw in the proposal load table, with a large increase in people submitting in reaction to the economic stimulus (ARRA) and following the movement of special programs into DEB.

3) These additional PIs were actually “new” people who had not submitted to DEB since at least 2006; and, we saw about 50% higher numbers of new people for each of 2010, 2012, and 2013 than typical in previous years. But, 2014 looks more like the longer-term norm.

4) The stimulus funding had a big, but temporary, effect by allowing an extra ~200 persons to be funded in both 2009 and 2010. While the effect on per PI success was large in 2009, it was much less in 2010 because of the 1000 additional applicants that year.

5) Excepting the stimulus years, the number of persons funded by research grants doesn’t show a trend or even all that much variation over this span: ~420 unique persons per year.

6) The growth in unique PIs we see includes both an absolute increase and an increasing proportion of female investigators among applicants, although the temporal range is small and the female proportion of applicants has yet to exceed 28%. At the same time, women have generally experienced a per-person success rate (17.7 %) similar to that of the general population (16.7%).

PerPI_Success_2014_2

A Quick Sensitivity Check

There’s a legitimate question as to whether counting PIs and CoPIs provides the best metric of success. Perhaps we should count just PIs? This is what the NSB report does. However, at the preliminary proposal stage, with only a single proposal cover page per project team, there are many instances of collaborative PIs that appear as CoPIs as well as collaborative PIs and CoPIs that don’t appear on a cover page at all. The constraints of the FastLane submission system at the preliminary proposal stage generally lead to undercounting total participants and artificially inflating the balance of CoPIs relative to PIs[iii]. Counting only PIs causes two problems: 1) it ignores a portion of the population at the preliminary proposal stage that would have been counted on full proposals and 2) it would artificially raise the per-PI success rate under the two-stage process relative to the pre-2012 submission process. So, to reflect funding reality as best we can, we cast a wide net and include everyone from the cover pages in the calculations above. However, we can also look to see if the numbers come out any differently if we constrain our calculations to only PIs. Other than the counts being somewhat smaller, the per-person success rates are generally not changed and are tightly correlated with results shown above.

Fiscal Year 2006 2007 2008 2009 2010 2011 2012 2013 2014
Total Unique PIs-Only Applied 1299 1322 1358 1419 1973 1706 2390 2305 2140
Unique Women Applied 314 303 321 366 506 458 684 687 660
Total Unique PIs-Only Awarded 232 292 276 397 361 250 266 261 257
Unique Women Awarded 56 63 69 113 108 66 74 76 63
Per-Person SuccessRate 17.9% 22.1% 20.3% 28.0% 18.3% 14.7% 11.1% 11.3% 12.0%
Per-Woman Success Rate 17.8% 20.8% 21.5% 30.9% 21.3% 14.4% 10.8% 11.1% 9.5%

PerPI_Success_2014_3

Based on the tight coupling of these measures, we continue with our analysis of per-person success using both PIs and CoPIs.

3-Year Window Per-Person Success Rate

In comparison to the annual success rate data, many of the details noted above are paved over by the 3-year window method. We’re not disparaging this method; it is quite useful, especially during steadier budget times. Because the typical grant lasts 3 years, the 3-year success rate window roughly measures the percent of the active PI population that could be continuously funded under current expectations for the size and duration of grants. However, in the case where we have multiple shocks to the system occurring over the reporting period, it can generate misperception.

Fiscal Window 2006-2008 2007-2009 2008-2010 2009-2011 2010-2012 2011-2013 2012-2014
DEB Unique PIs and CoPIs Applied 4110 4303 5221 5563 6798 7354 7790
Unique PIs and CoPIs Awarded 1130 1361 1511 1475 1301 1162 1197
Per-Person Success Rate 27.5% 31.6% 28.9% 26.5% 19.1% 15.8% 15.4%
Per-Person Success Rate (PIs-Only) 28.2% 32.5% 29.7% 27.8% 20.7% 17.9% 17.5%
NSF-wide (From NSB Report) Per-Person Success Rate (PIs-Only, excludes preliminary proposals) 37% 40% 40% 38% 35% 35% TBD

In this table, the windows affected by the extra stimulus (ARRA) funding are in italics and the windows affected by the new applicants to the preliminary proposal system are in bold. The 2010-2012 window sits at the intersection of the stimulus-elevated award numbers and preliminary proposal-driven increase in applicants. What we see here is that the pre-stimulus (2006-2008) and post stimulus (2011-2013 and 2012-2014) awardee numbers are quite similar. However, the applicant numbers have grown substantially, reflecting both the influx of new PIs in response to the stimulus and to the preliminary proposal system. This increase in the number of unique PIs/co-PIs applying in a given 3 year window drives the lower per PI funding success rate.

Notably, DEB’s per-person success rate is continually lower than the NSF-wide number but does follow the same pattern across the ARRA funding windows. The exclusion of preliminary proposal PIs from the NSF-wide counts leads to the increasing disparity between DEB and NSF-wide success rates from 2010-2012 onward.

Award Distribution

We can also compare the annual and 3-year window measures to gain insight into another aspect of per-person success rate. A relevant concern we often hear is that “the same well-funded people just get funded over and over again”. If that were true, we would expect persons funded in year 1 of a window to be funded again in year 2 or year 3. So, the count of unique awardees in the 3-year window would be smaller than the sum of the annual counts of unique awardees (i.e., Dr. X is counted once in the three year window measure but twice, once in year 1 and once in year 3, by the annual measure). But, if grants were spread out (and thus, fewer PIs with overlapping/continuous funding), there would be many fewer repeat PIs so the sum of the annual counts would be much closer to the 3-year window count. In our case we have:

Fiscal Window 2006-2008 2007-2009 2008-2010 2009-2011 2010-2012 2011-2013 2012-2014
3-Year Count of Unique PI/CoPI Awardees 1130 1361 1511 1475 1301 1162 1197
Sum of Annual Unique PI/CoPI Awardee Counts 1225 1493 1645 1613 1417 1271 1293

 

What this tells us is that fewer than 10% of awarded PIs in any 3-year window are repeat awardees during that period (~1.5 – 3.1% of all PIs who apply during that period).

If we step back and consider the whole 9 year period, we still find that the majority of PIs are funded only once.

PerPI_Success_2014_4

Even if they were all 5-year grants, continuous funding of a lab from DEB research grants alone is extremely unlikely for the vast majority of PIs.

 

Concluding Thoughts

1) The number of people being supported on DEB research grants (~420 persons on new grants per year) hasn’t changed much over this period, except for the temporary shock of the economic stimulus.

2) The stimulus, and a 3-year method of smoothing, really messes with the general perception of funding rates. (We actually hadn’t really thought about that much except as the one-year outlier we usually label in our posts. This was eye-opening in that regard.)

3) Funding rates, both per-person and per-proposal, are being driven down by increases in the applicant/application pool: primarily growth in actual participant numbers but some intensification of per-person activity is also possible.

4) Of 11,789 unique PI/CoPI applicants, only 2,970 (25% of all applicants) received any funding over the 9-year period examined. Of those 2,970 to receive funding, only 772 received multiple awards (26% of awardees, 6% of all applicants) that could potentially maintain continuous “funding” over this period. Any person applying to DEB’s competitive research programs is unlikely to be funded, and much less likely to maintain continuous support for a lab from this single funding source.

5) Coming back to our original motivation for this post, per-person success rates for funding in DEB were consistently ~10 percentage points lower than the NSF-wide submission and funding data in years leading up to the preliminary proposal system. The exclusion of preliminary proposals from NSF-wide statistics has only deepened the apparent magnitude of this disparity in recent years and has even altered the trajectory of PI participant counts for the agency as a whole.

[i] The 2015 version of the report, with NSF-wide numbers through fiscal year 2014 should be arriving soon.

[ii] Why are preliminary proposals excluded?

The short answer is: the records don’t neatly match up.

The longer answer is: Beyond the major issue that the entire process from receipt of preliminary proposals through decisions on the related cohort of full proposals crosses fiscal years and so defies straight-forward binning, the path from individual preliminary proposal to award can be surprisingly non-linear. Our ability to accommodate these complexities comes at the expense of our ability to enforce strong rules to ensure continuity of the data you provide to us. Collaborative proposals are a prime example. In many cases not all PIs and CoPIs are actually listed on the cover page of the preliminary proposal. When a full collaborative proposal is invited it results in several different cover pages that each contain a different set of names. There’s no guaranteed 1-to-1 mapping of PIs across the entire process. Also, the basic ability to associate a full proposal with a preliminary proposal is tied to the “institution” which is the official owner of the proposals (not the PI). So if a PI changes institutions, or a collaborative reorganizes, or any number of other things that happen quite regularly comes to pass, the system doesn’t allow the full proposal to be linked to the actual preliminary proposal record. There are also people who receive an invite but then elect not to submit a full proposal for various reasons. On top of which you also have a number of CAREER-eligible PIs who (with or without an invite) will submit CAREER based on their preliminary proposal. The twists and turns are multitude and in the choice between flexibly accepting them and rigid data quality, we generally come down on the side of broad acceptance.

[iii] This is why we ask you to submit a personnel list by email and list all of the people on the 1st page of the preliminary proposal project description to ensure reviewers get the full info. Unfortunately, tying those names to FastLane records is not currently practical.

Program Announcement: 1st call for use of NEON data and samples


NSF BIO has announced a new, short-term opportunity for funding to explore the use of National Ecological Observatory Network (NEON) data and samples. “NEON is a continental scale research instrument consisting of geographically distributed infrastructure, networked via cybertechnology into an integrated research platform for regional to continental scale ecological research.

The Dear Colleague Letter* calls for proposals for Conferences or EAGER research projects to begin making use of the data streams, specimens, and samples provided by NEON.

Initial responses to this opportunity must be received by email May 8, 2015. From the letter:

Interested PIs must email a 3-page (maximum) summary of their research ideas and planned activities to NEONresearch@nsf.gov by close of business, Friday, May 8, 2015.

The summaries will be reviewed internally and those ideas that best meet the goals of the Dear Colleague Letter will be invited to submit full Conference or EAGER proposals through FastLane in June with awards expected to be made before September 30, 2015.

Please read the full Dear Colleague Letter and contact NEONresearch@nsf.gov for answers to questions and further guidance.

*What’s a Dear Colleague Letter? See here.

 

DEB Numbers: Analysis of Broader Impacts


A recent paper in Bioscience by a AAAS Fellow (Sean Watts), an Einstein Fellow (Melissa George), and an NSF Program Director (Doug Levey) explores how the Broader Impacts Criterion was applied and reported in DEB proposals between 2000 and 2010. A major conclusion is that activities aimed at recruiting and mentoring students from underrepresented groups are proposed more than twice as often as they are eventually reported by PIs; of all the types of broader impact activities, broadening participation is by far the toughest to achieve. This result and others are discussed in the context of a recent review of the Merit Review Criteria by the National Science Board and resulting revisions to the Grant Proposal Guide.

Have you thanked a taxonomist today?


A whole year has come and gone and once again we’ve arrived at Taxonomist Appreciation Day!

Thanks to our current Systematics and Biodiversity Science Program Officers:

Maureen, Simon, Joe, David, Kelly, and Judy.

And, as we said last year (but it bears repeating):

Thanks to all those who have previously and continue to serve as Program Officers, Experts, and Administrative staff helping to manage our programs supporting taxonomy, systematics, species discovery, phylogenetics, and collections.

Thanks to the legions of reviewers who have contributed their expertise in identifying the best proposals in taxonomy and systematics for funding since the earliest days of NSF and to the PIs who are not just describing new species but pioneering new ways to do the work and share it with the scale and efficiency suited to the challenges faced by global biodiversity.

 

Join the conversation on Twitter: @NSF_BIO and #loveyourtaxonomist

How’s your bracket doing? No, not that one, this one.

Enjoy the pun-ditry over at Buzz, Hoot, Roar.

And, for the slightly more serious: check out the AMNH OLogy pages on wasps for some fun sciart comics suitable for kids to adults (h/t to Carly via @NSF_BIO on Twitter), funded by an NSF GRFP fellowship and a DEB Systematics program award.

 

Guest Post: A Shifting Landscape for International Biology-related Research


Editor’s note: Today we’re bringing you a guest post on issues related to international biology research written for us by Elizabeth (Libby) Lyons, Regional Program Coordinator: Africa, Near East and South Asia (ANESA) in the NSF International Science and Engineering (ISE) office. Libby is also a former DEB Program Officer. We’d like to thank Libby for taking the time to discuss this topic with us and her willingness to share her expertise and experience here on DEBrief. The content is important and applicable beyond DEB so we hope you’ll take the time to share it with your colleagues.

 

Late in 2014 a new international agreement became official that could affect any research using biological material in or from other countries. We want to help NSF-funded PIs adapt to any resultant changes so that the benefits of NSF-funded scientific discovery, workforce development and education, international collaboration, biodiversity conservation and/or capacity-building can continue in the United States and in partner countries.

We start with two important points:

  • This agreement applies to almost all international projects involving non-human biological resources, even if you don’t plan to transport material back to the U.S. and even if you don’t expect any commercial use! If you work internationally with non-human material that contains DNA the process for securing research, collection and/or export permits for your project could be affected.
  • Sovereign nations own their biological resources and associated traditional knowledge, and therefore have the right to make laws about their use and protection. NSF and your institution require you to follow those laws and secure the required permits!

Background:

The agreement, known as the Nagoya Protocol on Access and Benefit Sharing of Genetic Resources[i] (NP hereafter), aims to help countries develop standard protocols and protections for access to and sharing of benefits derived from their biological materials. Alas, in the near term, the NP is likely to place international biological research, especially fieldwork, under greater political scrutiny and increase its complexity due to variation across countries in, for example, interpretation of NP language, balance between use and protection, and stage of development of relevant laws. We note however, that in some countries there will be no change to current protocols, in some parts of the world there is growing regional cooperation[ii], and in others the permitting process has been simplified due to recognition of the importance of fundamental research with no commercial objectives.

Possible ways to adapt to this new landscape:

  • learn about the issues around the NP. The Swiss Academy of Sciences has published several documents on Access and Benefit Sharing involving non-commercial academic research, which provide an overview of terms, general processes and effective practices. Scientific societies and collections/museum consortia may also be resources for relevant information.
  • learn about the permitting requirements for country(ies) where you work. Country information will eventually be centralized at the NP Clearinghouse, though that effort is just starting. It may help to reach out to other researchers working in the same country to share knowledge and approaches.
  • apply for permits for the broadest scope of project you think possible, so that if your project or a related student project moves in a slightly different direction you not need re-apply.
  • strengthen relationships with your foreign collaborators. NSF encourages such collaboration and these scientists will likely be knowledgeable of the requirements in their country; some countries may require in-country research partners be named on permit applications.
  • be able to articulate how your research can benefit the partner country. Such benefits will usually be non-monetary, but from the perspective of partner countries, they include critically important outcomes such as discovery of biodiversity, co-training of students, academic collaboration and networks, and conservation and sustainable use of biodiversity.
  • consider adjusting the timing of your project. Learn about the permitting process in advance and apply for permits ASAP. Consider delaying the start date of your NSF award and/or the hiring of critical personnel (e.g., post-docs) until all final permissions are in hand.

We recognize that these international requirements could be cumbersome, but we emphasize the importance of compliance. In the past, non-compliance has had serious consequences for PIs, projects, U.S. universities and even international relationships between the United States and other countries.

PIs with questions are encouraged to get in touch with us. Either Libby or any of the ISE regional contacts can help. If DEB is your usual programmatic home at NSF, Simon Malcomber serves as a local point of contact.

 

[i] The official name is the Nagoya Protocol on Access to Genetic Resources and the Fair and Equitable Sharing of Benefits Arising from their Utilization (ABS) to the Convention on Biological Diversity (website: http://www.cbd.int/abs/about/ ).

[ii] Access and Benefit-Sharing in Latin America and the Caribbean: A Science-Policy Dialogue for Academic Research. Diversitas, June 2014.

DEB Numbers: CAREER Redux


Last August we provided a short post looking at CAREER proposal submissions across BIO and what effects the preliminary proposal system may have had on those numbers. In response to a question on our FY 2014 Wrap-Up post, we’re taking this opportunity to share a little bit more about the CAREER program.

Please read the previous CAREER post if you need an intro to the program.

Overall CAREER Trends

As we discussed before, CAREER submissions in DEB increased markedly from FY 2007 to FY 2009, creating a new normal over the last 5 years; annual submissions are now roughly double what they were in the early 2000s.

At the same time, the annual number of CAREER awards in DEB has also increased. However, CAREER awards have always been a small part of the DEB portfolio— a single digits percentage of total awards for many years.

CAREER2fig1

While these award numbers seem small, CAREER awards have increased since 2008 relative to the total number of awards made to our core programs.

CAREER2fig2

This pattern results from a combination of an increase in absolute numbers of CAREER awards and declining total numbers of other core program awards.

The funding rate (Number Awarded/Number Reviewed) for DEB CAREER proposals has been fairly flat in the long term but with some wild swings concurrent with the recent high submission counts and budgetary changes. Over the 13 year period examined here, there were 914 CAREER proposals reviewed in DEB and we made 108 awards. This gives a 12% funding rate.

CAREER2fig3

Even though the overall funding rate has been on a downward trend for the DEB Core Programs (see FY 2014 Wrap-up), CAREER proposals have suffered less and have closed the gap with regular proposals that was evident before FY 2009 (the spike in 2009 was due to ARRA – the stimulus funding). In the long run, the CAREER proposal funding rate is subject to the same external drivers as the overall success rate, submission counts and appropriated budgets, which adds a great deal of uncertainty to any prognostication based on current trends alone.

Under the preliminary proposal system (FYs 2013 and 2014), CAREER proposals fare about as well as regular proposals going through the entire two-stage process (7-8% funded). However, they fare much worse than the invited full proposals with which they share our fall panels. This backs up a frequent comment we hear from our fall panelists that the CAREER proposals don’t seem to do very well. As a whole, we would expect CAREER proposals to fare worse on average than the pre-screened set of invited full proposals in the same panel because the pre-screening wasn’t applied to the CAREER projects. At the end of the entire process though, the system is identifying and prioritizing for funding high-quality CAREER projects at the same rate as regular projects. This suggests that, on the whole, the project ideas in the body of CAREER proposals fare no better and no worse than the ideas in regular proposals through the DEB review system. Neither the myth that CAREERs are “easier” for eligible faculty to obtain nor the opposing claim that they more prone to being trashed in review are supported by the funding trends.

Cluster Numbers

We can break down the numbers of CAREER proposals and awards by cluster. Below, we’ve put the proposals and awards on separate figures to allow more vertical detail and show the contribution of each cluster to the total counts of CAREER proposals and awards.

CAREER2fig4

Submission numbers for the CAREER solicitation tend to mirror the distribution of Core Program submissions in general: Population and Community Ecology (PCE) receiving a plurality, followed closely by Evolutionary Processes (EP), and then Ecosystem Science (ES) and Systematics and Biodiversity Science (SBS). Prior to 2007, ES represented a larger portion of CAREER submissions than at present but has since been eclipsed by growth in submissions to PCE and EP. Our interpretation of this is that because CAREER proposals are all single-investigator projects they exhibit less potential for growth in this community (ES researchers) that places a strong emphasis on collaborative projects.

CAREER2fig5

Award numbers also provide for some interesting discussions within the limits of such small numbers. The growth in DEB CAREER awards is primarily due to increases in the number of awards made in the EP and PCE clusters. While the PCE growth has been consistent with the submission growth (12% funded prior to 2009, 12% funded since 2009), the awards numbers in EP have grown more than the submissions (6% funded prior to 2009, 16% funded since 2009). There has been little to no change in ES CAREER award counts, matching their lack of submission growth, and the same can be said for SBS.

PI Numbers

Pivoting from the cluster view, we also wanted to present some numbers relating to who applies for and who receives CAREER awards.

Times (re)submitted, 2002-2014 PI Count Awarded PIs Success Rate
1 414 58 14%
2 156 32 21%
3 55 18 33%

 

The first thing is that the plurality (if not majority[i]) of CAREER awardees were funded on their first submission. But, a majority of applicants also try only once for a CAREER. The success rate for these 1-timers is ~14%. However, the PIs who come back and try again do better (21%). While fewer PIs succeed at each resubmission, the number attempting resubmissions drops off much more quickly meaning those PIs are funded at much higher rates. Someone on a 3rd attempt at a CAREER is more than twice as likely to be funded as someone on their 1st attempt. (On a side note: to the extent we can do so, we observe a similar pattern in regular proposals too. But, it’s much harder to accurately track “resubmissions” due to PIs juggling multiple proposals, and changes in personnel, titles, and programs; thus the magnitude of the effect for full proposals is uncertain.)

Also, successful CAREER PIs are not necessarily those who are the newest faculty.

CAREER2fig6

Given the requirements of the CAREER solicitation, it can be advantageous to already have a lab up and running and to have gotten into the groove of being a researcher/educator. Also, prior experience with writing a successful grant helps when you’re attempting to write a successful grant for the challenging criteria of the CAREER program.

Closing Thoughts

One of the important messages here that is tough to illustrate directly with charts and tables is that CAREER Awards have a different purpose than regular research grant awards, and this distinction sometimes eludes PIs and reviewers. CAREER projects emphasize the integration of research and educational activities; this requirement is highly selective for a subset of PIs who are passionate about this professional nexus. In fact, these requirements can be so organizationally difficult to execute that we often advise the newest faculty against applying by this route when they are just getting their first lab together. A major failing of many declined CAREER proposals is inadequate integration of research and educational activities: it might have been a competitive regular proposal but because the PI selected a higher bar, they didn’t pass it.

As hard as it is to conceive of and present a strong case for a CAREER proposal, it can also be difficult for reviewers to rate these proposals when they themselves are not experts on the educational integration side of things. That is a good argument for allowing CAREER proposals to come in directly as full proposals because they can benefit from selection of ad hoc reviewers with strengths in those aspects otherwise on the periphery of expectations for a regular proposal. And, journeying into the realm of speculation, one might wonder if the increase in submissions actually contributes to an increased funding rate because reviewers then have a larger pool on which to base their expectations for these projects. Perhaps we hit some critical mass of understanding by panelists that helps overcome misperceptions. Either way, supportive panel reviews are an important part of justifying CAREER award decisions and the charts above should make it clear that we fully expect CAREER awardees to be able to pass that (elevated) bar.

While they are a valuable facet of our award portfolio that we support and try to give every consideration, we don’t have quotas and don’t provide a free pass to funding for CAREER proposals. Any decision to apply for a CAREER should be based on a weighing of your own variables: personal preferences, professional preparation, research interests, practicability, but not simply because someone said it would be “good to do”. To put it another way: CAREER proposals aren’t for everyone and you only get three attempts at it. If this is something you both want to do and feel you can do, that’s great. But, bowing to pressure to submit a CAREER project when it’s not a good fit for you or before you’re ready can distract from and delay pursuit of other potentially successful proposals.

Now that we’ve left you pondering whether it makes sense to submit at all, we want to try to close with something more positive for potential CAREER applicants to consider. Think about submitting a preliminary proposal before you submit a CAREER proposal. This is essentially a “no risk” proposition. Regardless of outcome, you will get some feedback on your ideas. This feedback may or may not be directly applicable to a CAREER proposal given the differences but you’ve gained useful experience. Even if you get a preliminary proposal rejection, you wind up with additional data that you can consider to make an informed (not reflexive) decision as to whether you can improve the work and present a strong case for an integrated educational component. And, you might just get an invitation for a full proposal. In that case you have not only gained feedback but a choice as to whether you want to pursue a full proposal or attempt to turn that idea around and transform the project into a full-fledged CAREER.

[i] Caveat: because we didn’t track individuals back before 2002, those who appear once during the early part of the data could be on a 2nd or 3rd submission.

New SEES Dear Colleague Letter: Food, Energy, Water UPDATED


2/12/15 UPDATE:

NSF is hosting a webinar for the FEW Dear Colleague Letter opportunity on February 25, 2015. This will include a live Q&A with Program Officers. Participants must register for the webinar no later than February 24, 2015. Please follow this link to the NSF even website for full details and registration: http://www.nsf.gov/events/event_summ.jsp?cntn_id=134146&WT.mc_id=USNSF_13&WT.mc_ev=click

 

Dear Colleague Letter: SEES: Interactions of Food Systems with Water and Energy Systems

Please check out the above link to learn about the new funding opportunity. This Dear Colleague Letter provides for Workshops and Supplements (and EAGERs for any of you applying to Math and Physical Sciences (MPS)) addressing the intersection of food, energy, and water systems.

Information about other ongoing and past SEES investments can be found on the SEES homepage: http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504707, and additional information about both SEES and the Food, Energy, Water nexus is available in the President’s FY2016 Budget Request to Congress, details here and here.