A Report on the 2012 NSERC Discovery Grants Results for Toronto Math

Fifteen¹ faculty members from the Department of Mathematics at the University of Toronto submitted proposals to the 2012 NSERC Discovery Grants competition. Of these, one was a first time applicant (En), two (Ga, Ia) applied after a successful appeal of 2011 results, and one (Cd) was an appellant whose appeal was denied but could reapply because the 2011 award was for zero dollars. The first table below shows the 2012 results (in thousands of dollars per year) with 2010, 2011 award amounts for those researchers. The second table shows similar data for Toronto mathematicians in the 2011 competition, including the amounts for researchers (1d, 2d, 3d, 4d) whose appeals of 2011 results were rejected. The average for 2012 awards to Toronto mathematicians was 153% the average for 2011.

Average Grant Amount (Toronto Math)

• 2006: \$27k/y
• 2007: \$26.3k/y
• 2008: \$26.5k/y
• 2009: \$25k/y
• 2010: \$25.3k/y
• 2011: \$19.3k/y
• 2012: \$29.5k/y

Instability Visualized

Viewing the results from the perspective of researchers in these competitions reveals instability in the Discovery Grants evaluation and appeals processes:

• Imagine the experience of researcher 2d. This person had five years at \$42k/y, was cut to \$30k/y in 2009, and successfully appealed that outcome. The result of the appeal was a one year reinstatement of the previous grant level at \$42k/y and permission to reapply to the 2011 competition. In 2011, this researcher’s grant was hacked to \$18k/y so this person files another appeal. The 2011 appeal is rejected.
• Contrast the experience of professors 4d and Ga. Both launched their Canadian research careers and entered the Discovery Grants competition for the first time in 2011. Ga’s appeal of the \$13k/y result from 2011 was successful and the 2012 competition led to a new result of \$30k/y for the next five years. 4d’s 2011 appeal was denied so this researcher is locked in for five years at \$11k/y. Which of these researchers is likely to have better HQP numbers at renewal time five years from now?
• The experience of 2011 appellant Ia is also a bit strange. After a long run of celebrated research funded at the \$40k/y level, this researcher’s funding level was dropped in 2011 to \$15k/y. That outcome was successfully appealed and the 2012 outcome was \$35k/y.
• The 2011 appeals by 1d, 2d, 3d, and 4d were all turned down so these researchers are locked in at relatively low funding levels for the next five years.
• NSERC deviated from standard policy (a “pilot program”) in their handling of the 2011 appeals of Toronto mathematicians. Toronto’s appeals were evaluated by multiple appeals advisers while appeals from other universities were evaluated by one. There is evidence² showing that Toronto appeals were denied even when one of the appeals advisers advocated for granting the appeal.

Consistent Results on Appeals Cases show 2011 was Anomalous

Proposals by three Toronto researchers were evaluated in both the 2011 and 2012 competitions. The outcomes for these proposals provide a comparison³ between the accuracy of the merit evaluations by the 2011 and 2012 Evaluation Groups and bin-to-funding assignment by the Executive Committee and NSERC staff. Here is the data, including the percentage adjustment from 2011 to 2012:

These three cases provide further evidence, consistent with the message in the public statement signed by over 300 Canadian researchers, that the 2011 evaluations were anomalous. Despite the consensus opinion from the Canadian math/stats community, a public message from a majority of the 2011 Evaluation Group, and advice from top administrators that the 2011 anomalies required an altered appeals process, NSERC chose not to reevaluate the scientific merit of proposals when considering whether to grant or deny an appeal. The grounds for a successful appeal required evidence of administrative errors; evidence of error in merit evaluation was not considered germane.

The Math-NSERC Liaison Committee is collecting data from department chairs about the 2012 competition for Section 1508. It might turn out that 2012 will be viewed as more consistent with expectations, a more accurate evaluation compared to 2011. This would be encouraging. However, the unsuccessful 2011 Toronto “pilot program” appellants (researchers 1d, 2d, 3d, 4d) face the next five years with inadequate funding to support their research programs.