Context

Grant peer review takes different forms. Perhaps the most common are (1) an expert committee that reviews all grant applications and rates or ranks their quality and (2) each application being sent to a small review panel (one or two reviewers) who may provide a final score or contribute to a larger expert panel’s discussion and rating or ranking. Some funders use a randomised component once certain criteria are met.21 22 The peer review committees that contributed to CIHR’s Project Grant Competition peer review operated as expert committees that reviewed all applications and rated or ranked them.

The CIHR Project Grant Competition awards approximately $C650 million of CIHR’s $C1.3 billion annual funding budget. Researchers at any career stage, who wish to conduct health-related research, are eligible to apply. For each Competition, approximately 60 Peer Review Committees adjudicate about 2000 grant applications across the breadth of the CIHR mandate which spans (1) biomedical, (2) clinical, (3) health systems and services, and (4) population health research themes (now pillars). The committees meet online in spring and autumn each year to evaluate and rate each application they are assigned.23 Typically, these meetings occurred in person, until the pandemic necessitated that they occur virtually. Until the fall 2020 Peer Review Committee meetings, CIHR peer review was conducted in-person. Since then, all peer review has been conducted virtually.

For the CIHR Project Grant Competition, each Peer Review Committee comprises up to 20 members (peer reviewers) plus three leaders—Chair and two Scientific Officers—who, with support from CIHR staff, assign applications to reviewers, lead the committee consensus discussion and summarise the committee discussion in written feedback for applicants. Members are recruited from the CIHR College of Reviewers, nominated by Chairs and/or Scientific Officers, or identified by Internet search (including Canada Research Chairholders list, Fellows Directory for the Canadian Academy of Health Sciences, publications, conference invited speakers, institutions in regions that are historically under-represented on Committees). When adjudicating each application, the Peer Review Committees are asked to consider (1) the significance and impact of the research, (2) the approaches and methods, and (3) expertise, experience and resources available to deliver on the research project objectives.

Peer review occurs in two stages. First, all submitted applications are initially reviewed and scored (rated) by a primary reviewer and two secondary reviewers, who provide a rating (on a 0–4.9 point rating scale) and written feedback. The second stage of the review process occurs at the Peer Review Committee meeting. Because only about 20% of the applications to the Project Grant Competition are ultimately funded, a streamlining process is first used to eliminate non-competitive applications so that the Committee has the maximum time available to discuss competitive applications. An application is streamlined (ie, receives three ratings and written feedback but is not discussed by the Peer Review Committee or considered for funding) if (1) the average of the reviewers’ ratings places the application in the bottom 60% of all applications that the Committee is considering, (2) at least one reviewer has identified the application as non-competitive, and (3) no Committee member objects to streamlining the application.

For applications that are discussed at the Committee meeting, the three reviewers are asked to reach a consensus rating (usually approximately the mean of the reviewers’ ratings) after the Committee discussion. Once the consensus rating is announced to the Committee, all Committee members are asked to rate the application (final rating) within ±0.5 of the consensus rating. Ultimately, applicants whose applications are discussed receive (1) the final rating (collated by CIHR staff after the Committee meeting), (2) written feedback from the Scientific Officer capturing the key elements that the Peer Review Committee considered during their discussion, and (3) the written feedback and ratings from the reviewers.

There are no interviews with applicants and no opportunity for applicants to rebut the Peer Review Committee’s feedback during the peer review/grant selection process. Applicants may submit a two-page Response to Previous Reviews if they choose to resubmit their application to a subsequent Project Grant Competition round.

Supplemental material

Data collection

Guided by a generic approach to qualitative research,25 the interview guide was developed based on a priori concepts of peer review and the study team’s experience with grant review. The interview guide included questions about participants’ background, training in grant peer review, strengths and challenges of the review process (including experiences of in-person and virtual peer review), conflict, bias, equity, diversity and inclusion. The interview guide can be found in online supplemental appendix 2. JSG and CLA conducted semistructured interviews with 18 participants via Zoom. Interviews lasted 30–65 min. The number of participants in this study is consistent with best practices for qualitative research.26

Processing and analysis

In accordance with our sample (eg, those who had participated on a grant peer review panel) and a priori topics, we used framework analysis to achieve our objectives. Participants’ original accounts anchored and guided our descriptions and observations.27 28 For analysis, we sifted, charted and sorted data based on key issues and themes using five steps. First, using Zoom, each interview was transcribed verbatim. One team member read the transcripts to obtain a sense of the interviews (Step 1. Familiarise). Then, we combined inductive and deductive approaches to develop a thematic framework. To guide our initial framework, we first identified themes of significance from the literature. To refine our framework, we incorporated topics that we recognised as frequently occurring in our data (Step 2. Identify a thematic framework). We then coded all transcripts using the thematic framework established in Step 2. We used the software Nvivo V.14 to manage the transcripts and analyse data (Steps 3 and 4. Index and chart). To compare and contrast themes within and across groups, we adopted the constant comparison method; we explored similarities and differences across the data (Step 5. Map and interpret).27

Trustworthiness

Four strategies reinforced the rigour of our study. We cross-checked full transcripts against original audio files for quality and completeness. JSG recorded reflexive memos during data generation and analysis. JSG and CLA met after the interviews to discuss emerging themes. Using NVivo, JSG applied our thematic framework to code full paragraphs of the interviews so that we did not lose contextual meaning. As a team, we discussed themes and those cases that did not ‘fit within themes’. Where there were disagreements (there were very few), we reviewed and discussed the original transcripts, to reach consensus on the theme. We replaced participants’ names with pseudonyms to report results.

Table 1

Study themes, descriptions and illustrative quotes

Lack of training and limited opportunities to learn create challenges when assessing grant applications

In response to questions (eg, “What training did you receive for your role as a reviewer/Chair/Scientific Officer”), participants drew on their own experiences as both a grant reviewer and as a grant applicant. They spoke about the lack of formal training for grant peer review; at best it might be considered a ‘learn as you go’ model. Participants drew on their own review philosophy and experiences as an applicant.

I learned from, you know, some of some of my mentors and when I watched them as chairs and those who brought me into the system and then kind of learned from them.

Participants emphasised the lack of in-person training or systematic feedback for grant peer review, but did mention CIHR’s written guides for reviewers, which were provided by CIHR to reviewers as weblinks to text material.

I did not receive training for any of those roles. Zero training.

I mean I was given all the documents you know… the guides to review and so on.

For those who mentioned the availability of resource material, there was no mention of how they used the materials or how useful the materials were, and the lack of training was still emphasised. Participants mentioned that the volunteer role of grant peer reviewer added pressure to their already full list of academic and life commitments. Participants found it challenging to balance their desire to train well to do the peer review role with all their other commitments.

Challenges in differentiating and rating applications of similar strength

Participants indicated that they were challenged to differentiate between grants of a similar strength—the group of grants that take the majority of the Peer Review Committee’s work time, which we have termed ‘the meritorious middle’ (differentiated from the bottom group of applications that are considered ‘un-fundable’ and the top group of applications that are considered exceptional). Participants discussed how, without a scientific ‘fatal flaw’ and lack of clarity on how to distinguish one fundable (deserving) grant from another, the decision on a grant’s score might be influenced by how interesting the topic was to the reviewers.

You can have a lot of grants where there’s nothing flawed and there’s a solidly proposed piece of work. You know there’s nothing wrong with [the] methods—there’s nothing that you could pick apart in terms of the theory or the research question. But there’s just another grant in the competition that is scored marginally higher because it catches the eye and the interest of the review committee, and it’s that intangible kind of interest piece.

‘Catching the eye and interest of the review committee’ are not best practices described in review guidelines, nor are they a reproducible, equitable or inclusive practice. Similarly, review decisions might be made based on the topic of the research and not the merits of the (very good) application:

…it’s not always dependent on how good you are as a scientist, it’s very much dependent on how fashionable your topic is.

While participants described decision-making based on ‘interest’ and ‘fashion’, they did not explicitly state how the approach threatened the review process. Rather, participants focused on the lack of clarity and challenges associated in the review of mostly high-quality grant applications. Participants described a review process that was apt for rating or ranking the outstanding applications and the weak applications (those considered as not fundable). Peer Review Committee members felt their challenging work was in how to reliably review and score the substantial proportion of grant applications that were considered ‘fundable’ (ie, ‘the meritorious middle’):

… at that point, you may as well throw them down the stairs.

In addition to a sense of frustration, there was also a distinct sense of defeat. Participants felt that there was no clear way to distinguish between the fundable applications. In an exasperated tone, one participant shrugged and stated:

That is really hard to grapple with in a peer review process……I honestly don’t think that the review Committee does a better job than a lottery.

Participants discussed rating and ranking at length, in the context of challenges with the current rating system. Some suggested that the full range of scores is not used when Peer Review Committee members are rating applications. One participant described the problem as ‘the mushy middle’.

In the mushy middle [is the problem]. The exceptional ones, usually, you know, come through.

But ones that are deeply, deeply flawed that really don’t need just an edit or bit of a fix, but actually need to go back to the drawing board—we rarely give those really low rankings or really low scores, right? And so, the one thing that, you know, I tend to push for—encourage—is to make sure that the verbal description of the score that you are giving actually reflects your opinion…we need to work with the full range of scores, so that we can better differentiate the few that are going to be funded.

Participants shared the sentiment that if a grant is not going to be funded, the consensus score (the score the committee decides at the meeting) and the comments must reflect that fact. The words ‘clarity’ and ‘clear message’ were used frequently throughout the interviews when speaking about not fundable grants. One participant exclaimed:

I despise the “this is 3.5”, and “that is 3.6” and then 3.7 …it’s creeping in that middle range… we need to send a clear message here if this grant going to be funded, if no, then … it needs to be reflected in the score.

Calibration (ie, members of the Committee reaching common ground and tuning (by consensus discussion) their individual interpretations of the application rating system to promote consistency and fairness in how the Committee rated each grant application.29 For example, the Committee might discuss and agree on what would constitute a rating of 3.5 as opposed to a rating of 4.1. Individual reviewer scores are not recalculated as z-scores to compensate for systematic differences between reviewers in CIHR’s Project Grant Competition) was raised as a strategy to provide clarity in rating grants. The responsibility for calibration landed solely on the Peer Review Committee Chairs.

I think the Chairs need to quickly establish this is a [outstanding] grant where you've got three reviewers who are like, you know, this is a 4.5, 4.6, 4.7 this is where the bar is set, this is where people are agreeing and then maybe identify one grant that everyone agrees wasn’t a good grant. And then work your way towards the middle…it’s sort of you establishing a floor and a ceiling and I always think that that’s a way to calibrate people …I began to get a better appreciation [through the review process] that most people still are very uncomfortable with the full-scale concept. And I get it, right? Nobody likes to give anybody a bad score.

Ranking instead of rating was also suggested as a strategy to improve the review process.

An emphasis on reputations and relationships in the review process to resolve decision dilemmas

The role that personal relationships played within the grant peer review process also reflected a serious threat to grant peer review. Although there was training on bias in the review process, participants noted the absence of strict and clear guidelines for review. As a consequence, unconscious (and sometimes conscious) bias crept into the process. Established researchers (famous by name) could ‘receive the benefit of the doubt’ in the review process:

You hope that it’s [grant review] based on merit, not who you are. But I have seen a degree of fascination with established career researchers who, in my opinion have not written the best grant proposal, get the benefit of the doubt—let’s just call it that.

Similarly, another participant described this as ‘old school, new school stuff’ and suggested that the reputation of the applicant was prominent in the review process. Another participant reflected on the role that an applicant’s Curriculum Vitae (CV) can have on the process in influencing decision-making and the unfair (inequitable) advantage afforded to some applicants:

I still see this happening, particularly with more senior career investigators, they get all excited about a CV that has 150 papers on and I'm like: “the research proposal doesn't make any sense” …but they have 150 papers, so that must be good, right? … that is a distinct conscious bias [and] it’s persistent now.

[it’s] kind of a human nature that we are all biased in some form or shape …and I think we do take that into consideration when it’s core [to someone’s work], because so-and-so is so well known in the field, or has been running this lab for [years]…But the methods aren’t very good, you know, so people will say oh we’re going to give them the benefit of the doubt so again, I think [the Chair is essential].

In addition to attributes of the applicant influencing the review process, the use of social moments and ‘networking’ among reviewers during in-person reviews may also serve as a threat to grant peer review because they preference those who are in the room. In discussion of in-person reviews, many participants noted that relationship building, during social times, was an important reward for people who volunteered their time to participate in the peer review process:

It’s the side conversations sometimes away from the grant review that are enriching and rewarding as part of the process.

Others noted the indirect benefits of participating in the in-person reviews as the informal networking that occurred:

the honest truth is that the in-person experience was really as much around getting together with your colleagues, which is always enjoyable, in my opinion.

While some enjoyed the indirect benefits of in-person reviews, others questioned the need for in-person review.

Although I agree social connections are important, I’m not sure that our panel meetings should serve that purpose.

Role of the Chair in clarifying how to assess equally meritorious grants

All participants noted the key role played by the Peer Review Committee Chair in grant peer review. The Chair is a researcher who manages the applications, ensures qualified reviewers are assigned to all applications and chairs the consensus meetings. The Chair role was described as ‘essential’ and critical to grant review:

it really sort of helps if you have a really good Chair.

Participants noted that an effective Chair guided the conversation and provided much needed direction when disagreements occurred. One participant noted, “I remember that the Chair was very … elegant in in bringing us back …into a discussion.” The Chair role was described as that of a facilitator, a mediator and in some cases an arbitrator who makes a final decision. Participants acknowledged the ‘responsibility’ of the Chair to manage conflicts:

Sometimes discussions can get heated …, especially if you have a reviewer that really just doesn’t like something about the grant and they are going to stand firm, because they really don’t think it should be funded,… like managing that—I think that’s the responsibility of the Chair.

The role that a Chair plays in minimising bias and ensuring trustworthiness and rigour was also discussed. Participants noted that while everyone ‘has bias’ ultimately it is the responsibility of the Chair to identify and address bias to ensure a rigorous grant review process.

Participants also discussed the role of the Chair in managing more challenging applications, including resubmissions. Lack of clarity around the ‘mushy middle’ was expressed, so too, lack of clarity and consistency in how resubmissions were handled. One participant discussed their role in managing resubmissions as a Chair:

Most recently, I was Chair of one of the panels, and when resubmissions came up people gave them a regular review. But in their comments they might say “we saw this one before.”. And sometimes I’ve heard comments and I had to had to intercede: they would say “well we’ve seen this one for the third or fourth time we need to either fund it or not, or give them a very strong message, like this is just not gonna do it.”. So, sometimes the reviewer would be trying to push it over that funding line with no other reason than this is the fourth time we’ve seen this and I’m having to say as the Chair “that’s not the reason to fund the grant”.

Chairs helped to clarify the peer review process for reviewers. The Chair was critical in promoting reproducibility and rigour. Beyond scientific skills, participants agreed that Chairs needed excellent interpersonal skills:

Sometimes it’s [the review process] managed well … and [it requires] a lot of its interpersonal skills, more so than scientific skills and how meetings are chairs and how individuals are coached.

Limitations

Our study focused on grant peer review by one health agency in the Canadian context with 18 reviewers. Most of the reviewers had experience in reviewing with only CIHR. While we believe many of the findings are likely universal, these are limitations of the current study. Future research would benefit from the inclusion of other granting agencies in other countries. Future research would also benefit from interviews with reviewers with experience from other granting agencies.

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by University of British Columbia (UBC) Research and Ethics Board (H21-03875). Participants gave informed consent to participate in the study before taking part.