Carole Lee Partners with Other UW Researchers to Examine Racial Disparities in Peer Review Scoring of NIH Grants

Receiving a National Institutes of Health (NIH) R01 grant is a key indicator of success for countless faculty. Not only do such grants provide research funding, they also signal a faculty member’s research independence and prestige, which is critical for tenure and promotion. So, when a 2011 study found that Black applicants for NIH R01 grants were significantly less likely to receive funding than white applicants with similar qualifications, the agency launched a contest for ideas to detect potential bias in its review process. Professors Carole Lee(Associate Professor, Department of Philosophy) and Elena Erosheva (Professor, Center for Statistics and the Social Sciences, Department of Statistics, and School of Social Work) won first prize for Most Creative Idea for New Methods to Detect Bias in Peer Review. This contest idea eventually led to a contract with NIH to conduct their proposed study.

Lee and Erosheva collaborated with Sheridan Grant, a PhD student from UW’s Department of Statistics, as well as Mei-Ching Chen, Mark Lindner, and Richard Nakamura (now retired) from the Center for Scientific Review at NIH. The study focused on the earliest peer review scores assigned to grant applications. Reviewers provide preliminary scores for significance, innovation, approach, investigator(s), and the investigator’s research environment. These scores inform each reviewer’s overall impact score, a measure for a proposal’s perceived ability to exert a sustained scientific influence. Reviewers’ overall impact scores determine which proposals move forward for further consideration in panel discussions – a decision point responsible for the largest source of racial disparity in the evaluation of NIH R01 grants.

Their study shows that Black applicants receive worse preliminary scores for all criteria, even after controlling for many application and applicant characteristics; and, they find that these differences in criterion scores completely explain racial disparities in overall impact scores. Their study also finds that differences in how reviewers weigh the importance of different criteria do not explain racial disparities in overall impact scores.

“NIH introduced scores for individual criteria as a way to increase transparency for applicants,” says Lee. “And, it is these same scores that fully account for the racial disparities we see in the preliminary overall impact scores that determine which proposals move on to panel discussion.”

“The Chronicle of Higher Education featured the paper and interviewed Erica Warner, an assistant professor at the Harvard Medical School whose research focuses on studying risk factors for breast cancer. As she said there, “[t]he approach that the NIH has taken in the enhanced peer-review system is to say, If we give reviewers clear, objective criteria, then the decisions they make will be solely merit-based. It doesn’t appear that that’s enough.”

Why do Black applicants receive lower scores for these individual criteria?  Their study identifies this as the next question that future research should address, to see whether disparities along individual criteria are driven by implicit bias or by differences in mentorship, productivity, topic choice, and/or other experiences impacting the receiving of grants.