Peering into the nuts and bolts of the peer review process

Last year, the MS Society dedicated over $6 million towards funding nearly 70 research projects related to MS from across Canada, encompassing the gamut of funding opportunities from operating grants to research studentships. You might be surprised to hear that not every submitted application is approved for funding. As a largely public-funded organization, the MS Society is committed to ensuring that only research of the highest quality and scientific merit is funded to make sure that donor dollars are spent wisely, but who makes that decision, and on what basis?

The cornerstone of the MS Society’s decision-making approach to awarding funding for research is the peer review process, which happens to be the international gold standard for ensuring the highest standards of quality and excellence are met in scientific research. During the peer review process, each application is rigorously scrutinized by experts in the field, who identify the proposal’s strengths and weaknesses, and determine whether the research will advance knowledge in the field and/or improve health and quality of life. Moreover, a properly-conducted peer review must uphold core principles of ethical integrity, and is designed to ensure that the evaluation process is fair, transparent and free of bias.

Before we delve into the nuts and bolts of the review process here at the MS Society, let’s briefly go over some history behind the peer review and how it came to emerge as the foundation of the scientific evaluation system, both for the awarding of grants and the selection of journal papers.

A brief history of peer review

Peer review in one form or another has existed for a long time; although a firm consensus on the origin of modern peer review is a matter of debate, many scholars agree that it can be traced back to the journal Medical Essays and Observations published by the Royal Society of Edinburgh in 1731. A similar system was adopted by the Royal Society of London, which in 1752 established a “Committee on Papers” for their journal Philosophical Transactions in order to assist the editors in selecting manuscripts for publication and giving them a stamp of authenticity. At that time, however, any guarantee of the validity of the findings presented in a submitted paper were largely based on the honour system, and it would be a long time before the peer review process we recognize today began to take shape.

Although peer review is undoubtedly a staple of the modern screening process for scientific articles, most people would be surprised to find out that it was only relatively recently that peer review became adopted by the majority of scientific journals. Many prominent journals, such as Nature and the Lancet, didn’t implement mandatory peer review until the 1960s and 70s. Nobody is certain what the driving force was behind the sudden shift towards institutionalized peer review in the second half of the 20th century, but many experts believe that it was a combination of an explosion in the number of research articles being submitted to journals and the increasing diversity and specialization of scientific disciplines that made it impractical for a journal’s editorial board to thoroughly scrutinize each and every article.

At around the same time, funding agencies that awarded research grants also began to adopt their own peer review processes in the face of mounting public pressure that demanded accountability in the spending of public funds. Agencies like the National Cancer Institute and forerunner to the National Institutes of Health established advisory committees to review grant applications in the early 20th century, and laws mandating peer review by public funding agencies were passed in the United States in the 1930s and 40s. Private sector granting organizations promptly followed suit, and today, most non-profit foundations that fund health research employ their own peer-reviewed competitions.

Review process for grants and awards at the MS Society

The MS Society uses a rigorous and extensive review process to identify the most promising research projects to be awarded funding based on the originality, feasibility, validity, and significance of each proposal. Essentially, experts from various research fields related to MS, such as immunology, imaging, genetics, physiology, etc., look at each application and determine if the project can be done, with the right people and tools, and will lead to breakthroughs and important discoveries in MS.

The following are important steps that are taken during the review process to determine which projects get funded, and which ones don’t.

Step 1: Building the review committees

When a research competition is launched, the MS Society invites expert researchers and clinicians in the field of MS to form a review committee based on their areas of expertise. For example, the Annual Research Competition, which is the MS Society’s core funding competition held every year, involved three review committees. They are:

  1. Biomedical Research Review Committee: this committee reviews operating grant applications related to basic science, which is focused on enhancing our body of knowledge about how MS develops and progresses. Basic research done in the laboratory can involve studying how the disease affects cellular and molecular processes, as well as how the disease develops in an organism and what aspects of the environment and genetics contribute to development of the disease.
  2. Clinical and Population Health Research Review Committee: this committee oversees operating grant applications that study MS in the primary care setting and examine the patterns, causes and effects of MS in the population.
  3. Personnel Awards Review Committee: this committee reviews applications for studentship awards – both master’s and doctoral – as well as Postdoctoral fellowships in both biomedical and clinical/population health research. These awards are vital for attracting and retaining bright young minds to the field of MS and training the next generation of MS researchers.

Each committee is led by a chair, an established researcher in the MS scientific community who oversees the entire process and presides over the review meetings. Recently the MS Society added community representatives – lay members of the public – to each of the review panels, which led to the renaming of the peer review process to the independent research review process. It’s great to see that the involvement of community representatives in the review process is now taking place across many research funding organizations.

Step 2: Assigning the applications and conducting independent reviews

Once the committees are convened, the committee chairs, in collaboration with MS Society staff, assign each application to reviewers who have considerable expertise in that particular research topic. At this stage in the process, any potential conflicts of interest are also identified (in other words, reviewers who have published papers or worked closely with an applicant in the past are spotted) to prevent the possibility of bias during the review. Each application is evaluated by a primary and secondary reviewer, who independently read the application from top to bottom, write a detailed critique of the application and assign an overall score. Some questions that each reviewer must consider when evaluating an application include:

  • Is the experimental design logical and feasible?
  • Is the study novel and does it contribute new information to our existing body of knowledge?
  • Is the researcher equipped and qualified to answer the study questions? Is the budget realistic?
  • Does the study contribute something new to our understanding of MS?

Step 3: Convening the review committees and finalizing evaluations

Once the reviewers evaluate their applications in private, each review committee meets in person to engage in an open discussion about the applications and their evaluations. For each application, committee members discuss individual comments brought up by the primary and secondary reviewers and bring up any issues that may arise from the discussion. The chair then asks for a consensus score (a score that the two reviewers can agree upon based on the discussion), after which each committee member (except for the chair and community representatives) votes within 0.5 points of the consensus score. All of the scores are tallied, and those applications that are scored above the designated cut-off score go on to be recommended for funding. To guarantee an impartial review process, committee members who were identified as having a conflict of interest with a particular application must step out of the room during the review and scoring of that application.

Step 4: Presenting a recommendation for funding

Each committee’s recommendation for funding is presented to the MS Society’s Medical Advisory Committee (MAC), a team of senior researchers and clinicians who oversee scientific and medical matters that impact the MS Society and its stakeholders. In this case, the MAC provides guidance throughout the research review process and examines the list of recommended applications through the wider lens of general trends in MS research. Once the MAC reviews the recommendations, it presents a final recommendation for funding to the MS Society’s National Board of Directors.

Community representatives: giving a louder voice to those affected by MS

A crucial part of the review process is engaging those people who are most affected by progress in MS research – namely, people living with MS, their families and caregivers – and giving them a voice. The MS Society is dedicated to enhancing participation by members of the MS community towards determining the future of MS research in Canada. Each of the review committees consists of anywhere between one and three (depending on the size of the committee) community representatives, whose participation ensures accountability of the review process, and that the research applications are relevant to MS. Community representatives also comment on the ease of comprehension of the lay summaries of the projects, and provide feedback on ways that researchers can communicate their work in an understandable way for the public, which is equally as important as their ability to communicate their work scientifically.

Overall, community representatives ensure that donor dollars are spent wisely on research projects that will have a significant impact on people affected by MS.

What are your thoughts on the research review process? Do you have previous experience reviewing research applications? I would love to hear your thoughts and questions, so go ahead and leave a comment below.

References

Guston D. (2003). The expanding role of peer review processes in the United States. In Learning from science and technology policy evaluation: Experiences from the United States and Europe. Eds. Shapira and Kuhlmann. Cheltenham, UK and Northampton, MA: Edward Elgar. 81-97.

Rennie D. (1999). Editorial Peer Review: Its Development and Rationale. In Peer Review in Health Sciences. Eds. Goodlee and Jefferson. London: BMJ. 1-13.

Shamoo A and Resnik D. (2003). Responsible conduct in research. New York: Oxford University Press. 68-72.

Leave a Reply

Your email address will not be published. Required fields are marked *