SINGLE REVIEWER ASSESSMENTS OF CERTAINTY OF EVIDENCE IN NETWORK META-ANALYSES PROBABLY DO NOT MISLEAD DECISION-MAKERS: A CASE STUDY.
TL;DR
Novice reviewers disagreed with the consensus more often than expert reviewers when assessing the CoE from NMA, however, having a single reviewer was associated with relatively few disagreements likely to have meaningful implications for decision-makers.
Abstract
Background
Systematic reviews with network meta-analyses (NMA) are the most comprehensive form of evidence synthesis for informing evidence-based practice recommendations. Rating the certainty of evidence (CoE) is a crucial step for drawing appropriate NMA's conclusions, but it can be time-consuming.
Objectives
To evaluate the disagreements of assessments by a single reviewer (novice or expert) versus consensus among two reviewers on CoE ratings of NMA used to support practice recommendations.
Methods
We conducted a methodological case study using the GRADE assessments for each comparison and outcome of interest from the COVID-19 Living NMA, which informed WHO recommendations. Expert and novice reviewers assessed CoE independently and resolved disagreements by consensus. We defined an "important disagreement" in CoE ratings as any disagreement that would lead to an increase from high/moderate to low/very low, or vice versa. We calculated the proportion of outcomes with important disagreements comparing the CoE rating from a single versus two reviewers. We performed a generalized linear mixed model analysis to compare the relative odds of disagreement between expert and novice reviewers.
Results
Six experts and five novice reviewers rated 6,652 comparisons. Compared with consensus ratings, novice reviewers exhibited higher disagreement rates across all GRADE domains, including assessments of direct, indirect, and network evidence. The disagreement rates for the overall CoE rating assessed by novice reviewers were higher than those of experts (10.6 vs 2.3%), but the important disagreement rate was similar (1.2% vs 1.0%). Novice reviewers were more likely to make important disagreements, but the association was not statistically significant (Odds Ratio, 1.3; 95% CI 0.9-1.8).
Conclusions
Novice reviewers disagreed with the consensus more often than expert reviewers when assessing the CoE from NMA. However, having a single reviewer was associated with relatively few disagreements likely to have meaningful implications for decision-makers.