Evaluating the quality of systematic reviews and meta-analyses published in behaviour analysis journals : An umbrella review

Abstract

High-quality systematic reviews and meta-analyses are essential for translating evidence into practice. We conducted an umbrella review to evaluate the methodological quality of systematic reviews and meta-analyses published in the field of behaviour analysis up to and including 2023. Eligible studies were identified through targeted searches of seven behaviour analysis journals using APA PsycINFO. Quality was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR 2) and Revised AMSTAR (R-AMSTAR) instruments. Temporal trends were analysed using Bayesian multilevel regression. The protocol was preregistered on the Open Science Framework (doi.org/10.17605/OSF.IO/U38Z4). We identified 64 reviews (16 of which included a meta-analysis), all of which were rated as ‘critically low’ quality using the AMSTAR 2 criteria. Methodological shortcomings included absent protocol registration, inadequate risk of bias assessment, and failure to assess publication bias. Mean R-AMSTAR adherence was 43.8% (range 11–66%) and increased by 1.08% per year (population-level average marginal effect from the multilevel logistic model (95% CI [0.59, 1.58]), indicating robust methodological improvement over time. Currently, many systematic reviews and meta-analyses in behaviour analysis do not yet meet contemporary standards of methodological rigour. Enhancing the quality, transparency, and consistency of evidence synthesis is vital if systematic reviews are to meaningfully inform practice.