You are here

Article Text

Article menu

Download PDFPDF

XML
Original research
Effect size reporting among prominent health journals: a case study of odds ratios
  1. Brian Chu1,
  2. Michael Liu2,
  3. Eric C Leas3,
  4. Benjamin M Althouse4,
  5. John W Ayers5
  1. 1 Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
  2. 2 University of Oxford, Oxford, UK
  3. 3 Department of Family Medicine and Public Health, Division of Health Policy, University of California San Diego, La Jolla, California, USA
  4. 4 Epidemiology, Institute for Disease Modeling, Bellevue, Washington, USA
  5. 5 Department of Medicine, Division of Infectious Diseases and Global Health, University of California San Diego, La Jolla, California, USA
  1. Correspondence to Brian Chu, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA 19104, USA; brianchu2010{at}gmail.com

Abstract

Background The accuracy of statistical reporting that informs medical and public health practice has generated extensive debate, but no studies have evaluated the frequency or accuracy of effect size (the magnitude of change in outcome as a function of change in predictor) reporting in prominent health journals.

Objective To evaluate effect size reporting practices in prominent health journals using the case study of ORs.

Design Articles published in the American Journal of Public Health (AJPH), Journal of the American Medical Association (JAMA), New England Journal of Medicine (NEJM) and PLOS One from 1 January 2010 through 31 December 2019 mentioning the term ‘odds ratio’ in all searchable fields were obtained using PubMed. One hundred randomly selected articles that reported original research using ORs were sampled per journal for in-depth analysis.

Main outcomes and measures We report prevalence of articles using ORs, reporting effect sizes from ORs (reporting the magnitude of change in outcome as a function of change in predictor) and reporting correct effect sizes.

Results The proportion of articles using ORs in the past decade declined in JAMA and AJPH, remained similar in NEJM and increased in PLOS One, with 6124 articles in total. Twenty-four per cent (95% CI 20% to 28%) of articles reported the at least one effect size arising from an OR. Among articles reporting any effect size, 57% (95% CI 47% to 67%) did so incorrectly. Taken together, 10% (95% CI 7% to 13%) of articles included a correct effect size interpretation of an OR. Articles that used ORs in AJPH more frequently reported the effect size (36%, 95% CI 27% to 45%), when compared with NEJM (26%, 95% CI 17.5% to 34.7%), PLOS One (22%, 95% CI 13.9% to 30.2%) and JAMA (10%, 95% CI 3.9% to 16.0%), but the probability of a correct interpretation did not statistically differ between the four journals (χ2=0.56, p=0.90).

Conclusions Articles that used ORs in prominent journals frequently omitted presenting the effect size of their predictor variables. When reported, the presented effect size was usually incorrect. When used, ORs should be paired with accurate effect size interpretations. New editorial and research reporting standards to improve effect size reporting and its accuracy should be considered.

  • clinical decision-making
  • evidence-based practice
  • general practice
  • methods

Data availability statement

The data used in the study are public in nature. The strategy to replicate our database is available in the text and the data are available on reasonable request from the authors.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjebm-2020-111569

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Data availability statement

    The data used in the study are public in nature. The strategy to replicate our database is available in the text and the data are available on reasonable request from the authors.

    View Full Text

    Footnotes

    • Contributors BC, ML and JWA initiated the project and led the design. BC led the data collection, and all authors participated in the data analysis. All authors participated in the drafting of the manuscript, read and agreed to the final submission. BC, ML and JWA initiated the project and led the design. BC led the data collection, and all authors participated in the data analysis. All authors participated in the drafting of the manuscript, read and agreed to the final submission.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.