To make the most of medical research funding, approval committees should run the numbers on proposals. Problem is, they sometimes cannot be fagged, or do not know how to. Adrian Barnett and colleagues found out why statistics aren’t applied to evaluate many projects.
They did it by asking human research ethics committees about the statistical basis of studies they approve – quite a few studies – and presented the results in a new paper.
The issue: In 2022, just under 200 committees considered 13,000 research proposals and approved 90%. While qualitative research does not need statistical input, Barnett and colleagues warn, “the current high number of research studies receiving approval without statistical review risks approving studies that will in the best-case waste resources and in the worst-case cause harms due to flawed evidence.”
When asked about using stats to analyse the methodologies of proposed projects, responses included indifference, ambivalence and assumptions that the experience of committee members is enough – “the label ‘expert’ is flexible, relative and contextually dependent,” the authors assert. And one respondent believed in the credibility-creating power of journals, “that projects that disclose an intention to publish results in peer-reviewed journals could be safely assumed to be of sound study design.”
Overall, they found only a quarter of committees included a statistician and less than 40 % “had adequate statistical scrutiny.”
What makes this surprising, as well as alarming, is that research approvers appear unfussed by the 2023 update to the National Statement on Ethical Conduct in Human Research, which states applicants must show funding committees how the study is “designed or developed using methods appropriate for achieving the aims of the proposal.”
The take-out: “Some Australian institutions are likely not meeting their obligations to ensure good study designs as per the National Statement. Some institutions may be unaware of this oversight due to an undeclared shift in responsibility between the institution and the ethics committee(s).”
What is to be done: “Scrutinising studies before they start will minimise research waste by highlighting design errors when they can still be corrected.”
Plus:
- All committees should make a public statement about the study design;
- Universities should reduce the number of ethics committees and pay statisticians to participate.
And the one that committees will never wear: “a mock application is written and submitted to multiple committees. … This approach could reveal the large variation in statistical oversight and may provide first-hand evidence to committees about the value of statistical expertise.”
