‘It-works-somewhere’

First, the ‘it-works-somewhere’ claim will be evaluated. RCTs in psychiatry may have bias in design, recruitment, patient populations, data analysis and presentation of findings. Studies are relatively small generally involving, at most, a few hundred subjects. The treatment effect sizes are small, which compounds the problem of clinical utility and translation. Many syndromes have high spontaneous recovery and placebo response rates, which complicate analyses and obfuscate effects. Definitions of syndromes are often imprecise and overlapping while simultaneously heterogeneous, and commonly result in highly mixed samples. Added to this, a variety of outcome measures are used while interventions are often only conducted in tertiary referral units.3 Such specialist units are usually found in academic institutions, where patients are referred due to more complex illness patterns. Such groups of patients tend to have poorer prospects of remission and are rarely reflective of the general population with mood disorders.

There is consistent evidence of selected or distorted reporting in RCTs. Chan et al 4 reviewed 102 trials and noted that the reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Over half the trials were reported either in part (incompletely) or not at all, with statistically significant results having higher odds of being reported compared with non-significant outcomes for both efficacy (pooled OR 2.4) and harm (pooled OR 4.7). In other words, a significant outcome results in a greater likelihood of being reported. More disturbingly, 86% of survey responders denied the existence of unreported outcomes despite evidence to the contrary. A prominent example of this in psychiatry is Study 329: an RCT comparing paroxetine, imipramine and placebo in adolescents. A recent re-analysis reported that paroxetine only produced a positive result when four new secondary outcome measures were used instead of the primary outcomes. Analysing the primary outcome measure revealed no group differences.5

Due to these concerns, the International Committee of Medical Journal Editors introduced a policy to require registration of all clinical trials prior to enrolment of subjects.6 Registration involves information about trial protocols and the specified outcome measures being made publicly available. However, the efficacy of trial registration has been called into question by a number of studies. In a review of five psychiatry journals that mandate registering of prospective clinical trials, it was reported that only 33% of trials were correctly prospectively registered and, of these, 28% had evidence of selective outcome reporting and 27% a large change in participant numbers. Overall, only 14.4% were correctly registered and reported.7 For psychotherapy RCTs, the results were even worse. Only 24.1% were registered and 4.5% free from selective outcome reporting,8 underscoring the fact that bias is not an issue just confined to pharmaceutical industry trials.

The other major attempt to improve the conduct and reporting of RCTs is the Consolidated Standards of Report Trials (CONSORT) guidelines. These provide an evidence-based minimum set of recommendations for reporting RCTs.9 While there is evidence that reporting in psychiatric RCTs has improved, over 40% of studies still do not adhere to the CONSORT guidelines.10

There is also the often cited influence of pharmaceutical marketing where there may be motivation for bias in design and external validity. A review of drug company authorship and sponsorship on drug trial outcomes reported that of 198 studies in three prestigious psychiatry journals (British Journal of Psychiatry, American Journal of Psychiatry and JAMA Psychiatry), only 23% were independently funded. Furthermore, independently funded studies were significantly more likely to report negative findings whereas industry-authored studies nearly always reported positive findings. Specifically, 74 out of 76 RCTs in this study demonstrated this bias11—although journal editors are also reluctant to publish negative studies suggesting that sources of this bias arise at multiple levels. Similar effects are found in psychotherapy RCTs. Larger positive effect sizes were found when authors’ allegiance to the studied psychotherapy existed and this allegiance effect was even stronger where the RCT was performed by the developer of the preferred treatment.12 13 Finally, the influence of publication bias remains an issue. A large survey of RCT researchers (n=318) revealed that around 25% of trials go unpublished and these unpublished studies are less likely to have favoured the new therapy. Interestingly, they noted that non-publication was primarily a result of failure to write up and submit trial results rather than a rejection of submitted manuscripts.14

Overall, while the ‘it-works-somewhere’ claim is somewhat more likely to be true over the past decade, it remains possible that many published RCTs are spurious or at least overstate their claims through a combination of methodological flaws (eg, type I and type II errors), selective reporting, marketing interests and publication bias.

‘It-will-work-for-us’

The ‘it-will-work-for-us’ claim is more difficult to evaluate. The CONSORT statement mandates a clear exposition of the recruitment pathway by which patients enter the RCT. This reporting is intended to enable clinicians to judge to whom the results of the RCT apply. But again the reality falls short. A review of trials leading to clinical alerts by the US National Institute of Health revealed that in relation to 31 eligibility criteria, only 63% were published in the main trial report and only 19% in the clinical alert.15 Inadequate reporting is even more of a problem in secondary publications such as clinical guidelines since space limitations and the need for a succinct message do not allow for detailed consideration of eligibility of trials or other determinants of external validity.16 Exclusion of common comorbidities is one of the common factors preventing real-world generalisability of RCTs. Furthermore, the population-level statistical approach to evidence-based medicine can ‘homogenise’ the complex heterogeneity of clinical reality and produce empirical data sets that lack clinical salience to real-world patients. It is not good to be an outlier when a standard population-based evidence approach is applied to your care.

Much less discussed (other than the type II error bias)17 is the possibility that true findings may be annulled because of reverse bias (ie, bias may under-estimate treatment effect). A potential source of reverse bias in psychiatric RCTs emerges from recruitment strategies. For example, participants entering clinical trials for depression are likely to be mild or moderately depressed, sometimes better diagnosed as having an adjustment disorder or a persistent depressive disorder, or a depression related to psychosocial adversity, but are all lumped into a ‘major depression’ category to meet recruitment targets. Patients may inflate their scores to get ‘free’ treatment while assessing clinicians may inflate scores to enhance recruitment.18 Many trials exclude those with common comorbidities, such as those with suicidal ideation, and only seldom is a developmental trauma history obtained to better inform diagnosis.19