Evidence-based medicine: Strengths and limitations
- Barbara Mintzes
- Early release
- 1 September 2013
Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.1 ~David Sackett (1996)
This quote, from one of the gurus of evidence-based medicine, highlights the link between the systematic evaluation of scientific evidence on effects of medical treatments, and applying this evidence to patient care decisions – taking into account both clinical expertise and patient needs. Evidence-based medicine is not simply applying scientific evidence. It requires an assessment and interpretation in light of individuals’ health needs and preferences.
Although the first randomised controlled trials in medicine date from the 1950s, for many years there was no way to understand the body of information produced by drug trials and other forms of medical research. Writing in 1979, Archie Cochrane said:
It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials.2
The Cochrane Collaboration (www.cochrane.org) was subsequently formed in 1992 and started its work looking at the effectiveness of obstetric interventions, as this was an area in which some interventions with substantial evidence of benefit were underused and others with little scientific basis remained widespread. The Cochrane Collaboration is an international network of more than 28 000 researchers from over 100 countries with different review groups based on clinical specialties. It helps healthcare providers, policy-makers, patients and their advocates and carers make well-informed decisions about health care by preparing, updating, and promoting the accessibility of Cochrane systematic reviews. A strength is the focus on methodological rigour, and Cochrane reviews are often seen as the ‘gold standard’ in systematic review. Over 5000 reviews have been published online in the Cochrane Database of Systematic Reviews (part of The Cochrane Library). The Collaboration also prepares the largest collection of records of randomised controlled trials in the world, called CENTRAL (published as part of The Cochrane Library).
The work of the Cochrane Collaboration and other evidence-based groups has grown significantly over the last 20 years. In 1981 there were only three meta-analyses indexed on Medline, in 1986 only 26. By 2011 the number had grown to around 8000.
Has the evidence-based movement been captured by commercial interests? Despite the rapid growth in the scale of activity to develop robust systematic review processes and procedures, there is a fundamental block in terms of limitations in the clinical trial evidence on the effects of new medicines. The requirements that manufacturers must meet to bring a new drug to market differ substantially from the clinical questions that physicians and patients face in order to understand a medicine’s place in therapy and decide whether or not to use it.
Studies of new pharmaceuticals are to a large degree shaped by regulatory requirements in order to obtain market approval. Currently, premarket randomised controlled trials are predominantly placebo controlled – or if not, they are non-inferiority trials. The latter are studies that are designed to test whether the drug is no worse than an existing treatment within a prespecified margin. Postmarketing surveillance data are mainly derived from observational studies. Regulators are becoming more interested in postmarketing safety surveillance, but there are no set standards for the rigour of the design of such studies.
Because most drug trials are designed and sponsored by the manufacturer, they are designed to present the drug in the best possible light and publication of all of the trial information is optional. When a drug becomes available there is usually no information on:
There is no requirement for manufacturers to prove that their products are at least as safe as existing drugs. This has meant that some new drugs have turned out to have a worse safety profile than existing drugs. For example, the most frequent serious adverse event associated with use of combined oestrogen and progestogen-containing oral contraceptives is venous thromboembolism. However, when a new oral contraceptive comes to market the manufacturer does not need to show that their newer pill has a lower or similar rate of venous thromboembolism to existing alternatives. Several newer oral contraceptives have been found to cause more venous thromboembolism than other oral contraceptives, including the drospirenone-containing products Yaz and Yasmin.3,4
In a regulatory environment without a requirement to establish an advantage in order to get a new drug to market, many new drugs are no more effective than existing alternatives. This overview is from La Revue Prescrire, a French independent drug bulletin that evaluates every new approved drug for its readership of over 30 000 practitioners.
The figure covers all new drugs and indications reviewed by La Revue Prescrire from 2002 to 2011.5 Over half were judged to be ‘me too’ drugs – i.e. nothing new. Around 7% had solid evidence of an advantage over existing drugs (whether major or minor). In around 15% of new drug reviews there was evidence of poor effectiveness or safety compared to drugs already on the market.
The pharmaceutical industry funds the majority of research, and therefore it also shapes the current research agenda. This leads to a bias of research focus on commercially viable interventions, where there is an anticipated beneficial outcome rather than harmful outcomes, and on products where there is likely to be a ready market. Consequently, there is not much research funding for drugs for neglected diseases affecting populations in low-income countries, where financial returns on research investments would be limited.
A body of research has examined the effects of industry funding on the evidence base and the types of bias it causes. A systematic review in the British Medical Journal in 20036 looked at the results of research trials in the US based on the type of funding the researchers received and the outcome of the trial data. It found that industry-funded trials are about four times more likely (95% confidence interval 3.0–5.5) to favour the sponsor’s drug than non-industry-funded trials. Another systematic review published in 2003 reported similar outcomes7, as have a number of more recent studies.
In late 2012, the first Cochrane systematic review on the effects of industry sponsorship was published, with a focus on studies of randomised controlled trials of drugs and devices.8 The authors found the same direction of effect as in previous systematic reviews but a lower magnitude: odds ratio = 2.2 (95% confidence interval 1.7–2.7) for favourable results and odds ratio = 2.7 (95% confidence interval 2.0–3.5) for conclusions as compared with non-industry-sponsored studies. The difference likely reflects the focus only on trials rather than inclusion of a broader range of research designs, including pharmacoeconomic analyses. The authors failed to find any clear trend of improvement over time.
Industry funding has also been found to influence the reporting of active-controlled trials that compare drugs in the same class (head-to-head trials). A cross-sectional study of head-to-head statin trials found a very strong association between the sponsor and the findings. Of 192 trials, 50% were sponsored by the test drug company, and these industry-funded trials were 20 times more likely to report results that favour the test drug over the competitor drug, and 35 times more likely to report favourable conclusions.9 Another study in 2010 looked at the outcome reporting among trials registered on the US Clinical Trials Registry covering six of the most heavily used classes of drugs (antidepressants, antipsychotics, proton pump inhibitors, lipid-lowering drugs, vasodilators). It found the proportion of favourable outcomes was high in trials funded both directly by industry and indirectly (i.e. trials conducted by non-profit organisations that receive industry funding).10
It is therefore clear from the literature that there is a direct link between funding source and outcomes of studies, and this leads to biased results. In a randomised controlled trial you would expect equipoise – you would not expect a situation where all trials funded by a specific source are two to three times more likely to be favourable than publicly funded trials.
The influence of the pharmaceutical industry extends beyond research. The other area where drug company influence is significant is the dissemination of information about the effectiveness of the drug and the trial results.
The pharmaceutical industry plays a dominant role in dissemination of information about drug products. It was estimated that US$57.5 billion was spent on drug promotion in the USA in 2004. This included the provision of samples, funding sales representatives, direct-to-consumer advertising, scientific meetings, promotions, advertisements and other unmonitored promotions.11 This is a massive investment – it would be great if 5% of this money went into a fund for independent research. It significantly outweighs the amount spent by public funders on the promotion and dissemination of health information, even in settings where there is a commitment to providing independent information. In the UK in 2005, the public sector spent only an estimated 0.3% of what industry spent on funding information dissemination, although the UK has more publicly provided health information than many other industrialised countries. Globally, Australia is one of the world leaders in having a larger commitment to publicly funded information.
This domination by industry poses a problem not only in terms of positively biased promotional information, but also the under-reporting of evidence about negative effects or harms.
There is also a lot of concern internationally about how the press represents or misrepresents research findings and how influential they are in creating public impressions of what drugs do or do not do. A review of press releases in EurekAlert (an online science news service) looked at the way journalists reported the results of a study and compared the reports with the study findings.12 It looked at different factors that were associated with ‘spin’ in the press release, i.e. misrepresentation of the study findings as more favourable and positive than they were. There was no association with the funding source of the study and the reporting of findings. Rather, the most significant factor was spin in the study itself (i.e. if the abstract represented the study findings as being more positive than the data would indicate).
The British Medical Journal has recently begun to provide a link to press releases on its website when it publishes a new article about a drug. A recent example of biased reporting in a press release was the 2012 reporting of the Heart and Estrogen/Progestogen Study (HERS) on the effects of hormone replacement therapy (HRT) in women.13 The press release was titled ‘HRT taken for 10 years significantly reduces the risk of heart failure and heart attack’. This implies the findings contradict the Women's Health Initiative study, but important details were not mentioned:
Another area of potential bias that is sometimes difficult to detect is conflict of interest. A recent study looked at the disclosures of medical experts who had been named as illegally promoting off-label use of products in court cases in the USA.14 Ninety-two people were named, 39 of them were authors of articles in the clinical area in which the off-label promotion occurred, and conflict of interest was adequately disclosed in only 15% of these articles.
Who and what is not being researched is another potential source of bias. When the COX-2 inhibitors (e.g. celecoxib, rofecoxib) were introduced they were marketed as being much safer than traditional non-steroidal anti-inflammatory drugs (NSAIDs) in terms of gastrointestinal bleeding. However, they had not been tested at all in people who were at high risk of bleeding (e.g. people who had previous gastrointestinal bleeds, the elderly). In an Ontario study of hospitalisations for the elderly for gastrointestinal bleeds, more bleeds occurred after these drugs began to be widely used, most likely because they were given to people who would not have otherwise been prescribed NSAIDs.15
Another source of bias in evidence-based medicine is the selective publication of research results. Erythropoietin and oseltamivir are two examples of publication bias leading to a distortion of the true therapeutic value of a drug.
Erythropoietin is useful for people with very low haemoglobin. It stimulates red cell production and reduces the need for blood transfusions in people having cancer chemotherapy or kidney dialysis. Erythropoietin was in one of the 10 classes of medicines most heavily promoted to the US public in 200516, with strong messages that the drug would increase energy levels, improve quality of life and allow patients to live as normally as possible. Problems arose when the use of the drug was extended well beyond the areas where it was proven beneficial, and there has since been evidence of higher mortality in patients who did not have very low haemoglobin before starting erythropoietin therapy. Another problem is that erythropoietin has the ability to promote the growth of tumours, which is concerning if it is prescribed for people with existing cancers (one of the approved indications). There is a strong association between the funding source and basic science studies that did and did not find this potentially harmful effect.17 The major study that led to clinical guidelines recommending the use of erythropoietin for kidney dialysis patients has been republished 14 years after the original publication (after the full trial data were accessed through the Freedom of Information Act).18 It shows that the original authors omitted very important information about serious adverse effects and potential for harm, and overstated the benefits in terms of improved quality of life.
The international stockpiling of oseltamivir highlights the worldwide impact of biased reporting. Stockpiling has occurred on two occasions – first in 2006 with the threat of bird flu, and more recently with the H1N1 pandemic in 2009. Billions of dollars were spent by governments all around the world to purchase bulk supplies of oseltamivir, believing that it would prevent the transmission of the H1N1 virus and flu complications in their populations.
In a Therapeutics Initiative review published in 2000, analysis of the clinical trial data showed that oseltamivir would only decrease the impact/duration of the flu by 0.7–0.8 of a day at best (about 18 hours) and that the effects on symptom severity were not able to be determined from the data. It found that the drug increased nausea and vomiting, and that there was no evidence that it prevented complications, hospitalisations or death. The trials excluded the elderly and chronically ill – the people who are at greatest risk from the virus.19
At the time Therapeutics Initiative did not realise how incomplete the evidence was, as this analysis was based on two published trials (one of which has subsequently been found to have been mainly ghostwritten). The US Food and Drug Administration approval of oseltamivir did not include publication on its website of the information submitted by the drug company until three years after the drug was approved for the market. For most drugs approved in the US, the Food and Drug Administration publishes a ‘review report’ on its website soon after approval, including a list of clinical trials submitted in the application, trial reports, and reviewer assessments.
The first Cochrane systematic review of oseltamivir in 2006 included a claim that it prevents flu complications.20 The review was based on one study report summarising the results of 10 pooled trials, 8 of which were unpublished, with most included patients in the unpublished trials.21
One of the positive aspects of a Cochrane review is that authors are required to routinely update their reviews and must commit to responding to any comments posted about the review. A Japanese researcher queried the findings of the 2006 pooled analysis of these 10 trials and asked whether the Cochrane reviewers had access to the full trial data for the 8 unpublished trials, and if not, how could the information supplied in the pooled analysis be trusted? As a result of this query, the Cochrane reviewers tried to obtain full reports of the unpublished trials. In the 2009 update of the Cochrane review, the claim that oseltamivir reduced flu complications was removed, as the data had not been disclosed. By 2012, when an updated Cochrane review was published, regulatory data had been disclosed. The authors found no effect on hospitalisation and could not pronounce on flu complications, due to incomplete and inconsistent reporting in the trials.
We found a high risk of publication and reporting biases … we are unable to draw conclusions about its effect on complications or transmission. We expect full clinical study reports ... to clarify outstanding issues. These full clinical study reports are currently unavailable to us.22
This experience demonstrates the discrepancy that can exist between published reports and the full reports of clinical trials submitted to regulators, and the importance of full disclosure.
Key evidence needed to support patient care decisions is often lacking. In particular this can mean there is no information about:
Is the dream of evidence-based medicine sinking like the Titanic? No – but there are key areas for policy change, including:
There is an important and stronger role for funders, payers and providers in research and resource allocation and managing the use of medicines through the provision of independent evidence-based information. One example of this is the approach taken by the non-profit health insurance plan, Kaiser Permanente, in the USA. They have developed an ‘in-house’ research team to assess the effectiveness of resource allocation, including outcomes from spending on new drugs. The team also provides the organisation with its own physician and patient information services on health conditions and drug and non-drug treatments. Kaiser Permanente physicians do not see sales representatives.
There is also a need to encourage reform of the regulatory landscape. This could include introducing limits to some forms of promotion of new drugs by the industry, promoting active pharmacosurveillance, and also promoting the design of pragmatic randomised controlled trials, systematic observational research and comparative effectiveness research through public funding.
Assistant Professor Mintzes provided a detailed declaration of her interests (see Appendix 2 for details)
Assistant Professor , School of Population and Public Health, University of British Columbia Vancouver, Canada
Scholar, Michael Smith Foundation for Health Research