Exclusions Before Randomisation 164
What to Look for in Exclusions Before Randomisation 164
Exclusions After Randomisation 164
What to Look for in Exclusions After Randomisation 166
Discovery of Participant Ineligibility 166
Post-Randomisation, Pretreatment Outcome 166
Protocol Deviations 167
Loss to Follow-Up and Retention 168
Proper randomisation means little if investigators cannot include all randomised participants in the primary analysis. Participants might ignore follow-up, leave town, or take acetaminophen when instructed to take aspirin. Exclusions before randomisation do not bias the treatment comparison, but they can hurt generalisability. Eligibility criteria for a trial should be clear, specific, and applied before randomisation. Readers should assess whether any of the criteria make the trial sample atypical or unrepresentative of the people in which they are interested. In principle, assessment of exclusions after randomisation is simple: none are allowed. For the primary analysis, all participants enrolled should be included and analysed as part of the original group assigned (an intent-to-treat analysis). In reality, however, losses frequently occur. Investigators should, therefore, commit adequate resources to develop and implement procedures to maximise retention of participants. Moreover, researchers should provide clear, explicit information on the progress of all randomised participants through the trial by use of, for instance, a trial profile. Investigators can also do secondary analyses on, for instance, per-protocol or as-treated participants. Such analyses should be described as secondary and nonrandomised comparisons. Mishandling of exclusions causes serious methodological difficulties. Unfortunately, some explanations for mishandling exclusions intuitively appeal to readers, disguising the seriousness of the issues. Creative mismanagement of exclusions can undermine trial validity.
Proper randomisation means little if investigators cannot include all randomly assigned participants in their primary analysis. Hence a crucial aspect of assessing a randomised controlled trial pertains to exclusions, withdrawals, losses, and protocol deviations. How should investigators handle participants who refuse entry, ignore follow-up, leave town, or take acetaminophen when they were instructed to take aspirin? Unfortunately, many inappropriate approaches to dealing with these types of problem actually seem logical and falsely appealing. Therein lies their insidious nature, because such inappropriate approaches can result in serious biases. Here we address the effect of exclusions made before and after randomisation.
Exclusions Before Randomisation
Investigators can exclude participants before randomisation. The eventual randomised treatment comparison will remain unbiased (good internal validity), irrespective of whether researchers have well-founded or whimsical reasons for exclusion of particular individuals. However, exclusions at this stage can hurt extrapolation, the generalisability of the results (external validity). For most investigations, we therefore recommend that eligibility criteria be kept to a minimum, in the spirit of the large and simple trial. However, some valid reasons exist for exclusion of certain participants. Individuals could, for example, have a condition for which an intervention is contraindicated, or they could be judged likely to be lost to follow-up. The trial question should guide the approach. Sometimes, however, investigators impose so many eligibility criteria that their trial infers to a population of little apparent interest to anyone, and, in addition, recruitment becomes difficult. If investigators exclude too many participants, or the wrong participants, their results might not represent the people of interest, even though the randomised controlled trial might have been meticulously done (i.e., the results could be true but potentially irrelevant).
What to look for in exclusions before randomisation
The eligibility criteria should indicate the population to which the investigators wish to infer. When judging the results of a trial, readers should make sure that the eligibility criteria are clear and specific. Most importantly the criteria should have been applied before randomisation. Readers should also assess whether any of the criteria make the study sample atypical, unrepresentative, or irrelevant to the people of interest. In practice, however, results from a trial will infrequently be totally irrelevant: ‘most differences between our patients and those in trials tend to be quantitative (they have different ages or social classes or different degrees of risk of the outcome event or of responsiveness to therapy) rather than qualitative (total absence of responsiveness or no risk of the event)’. Such qualitative differences in response are rare; thus, trials tend to have rather robust external validity.
Exclusions After Randomisation
Exclusions made after randomisation threaten to bias treatment comparisons. Randomisation itself configures unbiased comparison groups at baseline. Any erosion, however, over the course of the trial from those initially unbiased groups produces bias, unless, of course, that erosion is random, which is unlikely. Consequently, for the primary analysis, methodologists suggest that results for all patients who are randomly assigned should be analysed and, furthermore, should be analysed as part of the group to which they were initially assigned. Trialists refer to such an approach as an intent-to-treat (ITT) analysis. Simply put: once randomised, always analysed as assigned.
ITT principles underlie the primary analysis in a randomised controlled trial to avoid biases associated with nonrandom loss of participants. Investigators can also do secondary analyses, preferably preplanned, based on only those participants, for example, who comply with the trial treatment protocol or who receive the treatment irrespective of randomised assignment (generally referred to as per protocol [PP], on-treatment, or as-treated analysis). Secondary analyses are acceptable as long as researchers label them as secondary and nonrandomised comparisons. Trouble brews, however, when investigators exclude participants and, in effect, present a secondary, nonrandomised comparison as the primary randomised comparison from a trial. In reality, this analysis represents a cohort study masquerading as a randomised controlled trial. Exclusion of participants from an analysis can lead to misleading conclusions ( Panel 15.1 ).
For this trial, the researchers reported a primary analysis that compared rates of death from cardiac causes rather than from all cardiac deaths. In their analysis, inappropriate exclusions due to eventual discovery of patient ineligibility caused a problem : the investigators withdrew as ineligible seven patients who had received treatment (six in the treatment group and one in the placebo group) resulting in more patients who died being withdrawn from the treatment group than from the placebo group.
Moreover, results of a detailed audit of this trial by the US Food and Drug Administration (FDA) indicate that additional patients from the placebo group could have been declared ineligible on the basis of similar criteria, but were not. Furthermore, the trial protocol did not mention exclusion of ineligible patients after entry, particularly patients who had died. The researchers also excluded two deaths in the sulfinpyrazone group and one death in the placebo group as nonanalysable because of poor compliance. However, the trial protocol did not include plans to exclude patients because of poor compliance.
Additionally, the investigators used a 7-day rule. They declared as nonanalysable any death of a patient who had not received treatment for at least 7 days or who died more than 7 days after termination of treatment. The FDA review committee did not criticise this practice strongly, principally because the protocol described the 7-day rule, and also because the rule had little overall effect on the results.
Overall, these inappropriate exclusions did, however, affect the results of the study. Although the researchers initially reported a 32% reduction ( p = 0.058) in rates of death from cardiac causes for participants who took the drug, a reanalysis showed a weaker result. When individuals judged ineligible or nonanalysable were included in the originally assigned groups, the reduction was only 21% ( p = 0.16). It is noteworthy that only p values were provided.
We urge the use of confidence intervals in reporting results. Moreover, the fallout from inappropriate exclusions, as ascertained by the FDA, cast doubt over the trial. The FDA advisory committee announced that sulfinpyrazone could not be labelled and advertised as a drug to prevent death in the critical months after a heart attack because, on close examination, the data were not as convincing as they seemed at first glance.
Researchers often do not provide adequate information on excluded participants. Furthermore, in an older review of 249 randomised controlled trials published in major general medical journals in 1997, only 2% (5 of 249) of the reports explicitly stated that all randomly assigned participants were analysed according to the randomised group assignment. About half of the reports (119 of 249) noted an ITT analysis, but many provided no details to support this claim.
More recent data on ITT analyses are not encouraging. In the 50 trials of experimental pain models, none (0%) reported using an ITT analysis. Of the 123 trials of clinical pain, 47% reported using an ITT analysis and 5% reported using a modified ITT (mITT). In another review of 2349 trials, 25% were classified as an ITT analysis, 14% as an mITT analysis, and 61% as not reporting an ITT analysis. Moreover, the mITT classification is a misnomer in that it is decidedly not even close to an ITT analysis. ‘Whatever the definition used in the ITT approach, it is clear that the analysis performed in the mITT trials is a “PP [per-protocol] analysis”. It is important to underline that the authors of mITT studies inappropriately used the term “ITT” because in reality, the analyses they performed were substantially “PP analyses”, which can confuse the reader’. PP analyses, sometimes called ‘as-treated’ or ‘on-treatment’ analyses, represent distinctly non-ITT analyses in that they are nonrandomised comparisons. Indeed, the mITT trials displayed larger treatment effects than ITT trials, which supports the view that mITT trials represent nonrandomised, biased comparisons.
Additionally, researchers frequently do not report anything with respect to exclusions. Left in this information void, many readers deduce that certain trials used ITT principles and had no exclusions. We call this scenario ‘no apparent exclusions’. Readers commonly view trials with no apparent exclusions as less biased, when in fact unreported exclusions probably occurred in many of them. Indeed, trials with no apparent exclusions were methodologically weaker than those reporting at least some exclusions. In other words, some of the more biased trials might be mistakenly interpreted as unbiased, and many of the less biased trials as biased; we call this inconsistency ‘the exclusion paradox’. Until researchers comprehensively report exclusions after randomisation, readers should be aware of this unsettling irony.
What to look for in exclusions after randomisation
Before we launch into attributes of proper handling of exclusions after randomisation, we should acknowledge the tenuous ground on which any such discussion rests. Reporting on exclusions is poor, with the exclusion paradox misleading readers. Investigators should provide clear, explicit information on the progress through the trial of all randomised participants, and when such information is absent, readers should be sceptical. The flow diagrams specified in the CONSORT statement provide appropriate guidelines.
Optimally, of course, investigators would have no exclusions after randomisation and use an ITT analysis. Assessment of exclusions after randomisation is simple: none are allowed. All participants enrolled should be analysed as part of the original group assigned. Clinical research is not normally that simple, but the principle holds. One pragmatic hint for minimising exclusions after randomisation involves randomly assigning individuals at the last possible moment. If randomisation takes place when the participant is first identified, but before treatment is initiated, then any exclusions arising before treatment still become exclusions after randomisation. Investigators can address this potential difficulty by delaying randomisation until immediately before treatment begins.
If investigators report exclusions after randomisation, those exclusions should be carefully scrutinised because they could bias comparisons. Exclusions arise after randomisation for several reasons, including discovery of patient ineligibility; post-randomisation, pretreatment outcome; deviation from protocol; and losses to follow-up.
Discovery of Participant Ineligibility
In some trials, participants are enrolled and later discovered not to have met the eligibility criteria. Exclusions at this point could seriously bias the results, because discovery is probably not random. For example, participants least responsive to treatment or who have side effects might draw more attention and, therefore, might be more likely to be judged ineligible than other study participants. Alternatively, a physician who had treatment preferences for certain participants might withdraw individuals from the trial if they were randomly assigned to what he or she believes to be the wrong group.
Participants discovered to be ineligible should remain in the trial. An exception could be made if establishment of eligibility criteria is difficult. In such instances, investigators could obtain the same information from each patient at time of randomisation and have it centrally reviewed by an outside source, blinded to the assigned treatment. That source, whether a person or group, could then withdraw patients who did not satisfy the eligibility criteria, presumably in an unbiased way.
Post-Randomisation, Pretreatment Outcome
Researchers sometimes report exclusion of participants on the basis of outcomes that happen before treatment has begun or before the treatment could have had an effect. For example, in a clinical trial of a specific drug’s effect on death rates, investigators withdrew as nonanalysable data on all patients who died after randomisation but before treatment began or before they had received at least 7 days of treatment. This winnowing seems intuitively attractive, because none of the deaths can then be attributable to treatment. But the same argument could be made for excluding data on all patients in a placebo group who died during the entire study interval, because, theoretically, none of these deaths could have been related to treatment. This example illustrates the potential for capriciousness in addressing post-randomisation, pretreatment outcomes.
Randomisation tends to balance the nonattributable deaths in the long run. Any tinkering after randomisation, even if done in the most scientific and impartial manner, cannot improve upon that attribute but can hurt it. More importantly, this meddling sometimes serves as a post hoc rationale for inappropriate exclusions.
Post hoc rationalisation arises when investigators observe the results and then frame rules that favour their hypotheses. Assume that an investigator postulates that a drug used for treatment reduced the death rate associated with a particular condition. After analysis of data, however, the investigator notes that 14 deaths in the treatment group and two deaths in the placebo group arose before treatment had begun or before the drug had been taken for at least 7 days. She then rationalises the deaths as unrelated to treatment and withdraws them from analysis. Such a response would seriously bias her results, even though her reasoning in the report would likely seem logical.
Imposed a priori, such rules only complicate trial implementation; imposed a posteriori, they lead to biased and invalid results. In assessment of randomised controlled trials, identification of when researchers stipulated rules usually proves impossible. We prefer to find, in reports of randomised controlled trials, that investigators did not allow any withdrawal of participants after randomisation. The data of all randomised patients should be analysed. Planned or unplanned, the exclusion of nonanalysable outcomes on grounds of efficiency is not a generally accepted practice in the analysis of a randomised clinical trial.