Purpose of the Chapter
This final chapter has several purposes. Most importantly, it ties together concepts and skills presented in previous chapters and applies these concepts very specifically to reading medical journal articles. Throughout the text, we have attempted to illustrate the strengths and weaknesses of some of the studies discussed, but this chapter focuses specifically on those attributes of a study that indicate whether we, as readers of the medical literature, can use the results with confidence. The chapter begins with a brief summary of major types of medical studies. Next, we examine the anatomy of a typical journal article in detail, and we discuss the contents of each component—abstract or summary, introduction, methods, results, discussion, and conclusions. In this examination, we also point out common shortcomings, sources of bias, and threats to the validity of studies.
Clinicians read the literature for many different reasons. Some articles are of interest because you want only to be aware of advances in a field. In these instances, you may decide to skim the article with little interest in how the study was designed and carried out. In such cases, it may be possible to depend on experts in the field who write review articles to provide a relatively superficial level of information. On other occasions, however, you want to know whether the conclusions of the study are valid, perhaps so that they can be used to determine patient care or to plan a research project. In these situations, you need to read and evaluate the article with a critical eye in order to detect poorly done studies that arrive at unwarranted conclusions.
To assist readers in their critical reviews, we present a checklist for evaluating the validity of a journal article. The checklist notes some of the characteristics of a well-designed and well-written article. The checklist is based on our experiences with medical students, house staff, journal clubs, and interactions with physician colleagues. It also reflects the opinions expressed in an article describing how journal editors and statisticians can interact to improve the quality of published medical research (Marks et al, 1988). A number of authors have found that only a minority of published studies meet the criteria for scientific adequacy. The checklist should assist you in using your time most effectively by allowing you to differentiate valid articles from poorly done studies so that you can concentrate on the more productive ones.
Two guidelines recently published increase our optimism that the quality of the published literature will continue to improve. The International Conference on Harmonization (ICH) E9 guideline “Statistical Principles for Clinical Trials” (1999) addresses issues of statistical methodology in the design, conduct, analysis, and evaluation of clinical trials. Application of the principles is intended to facilitate the general acceptance of analyses and conclusions drawn from clinical trials.
The International Committee of Medical Journal Editors published the Uniform Requirements of Manuscripts Submitted to Biomedical Journals in 1997. Under Statistics, the document states:
Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to verify the reported results. . When data are summarized in the results section, specify the statistical methods used to analyze them.
The requirements also recommend the use of confidence intervals and to avoid depending solely on P values.
Review of Major Study Designs
Chapter 2 introduced the major types of study designs used in medical research, broadly divided into experimental studies (including clinical trials); observational studies (cohort, case–control, cross-sectional/surveys, case–series); and meta-analyses. Each design has certain advantages over the others as well as some specific disadvantages; they are briefly summarized in the following paragraphs. (A more detailed discussion is presented in Chapter 2.)
Clinical trials provide the strongest evidence for causation because they are experiments and, as such, are subject to the least number of problems or biases. Trials with randomized controls are the study type of choice when the objective is to evaluate the effectiveness of a treatment or a procedure. Drawbacks to using clinical trials include their expense and the generally long time needed to complete them.
Cohort studies are the best observational study design for investigating the causes of a condition, the course of a disease, or risk factors. Causation cannot be proved with cohort studies, because they do not involve interventions. Because they are longitudinal studies, however, they incorporate the correct time sequence to provide strong evidence for possible causes and effects. In addition, in cohort studies that are prospective, as opposed to historical, investigators can control many sources of bias. Cohort studies have disadvantages, of course. If they take a long time to complete, they are frequently weakened by patient attrition. They are also expensive to carry out if the disease or outcome is rare (so that a large number of subjects needs to be followed) or requires a long time to develop.
Case–control studies are an efficient way to study rare diseases, examine conditions that take a long time to develop, or investigate a preliminary hypothesis. They are the quickest and generally the least expensive studies to design and carry out. Case–control studies also are the most vulnerable to possible biases, however, and they depend entirely on high-quality existing records. A major issue in case–control studies is the selection of an appropriate control group. Some statisticians have recommended the use of two control groups: one similar in some ways to the cases (such as having been hospitalized or treated during the same period) and another made up of healthy subjects.
Cross-sectional studies and surveys are best for determining the status of a disease or condition at a particular point in time; they are similar to case–control studies in being relatively quick and inexpensive to complete. Because cross-sectional studies provide only a snapshot in time, they may lead to misleading conclusions if interest focuses on a disease or other time-dependent process.
Case–series studies are the weakest kinds of observational studies and represent a description of typically unplanned observations; in fact, many would not call them studies at all. Their primary use is to provide insights for research questions to be addressed by subsequent, planned studies.
Studies that focus on outcomes can be experimental or observational. Clinical outcomes remain the major focus, but emphasis is increasingly placed on functional status and quality-of-life measures. It is important to use properly designed and evaluated methods to collect outcome data. Evidence-based medicine makes great use of outcome studies.
Meta-analysis may likewise focus on clinical trials or observational studies. Meta-analyses differ from the traditional review articles in that they attempt to evaluate the quality of the research and quantify the summary data. They are helpful when the available evidence is based on studies with small sample sizes or when studies come to conflicting conclusions. Meta-analyses do not, however, take the place of well-designed clinical trials.
The Abstract & Introduction Sections of a Research Report
Journal articles almost always include an abstract or summary of the article prior to the body of the article itself. Most of us are guilty of reading only the abstract on occasion, perhaps because we are in a great hurry or have only a cursory interest in the topic. This practice is unwise when it is important to know whether the conclusions stated in the article are justified and can be used to make decisions. This section discusses the abstract and introduction portions of a research report and outlines the information they should contain.
The major purposes of the abstract are (1) to tell readers enough about the article so they can decide whether to read it in its entirely and (2) to identify the focus of the study. The International Committee of Medical Journal Editors (1997) recommended that the abstract “state the purposes of the study or investigation, basic procedures (selection of study subjects or experimented animals; observational and analytic methods), main findings (specific data and their statistical significance, if possible) and the principal conclusions.” An increasing number of journals, especially those we consider to be of high quality, now use structured abstracts in which authors succinctly provide the above-mentioned information in separate, easily identified paragraphs (Haynes et al, 1990).
We suggest asking two questions to decide whether to read the article: (1) If the study has been properly designed and analyzed, would the results be important and worth knowing? (2) If the results are statistically significant, does the magnitude of the change or effect also have clinical significance; if the results are not statistically significant, was the sample size sufficiently large to detect a meaningful difference or effect? If the answers to these questions are yes, then it is worthwhile to continue to read the report. Structured abstracts are a boon to the busy reader and frequently contain enough information to answer these two questions.
At one time, the following topics were discussed (or should have been discussed) in the introduction section; however, with the advent of the structured abstract, many of these topics are now addressed directly in that section. The important issue is that the information be available and easy to identify.
The introduction section of a research report is usually fairly short. Generally, the authors briefly mention previous research that indicates the need for the present study. In some situations, the study is a natural outgrowth or the next logical step of previous studies. In other circumstances, previous studies have been inadequate in one way or another. The overall purpose of this information is twofold: to provide the necessary background information to place the present study in its proper context and to provide reasons for doing the present study. In some journals, the main justification for the study is given in the discussion section of the article instead of in the introduction.
Regardless of the placement of background information on the study, the introduction section is where the investigators communicate the purpose of their study. The purpose of the study is frequently presented in the last paragraph or last sentences at the end of the introduction. The purpose should be stated clearly and succinctly, in a manner analogous to a 15-second summary of a patient case. For example, in the study described in Chapter 5, Dennison and colleagues (1997, p. 15) do this very well; they stated their objective as follows:
To evaluate, in a population-based sample of healthy children, fruit juice consumption and its effects on growth parameters during early childhood.
This statement concisely communicates the population of interest (healthy children), the focus of the study or independent variable (fruit juice consumption), and the outcome (effects on growth). As readers, we should be able to determine whether the purpose for the study was conceived prior to data collection or if it evolved after the authors viewed their data; the latter situation is much more likely to capitalize on chance findings. The lack of a clearly stated research question is the most common reason medical manuscripts are rejected by journal editors (Marks et al, 1988).
In addition to stating the purpose of the study, the structured abstract or introduction section may include information on the study’s location, length of time, and subjects. Alternatively, this information may be contained in the methods sections. This information helps readers decide whether the location of the study and the type of subjects included in the study are applicable in the readers’ own practice environment.
The time covered by a study gives important clues regarding the validity of the results. If the study on a particular therapy covers too long a period, patients entering at the beginning of the study may differ in important ways from those entering at the end. For example, major changes may have occurred in the way the disease in question is diagnosed, and patients entering near the end of the study may have had their disease diagnosed at an earlier stage than did patients who entered the study early (see detection bias, in the section of that title.). If the purpose of the study is to examine sequelae of a condition or procedure, the period covered by the study must be sufficiently long to detect consequences.
The Method Section of a Research Report
The method section contains information about how the study was done. Simply knowing the study design provides a great deal of information, and this information is often given in a structured abstract. In addition, the method section contains information regarding subjects who participated in the study or, in animal or inanimate studies, information on the animals or materials. The procedures used should be described in sufficient detail that the reader knows how measurements were made. If methods are novel or require interpretation, information should be given on the reliability of the assessments. The study outcomes should be specified along with the criteria used to assess them. The method section also should include information on the sample size for the study and on the statistical methods used to analyze the data; this information is often placed at the end of the method section. Each of these topics is discussed in detail in this section.
How well the study has been designed is of utmost importance. The most critical statistical errors, according to a statistical consultant to the New England Journal of Medicine, involve improper research design: “Whereas one can correct incorrect analytical techniques with a simple reanalysis of the data, an error in research design is almost always fatal to the study—one cannot correct for it subsequent to data collection” (Marks et al, 1988, p. 1004). Many statistical advances have occurred in recent years, especially in the methods used to design, conduct, and analyze clinical trials, and investigators should offer evidence that they have obtained expert advice.
Authors should provide several critical pieces of information about the subjects included in their study so that we readers can judge the applicability of the study results. Of foremost importance is how the patients were selected for the study and, if the study is a clinical trial, how treatment assignments were made.
Randomized selection or assignment greatly enhances the generalizability of the results and avoids biases that otherwise may occur in patient selection (see the section titled, “Bias Related to Subject Selection.”). Some authors believe it is sufficient merely to state that subjects were randomly selected or treatments were randomly assigned, but most statisticians recommend that the type of randomization process be specified as well. Authors who report the randomization methods provide some assurance that randomization actually occurred, because some investigators have a faulty view of what constitutes randomization. For example, an investigator may believe that assigning patients to the treatment and the control on alternate days makes the assignment random. As we emphasized in Chapter 4, however, randomization involves one of the precise methods that ensure that each subject (or treatment) has a known probability of being selected.
The authors should present information to illustrate that major selection biases (discussed in the section titled, “Bias Related to Subject Selection.”) have been avoided, an aspect especially important in nonrandomized trials. The issue of which patients serve as controls was discussed in Chapter 2 in the context of case–control studies. In addition, the eligibility criteria for both inclusion and exclusion of subjects in the study must be specified in detail. We should be able to state, given any hypothetical subject, whether this person would be included in or excluded from the study. Sauter and coworkers (2002) gave the following information on patients included in their study:
Patients undergoing CHE in our Surgical Department were consecutively included into the study provided that they did not meet one the following exclusion criteria: (a) inflammatory bowel disease, history of intestinal surgery, or diarrhea within the preceding 2 years, (b) body weight > 90 kg, (c) pregnancy, (d) abnormal liver function tests . ., (e) diabetes mellitus, (f) history of radiation of the abdominal region, and (g) drug therapy with antibiotics, lipid lower agents, laxatives, and cholestyramine.
For similar reasons, sufficient information must be given regarding the procedures the investigators used to follow up patients, and they should state the numbers lost to follow-up. Some articles include this information under the results section instead of in the methods section.
The description of follow-up and dropouts should be sufficiently detailed to permit the reader to draw a diagram of the information. Occasionally, an article presents such a diagram, as was done by Hébert and colleagues in their study of elderly residents in Canada (1997), reproduced in Figure 13–1. Such a diagram makes very clear the number of patients who were eligible, those who were not eligible because of specific reasons, the dropouts, and so on.
Figure 13–1.
Flow of the subjects through the study, a representative sample of elderly people living at home in Sherbrooke, Canada, 1991–1993. (Reproduced, with permission, from Figure 1 in Hébert R, Brayne C, Spiegelhalter D: Incidence of functional decline and improvement in a community-dwelling very elderly population. Am J Epidemiol 1997;145:935–944.)
Bias in studies should not happen; it is an error related to selecting subjects or procedures or to measuring a characteristic. Biases are sometimes called measurement errors or systematic errors to distinguish them from random error (random variation), which occurs any time a sample is selected from a population. This section discusses selection bias, a type of bias common in medical research.
Selection biases can occur in any study, but they are easier to control in clinical trials and cohort designs. It is important to be aware of selection biases, even though it is not always possible to predict exactly how their presence affects the conclusions. Sackett (1979) enumerated 35 different biases. We discuss some of the major ones that seem especially important to the clinician. If you are interested in a more detailed discussion, consult the article by Sackett and the text by Feinstein (1985), which devotes several chapters to the discussion of bias (especially Chapter 4, in the section titled “The Meaning of the Term Probability,” and Chapters 15–17).
Prevalence (Neyman) bias occurs when a condition is characterized by early fatalities (some subjects die before they are diagnosed) or silent cases (cases in which the evidence of exposure disappears when the disease begins). Prevalence bias can result whenever a time gap occurs between exposure and selection of study subjects and the worst cases have died. A cohort study begun prior to the onset of disease is able to detect occurrences properly, but a case–control study that begins at a later date consists only of the people who did not die. This bias can be prevented in cohort studies and avoided in case–control studies by limiting eligibility for the study to newly diagnosed or incident cases. The practice of limiting eligibility is common in population-based case–control studies in cancer epidemiology.
To illustrate prevalence or incidence bias, let us suppose that two groups of people are being studied: those with a risk factor for a given disease (eg, hypertension as a risk factor for stroke) and those without the risk factor. Suppose 1000 people with hypertension and 1000 people without hypertension have been followed for 10 years. At this point, we might have the situation shown in Table 13–1.
| Number of Patients in 10-Year Cohort Study | |||
|---|---|---|---|
| Patients | Alive with Cerebrovascular Disease | Dead from Stroke | Alive with No Cerebrovascular Disease |
| With hypertension | 50 | 250 | 700 |
| Without hypertension | 80 | 20 | 900 |
A cohort study begun 10 years ago would conclude correctly that patients with hypertension are more likely to develop cerebrovascular disease than patients without hypertension (300 to 100) and far more likely to die from it (250 to 20).
Suppose, however, a case–control study is undertaken at the end of the 10-year period without limiting eligibility to newly diagnosed cases of cerebrovascular disease. Then the situation illustrated in Table 13–2 occurs.
| Number of Patients in Case–Control Study at End of 10 Years | ||
|---|---|---|
| Patients | With Cerebrovascular Disease | Without Cerebrovascular Disease |
| With hypertension | 50 | 700 |
| Without hypertension | 80 | 900 |
The odds ratio is calculated as (50 × 900)/(80 × 700) = 0.80, making it appear that hypertension is actually a protective factor for the disease! The bias introduced in an improperly designed case–control study of a disease that kills off one group faster than the other can lead to a conclusion exactly the opposite of the correct conclusion that would be obtained from a well-designed case–control study or a cohort study.
