Clinical research can be broadly divided into two subsets: experimental research and observational research. The vast majority of new medical treatments and technologies are tested through experimental or interventional research, often in the form of randomized trials, before they are adopted into clinical use. In contrast, observational studies are primarily conducted on technologies after they have already been adopted and are being implemented in some sector of the healthcare community. Observational research occupies a critical niche within healthcare research that is complementary to experimental studies. Understanding the relative strengths, weaknesses, similarities, and differences between observational and experimental research is critical to accurately interpreting clinical research.

A Working Definition of Observational Research

Observational research is a research in which the investigator cannot control the assignment of treatment to subjects because the participants or conditions are not being directly assigned by the researcher. Observational research examines predetermined treatments, interventions, and policies and their effects. In practical terms, observational comparative effectiveness research (CER) is typically conducted within one of two settings, either within registries or as subgroup analyses within randomized clinical trials. Registries are generally created with a specific disease, treatment, or population of interest, and can occur within a specific institution, network of institutions, or geographic region within which clinically relevant outcomes are recorded. Subgroup analyses within clinical trials include any subset for which patients are not randomly assigned. Because subgroups are not randomly assigned, subgroup analyses share all the strengths and weaknesses of conventional observational studies, such as confounding and multiple hypotheses testing, and provide a similar level of evidence.

In contrast to observational research, researchers in experimental studies directly manipulate or assign participants to different interventions or environments. A third type of research involves descriptive studies, which are conducted without a treatment and are neither experimental nor observational (1). This type of research is used in the initial exploration and characterization of a healthcare issue. Descriptive studies play no direct role in CER, whereas experimental and observational studies are important in both developmental and CER.

Strengths and Weaknesses of Observational Research Within Comparative Effectiveness Research

Recent focus on the importance of CER was reinvigorated with passage of the Patient Protection and Affordable Care Act (PPACA) in 2010 (2). From a practical research standpoint, this emphasis on CER makes it important to define and understand observational research within the context of CER.

The Agency for Healthcare Research and Quality (AHRQ), the lead federal agency responsible for improving healthcare quality in the United States, defines CER as research that provides “evidence on the effectiveness, benefits, and harms of different treatment options” (3). This evidence is generated via comparative studies of drugs, medical devices, tests, surgeries, or ways to deliver health care. We find such evidence in one of two ways: through experimental studies or through observational studies. Each of these approaches has different strengths and weaknesses, and each uses both overlapping and distinct methods.

In a CER context, the strength of experimental studies is that they provide the strongest evaluation or validation of a specific, well-defined intervention or treatment. In particular, experimental studies provide the highest level of evidence for evaluating new therapies, which includes the current “gold standard” in CER—the randomized controlled trial (RCT). In recent years, several examples have arisen in which observational studies have been confounded by selection bias, which have later been contradicted by RCTs (Table 10–1).

One of the best-known examples of this is the effect of hormone replacement therapy on women’s health. Numerous observational studies had shown a cardioprotective benefit of this therapy against heart attack and stroke in women. However, a large randomized clinical trial later showed that women randomized to receive hormone therapy showed no difference in cardiovascular events, a reduced fracture risk, and significant increases in venous clot formation and embolic events (4). Another example is the concept that vitamin B12 and folate supplementation could reduce the risk of major coronary events. This hypothesis was initially supported by numerous observational data but was later refuted by the randomized Study of the Effectiveness of Additional Reductions in Cholesterol and Homocysteine (SEARCH) (5).

In both cases, the conclusion was that selection bias was likely confounding the results of observational studies. Women receiving hormone therapy appeared to differ systematically from women who were not receiving hormone therapy (prevalent user bias), and the previously observed association between hormone therapy and cardioprotection likely resulted from confounding by overall improved access to health care, patient demographics, and other unobserved phenomena associated with therapy and improved cardiovascular health. The use of hormone therapy in women and its effects on cardioprotection remain actively debated topics; nonetheless, they serve as important examples of discrepancies between observational studies and RCTs.

The strength of the experimental study—evaluation of a specific, controllable treatment, or exposure—is also responsible for its key weakness: being difficult, impractical, or even impossible to conduct in complex, poorly defined, or poorly controlled situations. This limitation becomes particularly apparent in complex or organization-level studies of healthcare systems, in unpredictable or emergency settings, and when examining factors that cannot be assigned for either ethical or practical reasons, such as patient demographics, genotypes, social factors, or disease status. A further area for caution with regard to experimental studies stems from the fact that they take place in tightly controlled clinical settings. An intervention tested by experts in tertiary-care academic medical centers might lack generalizability (i.e., not be applicable to general clinical practice or the general patient population), if the trial setting differs significantly from conditions in actual practice (Table 10–2).