Mussen et al. (2007a, 2009), in the course of their published works for developing a systematic approach to decision-making during the assessment of medicines, reviewed benefit and risk criteria through identifying these from the ICH’s Common Technical Document (CTD), EMA’s European Product Assessment Report (EPAR), and US FDA’s Medical Review. The identified criteria were subsequently verified through a survey and refined in a workshop conducted by CMR (CMR 2008) and produced recommendations for a future framework. The following efficacy parameters should be included in a benefit–risk framework:

The framework should also include the following safety parameters:

In constructing a benefit–risk balance, Mussen et al. (2009) recommended the following parameters as part of the framework:

In a review of the benefit–risk assessment models, Mussen et al. (2009) reviewed three general models, namely, “principle of threes” (Edwards et al. 1996), evidence-based model (Beckmann 1999), and Transparent Uniform Risk–Benefit Overview (TURBO) (CIOMS 1998). They were found unable to balance the benefits and risks and did not meet his criteria for a framework to assess benefits and risks. These models did not define clearly the type, quality, and relative importance of the data required. The models were simple, could not account for different attributing factors, and were not validated in practice. However, these models would collate the thoughts and considerations of the assessment and hence contribute to decision-making. Mussen et al. proceeded to develop a new framework which would function as a model for decision analysis. The MCDA (Belton and Stewart 2001) formed the foundation of this framework, as it allowed the balancing of multiple criteria, namely, the different benefits and risks of treatment with the medicine being assessed. This is a process described in the multi-criteria analysis (MCA) manual (Dodgson et al. 2009) which aimed at exploring the individual contributing aspects of the decision-making process before collating the outcomes to form the basis of the decision. There are three key phases of the MCDA process. The problem is first identified and structured, secondly the decision maker’s preferences are taken into account, and lastly, action plans are developed.

A final seven-step framework based on the MCDA principles was eventually developed (CMR 2010). The assessment of the benefit–risk balance was recommended to be carried out as follows:

A framework uses a set of underlying principles to provide an overarching structure in which essential processes can be carried out to achieve its objectives. Therefore, despite the use of values and weights, both the MCDA and the above seven-step approach should be considered as frameworks rather than quantitative methodologies, in recognition of the underlying MCA principles described above.

Part of the framework development involved participants in two CMR workshops (CMR 2004, 2005) who applied the framework in two clinical settings. The first involved the use of a new recombinant necrosis factor receptor inhibitor compared against methotrexate in managing rheumatoid arthritis and the other a hypothetical drug with cardiovascular safety concerns for treating schizophrenia. One utility of the framework was the provision of a platform for structured conversation and decision conferencing, which allowed an agreement despite a divergence in the opinions of the data. In addition, the workshops demonstrated that use of values and weights is required to provide a complete judgment on the benefits and risks. The framework was also applied to various other clinical scenarios (Mussen et al. 2007b). The final conceptual framework was adopted by CIRS (2009, 2010) and further refined through future workshops.

As part of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Risk–Benefit Management Working Group, Guo et al. (2010) conducted a literature review on quantitative methodologies for the assessment of benefits and risks of medicines. The search was not limited to a single stakeholder’s perspective and thus included tools used by regulatory agencies, pharmaceutical companies, and academia. They identified and reviewed 12 quantitative benefit–risk assessment models, which included the Quality-Adjusted Time Without Symptoms and Toxicity (Q-TWiST) (Gelber et al. 1993; Cole et al. 2004), number needed to treat (NNT)/number needed to harm (NNH) (Holden et al. 2003a, b; Laupacis et al. 1988; Cook and Sackett 1995), incremental net health benefit (INHB) (Garrison et al. 2007; Lynd et al. 2010), probabilistic simulation method (PSM) and Monte Carlo simulation (Lynd and O’Brien 2004; Shaffer and Watterberg 2006), multi-criteria decision analysis (MCDA), and stated preference method (SPM) (Ryan et al. 1998; Gan et al. 2004). Some models like the NNT used subjective weighting and allow a non-statistical or qualitative assessment, and others like the MCDA and SPM were useful in allowing joint assessment of both benefits and risks. Simple methods like the NNT and NNH are widely used, but it could not account for the quantum or value of the benefits and harms or allow the contribution of several relevant benefits and harms into the same context for decision-making. In addition, MCDA was found to be capable of handling missing data and uncertainties through use of relevant modeling tools and application of weights, as well as exploring the robustness of the outcomes through sensitivity analyses. While MCDA could account for the various factors contributing to the decision-making, it is nonetheless a relatively new and intensive tool that may be limited to more complex evaluations. The SPM is a theoretical tool that could incorporate patients’ preferences and the evaluation of benefit–risk trade-offs. This method would require the collection of patients’ treatment preferences, for which the current best practice to achieve this is still being developed. However, the SPM may be considered by healthcare professionals as it involves the opinions of the patients. Overall, it appeared that the reviewed methodologies were not adopted by the agencies and companies and were primarily for research purposes. Guo et al. (2010) concluded that some of these methodologies would be helpful to lessen concerns over the subjective component of assessment and provide the required transparency, but all have their own set of limitations. None was found to be able to function across all scenarios, and it was recommended that various tools be used to appropriately profile the benefit–risk balance. Due to the limited published information for net clinical benefit analysis, the principle of threes, and the net-benefit-adjusted-for-utility analysis, these methods were not reviewed.

As expectations of stakeholders change with the rapid advancement of science, regulatory agencies make plans to adapt and meet these changing needs. In EMA’s roadmap to 2015, they identified one of the strategic areas to be facilitating the access of medicines through reinforcing the benefit–risk balance assessment model, to be achieved through a set of priority activities (EMA 2011b). These included looking at appropriate quantitative tools, improving the quality and consistency of the outcomes, reviewing the EPARs to improve communication of benefit–risk decisions to stakeholders, and increasing the involvement of patients, academia, and healthcare professionals in the assessment of medicines to ensure their views are taken into consideration. A CHMP working group was formed in 2006 to look into methods to improve the transparency, consistency, and communication of benefit–risk assessment. A preliminary review of NNT/NNH, “principles of three,” Transparent Uniform Risk–Benefit Overview (TURBO), and MCDA was conducted, and the advantages and limitations of each were discussed. In their report, they emphasized that qualitative evaluation and expert judgment are not to be replaced by quantitative benefit–risk assessment. They recommended that a model for benefit–risk assessment should be structured and, of a qualitative approach, be able to describe explicitly the importance of benefits and risks in the context of the decision and the impact of the uncertainties on the benefit–risk assessment (EMA 2007). This led to the reflection paper for benefit–risk assessment of medicines as mentioned above and also the benefit–risk methodology project.

The benefit–risk methodology project was aimed at looking at tools and processes that provide aid to regulatory decision-making, training of assessors, and communicating benefit–risk decisions to stakeholders (EMA 2009), through a series of five work packages. The first work package (EMA 2011c) was to describe the practices of benefit–risk assessment within the EU for the centralized procedure. The key findings steered the movement of the remaining work packages, and these findings appeared to be reflective of the global environment. Among the key findings were:

In addition, the EMA produced a set of five criteria to verify a model’s applicability for benefit–risk assessment. These include logical soundness, comprehensiveness, acceptability of results, practicality, and generativeness.

As part of their benefit–risk methodology project, twenty-one approaches were reviewed, including three qualitative frameworks (BRAT, CMR framework, and US FDA’s benefit–risk framework) and 18 quantitative models in the second work package (EMA 2010). This was conducted with the above five criteria for a benefit–risk assessment model. In response to the observation in the first work package, they attempted to redefine benefits as favorable effects, harms or risks as unfavorable effects, and uncertainties as variations, bias, flaws, and deficiencies of the above types of effects. With regard to the qualitative frameworks, these were still under development at the time of the review, and hence limited comments were made. It was highlighted, however, that the uncertainties of benefits and risks, being of concern to regulators, should be addressed by these frameworks. The quantitative approaches were reviewed according to four broad categories based on their functions, namely, simulation, models, statistics, and measurements. Some of the approaches reviewed included the Markov processes (Sonnenberg and Beck 1993), TURBO, principles of three, QALYs/disability-adjusted life years (DALYs), Kaplan–Meier estimators (Kaplan and Meier 1958), and conjoint analysis (Johnson 2006). They concluded that four approaches, namely, the qualitative framework, MCDA, Bayesian statistics (O’Hagan et al. 2006; Ashby and Smith 2000), and decision trees (Goodwin and Wright 2009; Stonebraker 2002), would be useful to regulators and can comprehensively quantify a benefit–risk balance. A qualitative framework would be required to support any quantitative model and may be used for simple decision-making. Again, it was recommended that a combination of tools would be useful in selected situations involving magnitude, seriousness, and uncertainty of the effects. With the findings and understanding of the potential of the MCDA in this area, EMA proposed their own benefit–risk framework which consists of eight steps, the PrOACT-URL (Table 1.1). This is meant to be a flexible framework that can accommodate the various scientific methodologies for assessing benefits and risks, as well as a graphical representation of the outcomes of assessment.

The PrOACT-URL was subsequently applied to the third and fourth work packages. In the third work package (EMA 2011d), the framework guided the review of selected quantitative approaches conducted retrospectively using the European Public Assessment Reports (EPAR). The products reviewed were Acomplia^® (rimonabant), Cimzia^® (certolizumab), Sutent^® (sunitinib), and Tykerb^® (lapatinib) using a combination of MCDA, probabilistic simulation (PSM), Markov model, and decision tree. The use of the framework and the quantitative approaches allowed for different perspectives to be tested, reviewed the impact of uncertainties, as well as provided a structure to the review and communicated explicitly the objectives and trade-offs. However, this current method would be labor intensive and require the availability of suitable software to conduct the various analyses. Moreover, justifications for clinical judgment were not accounted for as the outcomes were to be quantified.

The ability of the PrOACT-URL to accommodate a quantitative aspect of benefit–risk assessment shown in this work package was reported and published by Phillips (Phillips et al. 2011). The fourth work package (EMA 2012) continued to support the findings in the third work package, the use of PrOACT-URL framework, and the value of graphical displays. It was recommended that the effects table be used for simpler cases and a full MCDA approach be employed for contentious cases. The last work package would be the development of training materials which have not been published at the time of this research. On top of the work to identify benefit–risk methodologies, EMA has also extended its transparency movement to include publication and public access to clinical trial data (EMA 2013b).

Since 2009 the US FDA has taken initiatives to explore systematic approaches to assess and communicate benefits and risks, in tandem with the efforts taken at the EU. The initiatives included the development of a framework to characterize and provide a structure for the benefit–risk assessment already existing in their decision-making processes, as well as communicate the reasoning behind the decision to all stakeholders (FDA 2012a). This led to the current five-step benefit–risk framework which was put together after a pilot project in 2012. The five steps are related to the five key areas to be discussed in the assessment of the medicine, namely, the analysis of the condition, the medical need for the product, clinical benefit, risk, and risk management (FDA 2013a). The strength of the evidence and its uncertainties would be considered during the assessment, with the reasons provided for the conclusion of each of the five areas. The outcomes of these five areas would then be cumulatively discussed, leading to the overall benefit–risk conclusion. The framework would also look into current treatment options, a summary of the submitted evidence for the benefits and risks and risk management plans. With the development of this initial framework, the US FDA embarked on the 5-year plan, starting 2013 till 2017, for a structured approach to benefit–risk assessment, which was part of the larger PDUFA V program. During this period they will further refine the framework and how this might be worked into their current clinical reviews to facilitate communication. Mullin of the US FDA, during a workshop conducted by CIRS (2011), commented that this structured framework had the potential to improve the predictability and consistency of decision-making as it is capable of clearly outlining both the available evidence and the uncertainties. It would also articulate the consideration and clinical judgment taken for the benefit–risk decision and hence improve the transparency of the decision-making process.

The US FDA acknowledged that the existing programs to facilitate patient representation may be inadequate, and thus they are committed to a new initiative, Patient-Focused Drug Development. This aims to obtain the patients’ perspective on the medical condition and the currently available therapies for a set of disease areas and runs till 2017. For each disease area, FDA conducts a public meeting and invites participation from FDA staff, the relevant patient advocates, and other interested stakeholders. Diseases covered thus far include chronic fatigue syndrome and myalgic encephalomyelitis (FDA 2013b), human immunodeficiency virus (HIV) (FDA 2013c), lung cancer (FDA 2013d), and narcolepsy (FDA 2013e). Other diseases planned for 2014 and 2015 include fibromyalgia and sickle cell disease. The US FDA has also published its own user’s guide on communicating benefits and risks (FDA 2011), which provides the expectations and standards of communicating risks.

In the MHRA’s corporate plan for 2013–2018 (MHRA 2013a), it was indicated that benefit–risk decisions should be made more informed by the experiences and perspectives of patients and views from other stakeholders. This is to be achieved through initiatives like more stakeholder partnerships to increase the understanding of benefits and risks of medicines and a better representation of patient and public views in regulatory decisions.

Through their new initiatives for the next 3 years, TGA will be focusing on increasing transparency and engaging stakeholders with a new framework for communications which is committed to relaying the benefits versus risks approach in their regulation of medicines (TGA 2013). This is to be achieved through information that is easily understood by patients and consumers and received and shared by healthcare professionals. TGA aims to provide accessible, clear, and consistent relevant information through various multimedia platforms. In addition, consumers would be consulted for the labeling changes. The stakeholder engagement is also extended to the healthcare professionals, in improving the awareness and accessibility to relevant information.

To a similar extent, the pharmaceutical industry has been also taking an initiative to address the need for an improved benefit–risk assessment by developing a structured, systematic, and transparent framework. Led by the Pharmaceutical Research and Manufacturers of America (PhRMA), the Benefit–Risk Action Team (BRAT) Framework sought to incorporate all relevant aspects of benefits and risks and focused on both qualitative and quantitative analyses, for the purpose of communication between the companies and regulatory agencies. The framework aimed to advance the reproducibility, transparency, and communication of the basis of the benefit–risk decisions (Coplan et al. 2011). This six-step framework (Table 1.2) is a flexible structure which allows the use of appropriate scientific tools to analyze the outcomes.

In a workshop organized by CIRS (2011), Hughes from Pfizer reviewed the steps of the BRAT framework and the history of its development. The process of BRAT framework starts with defining the decision context (including the formulation, indication, patient population, comparators, and decision perspective). Next, the benefit and risk outcomes are identified and selected, followed by the creation of an initial value tree which determines the preliminary set of outcome measures. In step three, source data are extracted to support outcome measures and input into summary tables. The framework is then customized and the value tree reexamined and revised to incorporate any additional clinical context. In step five, the outcome is assessed for its importance, with informal or formal weighting methodologies being employed to determine the relative importance of all outcomes. Finally, the key measures and data are summarized in a visual format to aid the interpretation and decision, information gaps are filled in, and sensitivity analyses are conducted.