The evaluation of medicines has traditionally been conducted as separate assessments of efficacy and safety, in which a regulatory decision is based on proven efficacy supported by clinical studies matched with an acceptable safety profile. The trend in the assessment of benefits and risks is currently towards a holistic discussion of the benefits, risks, and overall benefit–risk balance. This allows for a clear view of the relationship between the benefits identified and the risks potentially expected from the treatment and how the eventual balance is achieved to justify a regulatory decision for the medicine. Over the years, major regulatory agencies and pharmaceutical companies have indeed made progress in improving the frameworks for the assessment of benefit–risk balances, but these are largely based on individual efforts due to the lack of a common universal framework. This suggests the beginning of a challenge to implement a universal framework, as these stakeholders are striving to develop a framework specific to their own jurisdictions and suited to their purposes. Without a universal framework, the current lack of consistency in making regulatory decisions and transparency of communication may be further perpetuated, leading to misunderstandings among the stakeholders and the potential unavailability of important medicines in some jurisdictions.

In reviewing the current environment on the use of benefit–risk assessment frameworks, it was found that both agencies and companies were using either qualitative or semiquantitative systems. Among the companies, different approaches may be employed for product development and during regulatory submission. The majority of organizations who are currently using semiquantitative systems were not satisfied, and many expressed concerns about adopting a methodology that did not match the requirements of the other stakeholders, given that there is no one framework that is recognized by all. It was hoped that a universal framework would be structured, standardized, and applied throughout product development to submission for registration. Indeed, when the reason was sought as to why semiquantitative or quantitative systems were not used, the majority indicated the lack of a scientifically, validated universal framework.

A disparity was observed in the opinions of the current methodologies used for the assessment of benefits and risks. While the agencies and companies considered Bayesian statistics and MCDA as useful and relevant, these were not the main tools they were utilizing, namely, the qualitative approach, NNT/NNH, and evidence-based benefit–risk model. In the assessment of benefits and risks, most agencies and companies frequently assigned values to these parameters but not the assignment of weights or relative importance. It could be that weighting was carried out implicitly and considered during the evaluation of the overall benefit–risk balance. In communicating benefit–risk decisions, none of the agencies had used visualization tools, while the companies had such tools for internal communication and infrequently for health professionals and patients. Therefore, this lack of a universal framework could have led to the inconsistent approaches in the assessment and communication of benefits and risks across the agencies, companies, and within these organizations themselves.

The agencies and companies believed that a benefit–risk framework should be used for the life cycle of a medicine. This is a consistent finding as confirmed by an earlier workshop conducted by CIRS (CMR 2008) for various stakeholders including the agencies and companies. Such a framework should enhance the quality of communication and enable the assessment of benefit–risk management plans. In developing a framework for the future, it would be useful to have a coordinating group to guide its direction and application and to involve relevant stakeholders. A framework should confer the advantages of an appropriate documentation and the enhancement of communication.

Seven factors were identified which both agencies and companies agree would be relevant to reviewing a framework. These included logical soundness, comprehensiveness, acceptability of results, practicality, specificity and sensitivity, presentation (visualization), and scope. The first four factors are similar to those used in the first and second work packages by the EMA in their benefit–risk methodology project (EMA 2010, 2011b) with the last factor generativeness not being used in this research. However, in order to reflect the scientific robustness that is critical for the assessment of benefits and risks, statistical concepts of specificity and sensitivity were added to the list. As it was then known that the graphical presentation of results would help communication (CMR 2010; CIRS 2011), visualization was added as a factor to review if the framework would support this upcoming communication strategy. Lastly, to ensure that the benefit–risk assessment framework would be applicable to all scenarios and for the entire product life cycle, scope was added as the final factor in reviewing such frameworks. In the review of benefit–risk assessment methodologies, the IMI PROTECT (2011b) had referenced the EMA criteria in the aforementioned work packages and had put the required emphasis on visual presentation. Therefore, it is believed that this new set of seven factors for reviewing a benefit–risk assessment framework not only encompassed those used for two other major projects, but is also a reflection of the contemporary ideals among the agencies and companies for such frameworks.

The lack of an accepted and validated framework was a significant barrier for agencies and companies. Additional barriers included the absence of a consensus on the needs of the stakeholders and direction of the purpose and utility of a framework, as well as the lack of acceptance by the major regulatory agencies. In addition, the universal framework should be comprehensible, be easy to understand and use, be flexible, and accommodate the different scientific methods of assessing benefits and risks. The outcomes of EMA’s work packages (2011d, 2012) and IMI PROTECT (2011b), both of which utilized the PrOACT-URL framework, confirmed that a qualitative and flexible framework would be required to achieve the above.

The requirements for a universal framework ought to be sought from the stakeholders whose inputs will directly affect the benefit–risk decision and the final regulatory outcome. These have been identified as the regulatory agencies, pharmaceutical companies, physicians, HTA agencies, and patients. While this study obtained only the views of the agencies and the companies, there are ongoing studies to assess the contribution of the other stakeholders to the decision-making process. These include patients’ involvement (EMA 2011e; FDA 2013a, b, c, d, e) and health technology assessment (HTA) agencies (EMA 2013c). While the study was conducted with major and medium-sized agencies and companies, it may not represent the entire regulatory environment.

While there is no common framework for major regulatory agencies and companies, some do have their own frameworks for the assessment of benefits and risks. The EMA’s eight-step PrOACT-URL (EMA 2010) (also used by the IMI PROTECT), the US FDA’s five-step benefit–risk framework (FDA 2013a), the BRAT six-step framework (Coplan et al. 2011), the CIRS’ seven-step framework (CMR 2010), and the Novo Nordisk’s Benefit–Risk Assessment in New and Old Drugs (BRAIN; CMR 2010) were compared. With the exception of the US FDA’s framework, the rest were based on the principles of MCDA, which was earlier confirmed in the course of this research as a useful and relevant methodology. The eventual eight-step framework which has been developed in this research includes the defining of the decision context, building the value tree, refining the value tree, evaluating the options, assigning relative importance of the benefits and risks, evaluating the uncertainties, presenting the outcomes in a graphical manner, and finally applying expert judgment and communicating the decision. Across the frameworks compared and the final eight-step framework, there are four common requirements considered fundamental to assessment, namely, framing the decision, identifying the benefits and risks, assessing the benefits and risks, and lastly interpreting and recommending a decision. However, the differences among the frameworks lie in the activities conducted to fulfill these requirements. Apart from the framework used by the US FDA, the rest advocated the use of weighting and valuing and the use of either an effects table (as in the case of PrOACT-URL) or other appropriate visualization tools. This observation may be related to the fact that these frameworks follow the principles of MCDA while the US FDA was a unique qualitative framework. However, it should be noted that while the US FDA framework did not explicitly advocate the use of the weighting, valuing, and visualization, it appears that it would be able to accommodate such activities.

Between the final eight-step universal framework (Fig. 8.1) and the EMA’s PrOACT-URL, the latter had more emphasis on the discussion of risk tolerance and the consistency of decisions, i.e., linked decisions. While this may be discussed as part of the uncertainties or implied with the use of weightings under the new framework, there may be value in soliciting directly the views on the risks the evaluator is willing to accept and how well the basis of the recommended decision aligns to previous ones made for similar scenarios. Among the BRAT framework, the BRAIN, and the new eight-step framework, it appears the former two encourage the use of quantitative tools to provide a metric representation of the effects and scores, while the new framework may accommodate a qualitative discussion instead. Overall, the eight-step framework has struck a balance between more prescriptive frameworks requiring some quantitative outcomes and those which otherwise are too general in guiding the assessment of benefits and risks. As such, the universal utility of the final eight-step benefit–risk assessment framework is supported by the above comparison, and it was developed with elements common to the other existing frameworks and used by the two major regulatory agencies and pharmaceutical companies. Hence, if all the processes of the final eight-step framework are carried out, the outcomes are expected to complete and fulfill the requirements of the other existing frameworks. This universal benefit–risk assessment framework is expected to enhance the objectivity and transparency of the decision-making process by providing a structured approach that could be adopted by regulatory agencies and pharmaceutical companies.

In order to utilize the steps of the framework, a system for documentation of the assessment outcomes and effective communication must be in place. In the absence of the principles and methodologies for benefit–risk assessment from other major regulatory authorities, the published reflection paper by EMA (2008) was used as a reference for the development of the tool to document and communicate the outcomes of assessment and the basis of the decision in a consistent and transparent manner. The developmental version of the Benefit–Risk (BR) Template was tested for functionality by the Consortium, consisting of TGA, Health Canada, Swissmedic, and HSA. This was carried as a retrospective feasibility study. The final version consists of two sections, namely, the “BR Template” and “Benefit–Risk Summary.” The template was then reviewed against the core elements of the universal framework, namely, framing the decision, identifying the benefits and risks, assessing benefits and risks, interpretation, and recommendations. In relating to the eight-step universal framework, the BR Template fully supports these requirements. To facilitate the use of the framework and template, a user manual was developed. This consisted of two sections, namely, a glossary and instructions for completing the template. Consequently, the Benefit–Risk Assessment Support System (BRASS) was developed and consisted of the eight-step benefit–risk assessment framework, the Benefit–Risk Template, and the User Manual.

It could be argued that the evaluation of BRASS by the four agencies of the Consortium would not represent the opinions of all stakeholders and thus undermines its utility as a universal framework. However, it should be noted that the universal framework was reflective of the current principles used by the major reference regulatory authorities and companies. As justified above and also at a workshop (CIRS 2012b) attended by senior decision makers of agencies and companies, there was an agreement that the final eight-step universal framework covered the essential elements in other existing frameworks.

Subsequently, a prospective study was therefore conducted with three agencies, namely, TGA, Health Canada, and HSA, to review the potential value of the BR Template and User Manual. In order to achieve consistency in evaluating the responses of the agencies, a study evaluation tool was developed which included four sections, namely, user-friendliness, documentation, applicability, and general comments. Navigation functions were found sufficient to guide the user in locating different sections of the template. The User Manual too was found to be adequate though more specific details and examples of use could be provided. Overall, the BR Template and User Manual were found to be user-friendly.

The BR Template was studied for its appropriateness in documenting relevant information supporting the benefit–risk decision, benefits and risks, weights and values, study outcomes, safety information, and overall conclusion. For information to support the decision, the template was found satisfactory in documenting the various relevant conclusions, with proposed modifications to allow greater details to be presented. The template was found more acceptable in documenting benefits than risks, and consequently, there were recommendations to provide greater clarity in the risk definitions and how these are to be selected for the benefit–risk assessment. Divergent views were obtained for the template’s use in documenting weights and values. However, this observation is very probably related to the current state of knowledge in applying weights and values for the assessment of benefits and risks. It is expected that in the future when more assessors are better acquainted with the concepts and application of weights and values, the opinions of the use of the template for this aspect would be better reflected.

In reviewing its applicability, the BR Template was assessed on its ability to contribute to decision-making, consistency in standard of assessment, transparency, communication to stakeholders, and consistency of decisions between agencies. With the exception of one agency, the above functions were deemed to be fulfilled by the template. The main concern of Health Canada, who disagreed, was that the template was neither able to capture critical thinking nor other significant contributing factors such as the additional analyses that the reviewers requested from sponsors. If clarification is provided on the existing availability of appropriate sections in the BR Template to discuss these other contributing factors, it is believed that all three agencies would agree on the template’s applicability. All the agencies agreed that the BR Template can ensure consistency in decision-making through improving regulatory memory, acting as an audit tool, and contributing to post-marketing activities. The outcomes demonstrated the value of the template and User Manual and its potential use in documenting and communicating benefit–risk decisions. Overall, all three agencies found the template and User Manual fit for purpose with amendments. Importantly, all three agencies found the BR Template useful in documenting the uncertainties relevant to the identified benefits and risks. The potential and practicality of the BR Template in documenting, reporting, and decision conferencing of benefit–risk decisions was therefore demonstrated. It would be of interest to evaluate the use of the template with an established mature agency, in particular the EMA, since their guidance was the basis of the template (EMA 2008). This would help to convince stakeholders that the BR Template is applicable across regulatory agencies of all levels of establishment and maturity.

To assess the template’s ability to act as a suitable tool for communication, the three agencies were asked if they are willing the share the completed BR Template and the summary section with stakeholders. Though it appeared there are reservations in sharing the entire completed BR Template, this view was due to concerns over confidentiality and memorandums of understanding with the stakeholders and not the functionality of the template. One agency, HSA, however felt that information for the public and media should be amended as the BR Template may contain information that is too technical for their understanding. Regarding the more succinct BR Summary section, the agencies would consider sharing this with stakeholders provided more details can be provided in this section and if the information is amended to tailor to the level of understanding for patients and media. It is noted that both TGA and Health Canada already provide public available reports and would thus be comfortable with the inclusion of more in-depth contents. HSA on the other hand is establishing itself as a maturing agency and may be more conservative in making available the information relating to their decisions. Nonetheless, the BR Template and the BR Summary Template allows amendments and can be tailored to suit each agency’s needs.

Given the different regulatory capacities and maturation of the regulatory agencies across the world, some are leading this field, while others, like those from the emerging markets, would likely leverage on the decisions of the major regulatory agencies. Therefore, it is important that the basis of the decisions of the major agencies is effectively communicated to the rest of the stakeholders, which would include the agencies from the emerging markets. Although there may be publicly available assessment reports, these may contain a significant amount of information to review that would require both time and scientific capabilities that are not available. The potential use of the BR Summary section as a stand-alone tool for documenting and communicating benefit–risk decisions was thus identified, in the hope that it may aid the emerging markets. Consequently, this section was extracted and transformed into a stand-alone tool, now known as the BR Summary Template.