, Sam Salek2 and Stuart Walker3
(1)
Centre of Regulatory Excellence, Duke-NUS Graduate Medical School, Singapore, Singapore
(2)
Department of Pharmacy, University of Hertfordshire, Hatfield, UK
(3)
Centre for Innovation in Regulatory Sciences, London, UK
Introduction
The current climate in regulatory science seeks transparency of decision-making and communication to stakeholders for accountability. The results from Chap. 3 showed that both regulatory agencies and pharmaceutical companies believe that a benefit–risk framework would enhance the quality (transparency and consistency) of decision-making; provide documentation for a systematic, structured discussion; and act as a tool for communication. A tool was thus developed (Chap. 4) with inputs from the Consortium (consisting of TGA, Health Canada, Swissmedic, and HSA), and the resulting universal Benefit–Risk (BR) Template was designed to enhance communication and documentation of benefit–risk decisions. This study aims to review the potential value of the BR Template for regulatory agencies.
Objectives
The objectives are to:
Examine the value of the BR Template for documenting the benefit–risk assessment decision of new active substances during the review process
Evaluate the BR Template as a tool to communicate the benefit–risk decision to other stakeholders in a systematic, structured manner
Determine if the BR Summary section of the BR Template is adequate as a stand-alone tool to communicate a benefit–risk decision to stakeholders
Methods
Three similar-sized regulatory agencies agreed to participate in this prospective study which was conducted as non-comparative evaluation. The study package, consisting of the BR Template (which included the Benefit–Risk Summary section) and User Manual described in Chap. 4, were sent to the three agencies. The reviewers in the respective agencies selected a product undergoing active evaluation and completed their own assessment report as well as the BR Template. Following this process, the reviewers were sent a study evaluation tool (Fig. 5.1), which they completed and returned.
Fig. 5.1
The study evaluation tool
The study evaluation tool was developed as a questionnaire consisting of 56 questions divided into four sections, namely, user-friendliness, documentation, applicability, and general comments. There were three systems of rating:
Excellent, good, fair, and poor (comments to be provided for ratings of “fair” and “poor”)
Fit for purpose, fit for purpose with modifications required, and not fit for purpose (comments to be provided for the latter two choices)
Yes and no (comments to be provided for rating “no”)
Most of the questions had an open field for comments, allowing the participants to provide any issues of concern or relevant points that were not addressed by the questionnaire. The questionnaires were sent via e-mail directly to the participants. Completed responses were received via e-mail, as instructed to the participants. All responses were collated into a single group, and outcomes were presented according to their respective sections in the study evaluation tool. All data were expressed as direct ratings provided by the responders. Free-text comments were collated and presented in appropriate categories.
This was designed as an exploratory study, and the outcomes were interpreted to provide qualitative inferences relating to the objectives. No statistical analyses were planned or conducted.
Results
The outcomes will be presented in four parts:
Part I: User-friendliness of the BR Template
Part II: Appropriateness (fit for purpose) of the documentation
Part III: Applicability of the BR Template
Part IV: Usefulness of the BR Template
None of the agencies used visualizations, and hence no outcomes were documented for these items in the survey tools.
Part I: User-Friendliness of the BR Template
The BR Template has three features that assist the user in locating selected pages within the document, namely, the tabs at the top of each page, the “Go to page” button, and the page thumbnails (Fig. 5.2).
Fig. 5.2
Navigation functions of the BR Template
The thumbnails were not used, while the tabs and “Go to page” buttons were rated either good or excellent (Table 5.1). The agencies suggested that the use of bookmarks for the sections and subsections would be preferable, as well as a search function for identifying keywords within the document.
Table 5.1
Practicality of the navigation functions
Agency | Tabs at top of page | “Go to page” button | Page thumbnails |
---|---|---|---|
A | Excellent | Excellent | Did not use |
B | Good | Good | Did not use |
C | Good | Good | Did not use |
In addition to navigation features, the BR Template incorporates four functions to print, e-mail, view the form (Fig. 5.3), and auto-populate information for fields requiring the same inputs (Fig. 5.4).
Fig. 5.3
Document support functions of the BR Template
Fig. 5.4
Auto-populate function of the BR Template
Conclusion on the usefulness of the print, e-mail, and view functions were not provided as Agency A did not use the former two functions, Agency B experienced a technical issue that prevented them from getting back to the document after using these three functions, while Agency C rated these support functions as good. However, the auto-population function was considered useful by all, being rated as good or fair (Table 5.2).
Table 5.2
Usefulness of the document support functions
Agency | Print function | E-mail function | View full form function | Auto-populate function |
---|---|---|---|---|
A | Did not use | Did not use | Excellent | Good |
B | Poor | Poor | Poor | Fair |
C | Good | Good | Good | Good |
The User Manual was provided as a guide to help the reviewer in completing the BR Template and included a glossary of commonly used terms. Agencies A and C rated the manual as good or fair in terms of clarity, comprehensiveness, and applicability (Table 5.3). Overall, the agencies believed more details are needed, e.g., case studies and examples to improve the usefulness of the User Manual. Agency B would like to have more guidance regarding the intention of the BR Template, the level of details of the outcomes, and the type of information required. Agency C commented on the need for examples to show how weighting and valuing may be carried out as this concept is new to the agency.
Table 5.3
Appropriateness of the User Manual
Agency | Clarity of instructions | Comprehensiveness | Applicability |
---|---|---|---|
A | Good | Fair | Good |
B | Poor | Poor | (Not reported) |
C | Good | Good | Good |
Other comments received on enhancing the technical aspects of the template include:
Allow changes in fonts (e.g., size, underlining, italicizing), use of bulleted listings within text boxes, and use of the tab key within a cell in the tables
Allow for the use of the tools for commenting and marking up (highlighting and crossing out functions) in Adobe Acrobat Professional as these would be useful for supervisors or managers recommending revisions to the document
Ensure that the text copied and pasted from a Word document retains the original formatting (underlining, italicizing, symbols)
Part II: Appropriateness (Fit for Purpose) of the Documentation
The BR Template incorporates five conclusions that are considered important in making a benefit–risk decision, namely, the background (decision context), quality, nonclinical, human pharmacology, and clinical conclusions. Of the five, the agencies believed the clinical conclusion is fit for this purpose (Table 5.4). For the remaining four, the template could allow for more details as the actual benefit–risk assessment was carried out in much greater depth and the sections may not accommodate such a level of information.
Table 5.4
Documentation of relevant information supporting the benefit–risk decision
Agency | Background information | Quality conclusion | Nonclinical conclusion | Human pharmacology conclusion | Clinical conclusion |
---|---|---|---|---|---|
A | Fit for purpose | Modifications required | Modifications required | Modifications required | Fit for purpose |
B | Fit for purpose | Modifications required | Modifications required | Modifications required | Fit for purpose |
C | Modifications required | Fit for purpose | Fit for purpose | Fit for purpose | Fit for purpose |
Agency B commented that these were the only sections to discuss the contributions from quality, nonclinical, and human pharmacology in the entire template, whereas the rest of the template is dedicated to clinical benefits and risks. However, Agency B believed that if the intention of the BR Template was to feature only a high-level summary of the significant findings, then it would suffice. It was mentioned that a considerable amount of evaluation was conducted for those aspects for a new active substance and that this section would not be able to accommodate these findings. Without allowing the reviewer to provide details on the relevant studies, it would be difficult to explain the relevance of the reported issues and concerns. For completeness, Agency A recommended the inclusion of subheadings for pharmacokinetics, pharmacodynamics, and drug interactions to further guide the reviewer. Agency C preferred the background information to allow a discussion on related applications and products that may contribute to decision-making.
It should be noted that the BR Template was designed to present and communicate only the significant findings that would affect the benefit–risk decision and that the corresponding details would be expected to be available from the original assessment report for the product.
The template was seen by the agencies as being able to document benefits and risks identified by sponsors and the reasons for including or excluding them (Table 5.5).
Table 5.5
Documentation of benefits and risks
Benefits | Risks | |||||
---|---|---|---|---|---|---|
Agency | Reasons for inclusion or exclusion of all benefits | Relevant benefits as identified by sponsor | Selected list of benefits to be included in the benefit–risk assessment | Reasons for inclusion or exclusion of all risks | Relevant risks as identified by sponsor | Selected list of risks to be included in the benefit–risk assessment |
A | Fit for purpose | Fit for purpose | Fit for purpose
Stay updated, free articles. Join our Telegram channelFull access? Get Clinical TreeGet Clinical Tree app for offline access |