19

Signals and Signaling in the Context of Risk Management

Enormous efforts are made, in terms of people, time, cost, and technology, to collect AE data in companies, governments, and elsewhere. The collection of vast amounts of data is meaningless in and of itself. It is only when these data are organized and analyzed for new safety issues (which are then acted on in the context of risk management) that the true value of this effort becomes apparent. The hunt for meaning is known as “signaling.”

The Uppsala Monitoring Centre defines a signal as follows:

They comment further:

A newer definition has been proposed:

Not everyone agrees that all signals based on single cases should not be pursued. Sometimes, however, rare events are picked up after a single case, and certain AEs are almost always due to drugs (e.g., Stevens-Johnson Syndrome, fixed drug reactions). Thus, a single case can be a signal. More common problems (e.g., myocardial infarctions) might not be worth pursuing as a signal if there is only a single case in a middle-aged diabetic smoker. But in a 12-year-old it is worth pursuing.

Signals may be “qualitative” (based on spontaneously reported data) or “quantitative” (based on data mining, epidemiologic data, or trial data). The signal may be a new issue never before seen with this product, or it may be the worsening or changing of a known AE or problem (e.g., a previously unaffected patient group is experiencing this problem, or the incidence has increased, or it is now fatal in those it attacks, whereas before it was not). As noted above, qualitative signals may be based on one single striking case or on a collection of cases. In addition, qualitative signals may also be based on preclinical findings; experience with other similar products in the class (“class signals”); new drug or food interactions; confusion with a product’s name, packaging, or use; counterfeiting issues; quality problems; and more. Thus, the word “signal” is being expanded.

“Signal” is primarily used to refer to marketed products, although the term is occasionally used for new issues in clinical trials. Some people use the term “potential signal” to indicate an issue with minimal data (e.g., only one case report), whereas others use the term “weak signal.”

Some signals are very difficult if not impossible to pick up. Signals with very long latencies (onset well after the drug use has ended) or which skip a generation (DES and vaginal cancer; see Chapter 6) require exceedingly astute observers or great luck to be found.

Identifying signals, however, is not enough. The signal must be further investigated by doing what is variously called a “signal workup,” a “signal inquiry,” or a “pharmacovigilance investigation.” The ultimate goal and true raison d’être of signal discovery and investigation is to determine whether the newly identified problem is indeed due to the drug, and is of sufficient severity and frequency in relation to the benefit, to require alerting physicians, nurses, pharmacists, and patients via a change in the product labeling, television, internet, social media, and other announcements or, in more severe cases, recalling of the product or stopping a clinical trial.

Signaling is not passive. It is proactive. No longer does one wait for AEs or SUSARs before acting. Rather an active signaling effort must be done throughout a product’s life cycle. The goal is to anticipate, evaluate, and minimize problems, not to react to them after the fact.

Signals are looked for in multiple ways. The oldest method is essentially passive and relies on the collection by pharmaceutical companies, government health authorities, or third-party organizations (academic centers, medical registries) of spontaneous AE reports and aggregate analyses of these reports plus any others picked up from other sources, such as solicited cases, compassionate use, surveys, etc. They are then reviewed individually and in aggregate, looking for “striking,” “unusual,” or “unexpected” AEs. Medically qualified people (physicians, nurses, pharmacists) examine large quantities of data, attempting to find the proverbial needle in the haystack, and either discuss within the organization the “potential signals” found or post them publically (see FDA’s potential signal website in Chapter 21). This technique is elegantly known as “global introspection.” It is quite time-consuming and laborious, but in the hands of astute and clever clinicians does indeed pick up major problems and still remains, in many respects, the cornerstone of signal generation and identification around the world. It obviously relies on the good will and perspicacity of the reporting physicians, nurses, pharmacists, and patients to send AE reports into the companies or health authorities (without remuneration) and on the goodwill and competence of the data analysts.

It is most sensitive when

• The signal is very unusual and rarely seen in general (e.g., aplastic anemia).
  
• The signal is rarely seen with that drug class (pulmonary fibrosis with beta-blockers, e.g., practolol).
  
• The signal is rarely seen in that cohort of patients (e.g., cataracts in young nondiabetic patients).
  
• The signal is fatal, particularly in patient groups who classically do not have high mortality rates (e.g., deaths in 20-year-olds).
  
• The signal is expected to be seen because it has been reported in other drugs in the same class (e.g., rhabdomyolysis with a new statin).
  
• The signal is expected because it is due to an exaggeration of the drug’s pharmacologic effect (e.g., syncope in patients taking an antihypertensive).
  
• The AE in question is seen almost exclusively with drugs (e.g., fixed drug reaction).
  
• The causality is crystal clear (e.g., the tablet is large and sticky and gets stuck in the oral pharynx, producing obstruction; or when immediate swelling and itching is seen at the site of a drug being injected).
  
• No other drugs, OTC products, neutraceuticals are being taken by the patient(s) in question.
  
• The drug is being taken for a short time, and there are no or few confounders.
  
• The patients are otherwise healthy and have no other medical problems beyond the one being treated with the drug in question.
  
• There is a positive rechallenge (reaction reappears upon drug reintroduction after a positive dechallenge).
  
• The AE is different from the signs, symptoms, and problems seen with the disease being treated and would not be confused with the disease itself.

It is less sensitive when

• The signal has a high background incidence in the general population (e.g., headaches, fatigue).
  
• The signal has a high background incidence in the population being treated (e.g., myocardial infarctions in middle-aged hypertensive smokers).
  
• The signal represents a worsening of the problem being treated (e.g., fialuridine’s, producing worsening and fatal hepatitis in patients being treated for hepatitis; see Chapter 52).
  
• The patients are taking multiple drugs (polypharmacy, intensive care unit).
  
• The patients have major underlying medical problems producing disease, signs, and symptoms (e.g., oncology patients).
  
• The drug is taken chronically, and many intercurrent illnesses and problems occur over time (confounders).
  
• There is a negative dechallenge (reaction continues even after stopping drug), or the drug in question is not stopped in the patient and the AE disappears by itself anyway.

This is a technique that has been on-again, off-again. It has been in favor and out of favor. It basically relies on a statistical calculation of reporting frequency in the current period versus a previous period (e.g., 1Q2011 vs. 1Q2010) to see if there is an increase in reporting of a particular AE. The technique is easy to do, and can be computerized and run for all reported AEs for a drug. It has, in practice, turned out to be not very useful. Although signals are, by definition, generated by this process (some AEs go up [i.e., are more frequent and thus a signal], some go down, and some remain the same), they have, in general, turned out to be false alarms or meaningless, or were easily picked up by other means.

Nonetheless, it is recommended that frequency analyses be done in PSURs. See Volume 9A Section 6.3.10, “An increased reporting frequency of listed adverse reactions, including comments on whether it is believed the data reflect a meaningful change in adverse reactions occurrence.” The U.S. Food and Drug Administration (FDA) also required frequency analysis until 1997 in its NDA periodic reports, but ended that because it was found to be of little practical use. The FDA has, however, in the proposed regulations published in 2003 (“The Tome”), proposed reinstating its use. Nonetheless, some organizations still do this on a routine basis. The FDA has also recently asked that frequency analyses be done in clinical trials to see whether an SAE’s occurrence (incidence) is rising.

This term is used, sometimes somewhat pejoratively, to describe various automated or semiautomated techniques that generate signals from existing databases. These techniques use raw case report data and arrays of drug-AE combinations to calculate “expected” versus “observed” numbers or reporting rates, and use observations of excess reporting as signals. Various techniques exist, including proportional reporting rate (PRR), gamma poisson shrinker (GPS), urn-model algorithm, reporting odds ratio (ROR), Bayesian confidence propagation neural network—information component (BCPNN-IC), adjusted residual score (ARS), and others. These techniques attempt to extract signals that are not obvious using global introspection. Some feel that this is a largely futile exercise since spontaneous reports are “dirty” data with unknown and unknowable numerators and denominators and you cannot “make a silk purse out of a sow’s ear.”

However, much work is being done on making better use of “dirty” data. One example of a success is called fractional or proportional reporting rates (PRR), also know as “disproportion” reporting rates or “signals of disproportionate reporting (SDRs)”:

For each AE, the calculation of the proportion of that AE as a function of all AEs reported for a drug is calculated and compared with the proportion of that AE for all other drugs in the database.

For example, liver failure for drug X was reported 95 times out of the 1418 total AEs for drug X. For the entire AE database of all drugs (except drug X), the score of liver failure was, for example, found to be 2,243/41,540 = 0.054.

Calculation

(95/1418) /(2243/41540) = 1.24

0.067/0.054 = 1.24

So liver failure with drug X is seen with a proportion (or “score,” “statistic,” “disproportion,” or PRR) of 1.24 (that is, 24% more liver failure with drug X compared to the rest of the drugs in the database).

Is this a signal? In theory it is, since the proportion is higher than for other drugs. But at only 24%, this is somewhat small if other AEs show a proportion of 200% or 400%. If the proportion of the AE for the drug in question was the same as the proportion for the whole database, the number would be 1.00. This means that the same reporting rate for liver failure is seen with drug X and the rest of the drugs in the database. If there were proportionally fewer liver cases with drug X, the score would be <1.00. Does this mean that drug X actually protects against this AE? In theory, this is the logical extension of this line of reasoning, but it would take much more than this to think there is a therapeutic effect (of sorts) to prevent this AE.

The level at which one considers a signal to be generated could be chosen as anything >1.00, although this will probably produce many false positives. In practice, one might take a high score above, say, 2.0 or more before one starts considering these to be signals. If one does this for all 80,000 or so MedDRA terms, then one might expect about 40,000 signals (values >1.0) if the distribution were random. This is obviously not practical. Alternatively, one might look at the top 10 or 20 scores. Another technique would be to use more complicated filters such as a PRR>3 and a chi-squared test >5 and more than three cases with the drug in question.

It is also useful to look at the scores periodically to see whether a particular AE is increasing. That is, it is becoming more disproportional over time and thus may be a stronger signal.

To be useful, the database must be reasonably large (though it is hard to say how large). If additional cases are needed to expand the database, it is possible to download cases from the FDA AERS database from the FDA website, from the Health Canada safety database, the MHRA DAP reports, the EMA Eudravigilance database, or the UMC’s Vigibase (see Chapter 8). There may be significant logistical issues in uploading or manually entering cases from these databases into another database. There are many other issues than can make this technique less useful. The other drugs, patients, diseases, and characteristics of the rest of the database should be similar to that of the drug in question. An extreme example would be studying injection site reactions for drug X compared to the rest of the drugs in the database, none of which is given by injection. There would be no injection site reactions for tablets. Or more subtly, if the drug in question is given mainly to elderly diabetics, comparing it to the AE pattern of other drugs given to children would similarly not be very meaningful.

For further details, see Evans, Waller, Davis, Use of proportional reporting ratios for signal generation from spontaneous adverse drug reaction reports (Pharmacoepidemiol Drug Safety. 2001;10:483–486). A comparison of different techniques and thresholds is found in Hochberg, Hauben M, Pearson RK, et al., An evaluation of three signal-detection algorithms using a highly inclusive reference event database, Drug Safety 2009;32(6):509–525; and Deshpande, Gogolak, Weiss Smith, Data mining in drug safety: review of published threshold criteria for defining signals of disproportionate reporting (Pharm Med 2010;24(1):37–43). Various data mining techniques are also described in FDA’s 2005 Guidance on Good PV Practices (see below). Whether these or other methods will ultimately prove useful remains to be seen.

Information should be obtained, as appropriate, from sources other than spontaneous reports. Other sources include nonclinical study data, such as toxicology and pharmacology data, including animal data, the medical and scientific literature, clinical trial data (not all of which may be found in the drug safety data base—nonserious trial AEs may not be kept in drug safety’s database), external databases (FDA, UMC, etc.), product quality complaints and manufacturing deviations, regulatory authority comments in PSURs or direct communications to the company, and so on. If a Risk Management Plan (RMP) or REMS is in place, signaling should be done with this in mind.

After data have been found from all of the sources noted above (ICSRs, aggregate data, data mining, solicited cases, etc.), the results should be tabulated, reviewed, and “triaged” to determine which findings deserve further consideration now and which go into a “holding box” waiting for more data. There is no precise formula to determine which signals should be investigated rapidly and aggressively and which can sit. Some factors to consider include whether the drug is widely used, whether the signal in question is serious/severe or not, whether the patients are seriously ill, whether the problem is reversible, whether the investigation is easily done, whether the outcome of the investigation can be known in a shortish time rather than years, whether there is health authority or other external pressure (e.g., publicity, internet activity), and (probably unfortunately) monetary cost.

Organizational Team

Each organization, whether a drug company or a health authority, needs to have a team (formal usually but may be ad hoc if appropriate) to evaluate the signals. This is usually a multidisciplinary team that reviews, analyzes, and may also make recommendations on signals. It may be empowered to make decisions or it may function to deliver data and multiple action choices to more senior management personnel. Members include physicians and healthcare personnel from drug safety, epidemiology, clinical research/development, regulatory affairs, biostatistics, quality, risk management, legal (sometimes), pharmacology/toxicology (sometimes), manufacturing (sometimes), and others as needed, including external subject-matter experts. Marketers and sales personnel should not be on the team.

Signal Workup

Once the signal list has been prepared, the list needs to be prioritized for workup according to the criticality of the signals and the resources available. Of course, lack of resources is never an acceptable excuse for not working up a signal that is important to the public health. This will be an unacceptable reason with the HA (or in court!) for incomplete, inadequate, or slow signal workup, which jeopardizes public health. But realistically speaking, resources will play a role in prioritizing.

Prioritize

There are many ways to prioritize signals. Red, yellow, green is one way, or numerical priorities on a 1 to 5 scale are used. Whatever method is chosen, though, it should be documented and consistently used. Exceptions will not be well looked-upon by inspectors.

Do an initial priority assessment. Highest priority should go to drugs that are new, where the AEs are serious or severe, where there are tampering or counterfeiting issues, where the patient population is ill or apparently at high risk, where the drug is known to be toxic, where many people use the drug, and to black triange drugs (the designation in United Kingdom labeling of a new and/or dangerous drug), etc. For better or worse, other issues also enter into prioritization, ones that are less related to public health, such as politics, sales volume, need to “protect” the drug, adverse publicity, showing due diligence in tracking, and working up safety issues (e.g., FDA’s Potential Signal website, Web Resource 19-2). Conversely, drugs whose AE profile is mild and where few adverse consequences on the public health are seen or expected would have lower priority. Minor AEs of toxic drugs would probably fall somewhere in the middle of prioritization.

Although difficult to do, efficacy should also be taken into some account when deciding on initial prioritization. Drugs with minimal efficacy with potential new, severe AEs should have a higher priority. To put it another way, if the drug in question was “placebo” such that no efficacy was expected (forgetting placebo effects for the moment), then no AEs at all should be tolerated, and this drug would get a high priority for signal workup.

The CIOMS VIII Working Group suggests the following points to consider in prioritizing signals: medical significance (serious, irreversible, etc.), increasing PRR scores, an important public health impact, easily retrievable data elements, and temporal clustering. See Practical Aspects of Signal Detection in Pharmacovigilance, Report of CIOMS Working Group VIII, Geneva 2010 (see Chapter 36). This publication is a fine review of the state of the art in signaling as of 2010.

Arrange and Review

Next, the drugs in question should be arranged on a spreadsheet or put into a database. There are various ways to do this. Some suggestions are made here.

One may create an overall summary signaling spreadsheet and then a daughter spreadsheet for each drug/signal combination (e.g., one sheet for Drug X and elevated liver tests or another sheet for Drug Y and atrial arrhythmias). Cases or case series should be arranged on the sheet using a simple or augmented CIOMS II line-listing format, with “augmented” referring to adding additional data to the line listing, such as a brief narrative, clinical course, or causality (see below). Cases may be arranged by date, by seriousness, or by some other factor. Various software programs are available for useful and creative displays of the data (see below).

Next, it is often useful to do causality assessments. In many cases, this should be done again at the time of signal evaluation even if the cases have an earlier causality from the investigator, reporter, company, or patient. Note that many companies do not do causality assessments on spontaneous reports, as they are presumed to be possibly related by convention. Thus causality on these cases should be done now. Hindsight, time, and new data may change the original causality determinations. There is no uniformly accepted international classification. Choose a system and stick to it (e.g., related, possibly related, weakly related, unrelated, insufficient information/unknown).

Causality should be assigned to individual cases and to the group of cases as a whole. In a case series, no single case may be clearly due to the drug, but the weight of the evidence of the sum of the cases may strongly suggest a likely signal.

The signal should be assessed in terms of

• Magnitude and seriousness of the reaction—public health risk
  
• Demographics—age, gender, ethnic background, weight
  
• Effect of exposure—duration and dose—changes in risk over time
  
• Concomitant medications
  
• Drug interactions
  
• Comorbid conditions and other confounders
  
• Biological plausibility
  
• Alterative treatments and therapies
  
• Other issues (e.g., HA request for workup, publicity)

Next, each drug/signal combination should be assigned a signal level based on review of the cases and causalities. Be reasonable in terms of what constitutes a signal. Always keep in mind the benefit–risk balance: not all risks can be eliminated. One such classification is

• Strong: a series of well-documented cases with no alternative causes and ideally with at least one positive rechallenge (rechallenge criterion not applicable in, for example, irreversible adverse events, hepatotoxicity, etc.)
  
• Fairly strong: a series of generally well-documented cases with few alternative causes and ideally at least one positive dechallenge
  
• Average: a series of cases of variable quality
  
• Fairly weak: a series of cases that have significant limitations regarding plausible temporal associations or for which there are likely alternative explanations
  
• Weak: a series of cases that are generally incompletely documented, lack plausible temporal associations, or are generally explainable by alternative causes or similarly

And then assign an action:

• A signal warranting immediate action to protect public health. These actions may be temporary (if the signal is ultimately determined to be unfounded) or permanent.
  
• Signal warranting intensive follow-up and further investigation in the form of a clinical trial, an epidemiologic trial, outside consultation, and so on.
  
• Signal warranting further investigation and follow-up of the current cases (e.g., for outcomes); to be reexamined in 60 days.
  
• Weak signal: continue watching; no further action at this time.
  
• Not a signal: no further investigation needed.

The Workup

At this stage, the signals that have been chosen for workup should be so designated and the workup begun. Various steps that can be done include the following:

• Search for additional cases using the appropriate MedDRA terms (or SMGs) in the clinical trial database if some cases (e.g., nonserious clinical trial AEs) are not also found in or have been reconciled with the safety database.
  
• Search for similar or additional cases in external databases (see Chapter 8) such as the UMC database in Uppsala, Health Canada’s database, the MHRA Drug Analysis Printouts (DAPs), FDA’s AERS database, and FDA potential signals listing (see Chapter 21).
  
• Consider other databases that can be used for epidemiologic studies in addition to the spontaneous reporting databases noted in the previous bullet. These include Prescription Event Monitoring Databases (Drug Safety Research Unit in the United Kingdom, Web Resource 19-3), Linked Administrative Databases (U.S. private healthcare databases), United Kingdom General Practice Research Database (GPRD) (Web Resource 19-4), as well as specialized databases, such as teratology databases or disease-specific databases (e.g., cystic fibrosis), and governmental databases (e.g., Canadian provinces). The organization Bridge to Data (Web Resource 19-5) has a compilation of more than 90 worldwide databases with descriptions of their characteristics, allowing the user to find databases that may suit the signal workup.It may be useful to engage an expert in pharmacoepidemiology at this point to find the right databases and assist in designing the appropriate study.
  
• Search out additional literature cases using PubMed, Google Scholar, or other search engines and databases. See if the signal is listed on FDA’s potential signal database.
  
• Consider reviewing the AE profiles and class effects of similar drugs in that class.
  
• Consider more complex, time-consuming, and expensive procedures to validate, strengthen, or refute a signal, such as epidemiologic (observational) studies in large databases (e.g., claims databases), to detect or find rare AEs and obtain information in large patient populations (e.g., tens of millions of patients), targeted clinical trials, and large simple safety studies (LSSS).

The Conclusions and Next Steps

The reviewers should come to a conclusion or conclusions for recommendation to the decision maker or safety committee (see below). As noted, many classifications are available; pick one and stick to it. The conclusions may be along the lines of

• Red Signal–High Priority: SAE previously unknown or unlabeled or inadequately labeled. Quality issues such as adulteration or contamination. May be accompanied by media attention and public scrutiny despite the only weak or incompletely documented cases. If confirmed, will lead to a reevaluation of the benefit–risk analysis and likely a change in labeling, product withdrawal, and so on.
  
• Yellow Signal–Medium Priority: Further evaluation of the signal is required but the criteria of the Red category are not met. If confirmed, these signals are expected to lead to a change in the risk–benefit analysis and may require changes in the labeling/packaging in the AE section and possibly also in the indications, contraindications, warnings, and adverse event sections.
  
• Green Signal–Low Priority: AEs that are already known or labeled and felt not to be a significant safety problem. Signal investigation at this time may be minimal, deferred, or simply kept on a “watch list” looking for further case reports (if any) before reevaluation. Workup now would not be a good use of resources.

The Safety Committee

Following the prioritization and workup, a mechanism to conclude and act on the signals is needed. This may be a senior safety/risk management committee or it may be an individual (the chief medical officer, for example). Whatever mechanism is used, there must be a formal written procedure to review and adjudicate signals on a regular basis. There should be an empowered decision maker in the form of either a person or a committee.

For emergency signals, the committee should be able to meet within 24 hours (or even sooner). In a pharmaceutical company, this could be a senior safety committee composed of the chief medical officer, chief safety officer (if not the same person), and heads or senior people from drug safety/pharmacovigilance, regulatory affairs, labeling, clinical research, the legal department, preclinical (animal) toxicology/pharmacology, risk management, epidemiology, and other corporate subject matter experts as needed (e.g., formulations). If the product is studied or marketed outside the home country, the needs of these countries must also be represented in the decision and action steps. The marketing and sales and similar departments should not, in general, be represented on this committee, as this must be a medical–public health decision. In occasional instances, outside expert consultants, as neutral as possible, given that they are paid consultants to the company, may be invited to join if appropriate.

In a health authority, the committee structure should be constituted in a similar manner, with senior medical, toxicology, pharmacology, labeling, risk management, epidemiology, and legal subject-matter experts as well as any other members needed, depending on the structure of the health authority. Attention should also be paid to actions of other health authorities around the world.

The safety committee needs to come to conclusions about issues presented to it. It should never routinely request more data at successive meetings for a particular problem or use other bureaucratic mechanisms to delay a decision. Relevant data should be requested and rapidly obtained and decisions made. These decisions should be documented in minutes. The outcomes should consider the public health and risk management/minimization and what action steps, if any, need to be taken:

• Label change, variation, and so forth, for marketed drugs (e.g., new ADR, warning, precaution, contraindication); dear doctor/healthcare professional letter; drug withdrawal and, if so, to what level (consumer, pharmacist, wholesaler); and communication plan to the health authorities, public, and healthcare professionals.
  
• Further study and consultation regarding this signal.
  
• If in clinical trials, stop or change studies to enhance patient protection, notification of the data monitoring committee and/or IRB, adjudication committee, changes in the investigator brochure, and informed consent.
  
• Notification of the applicable health agencies (competent authorities) by phone, e-mail, or letter.
  
• Other follow-up actions and further review by the committee at a later date.
  
• Effect on the Risk Management/REMS program in place or, if one is not in place, whether to put one in place rapidly.
  
• Mechanisms to handle the public announcement and any issues that might arise from that, including legal actions and adverse publicity.
  
• If a REMS or RMP is in place, the signal should be considered in context with the plan. It may be necessary to revise, change, or update the plan in consultation with the health authority.
  
• Either inside or separate from the plan, it may be necessary to take further risk minimization actions.
  
• Recall, withdrawal, etc.

When the committee is in a health agency, depending on legal responsibilities and regulations, the committee needs to decide on label changes, withdrawal, and study cessation in the same manner as noted above for companies.

For an excellent review of signaling, see Practical Aspects of Signal Detection in Pharmacovigilance, Report of CIOMS Working Group VIII, Geneva 2010 (see Chapter 36).

Stay updated, free articles. Join our Telegram channel

Join