Productivity and Project Success
Measures of productivity are like statistics on accidents: they tell you all about the number of accidents in the home, on the road, and at the work place, but they do not tell you how to reduce the frequency of accidents.
–W. Edwards Deming. From Quality, Productivity, and Competitive Position.
DEFINITION AND DESCRIPTION OF PRODUCTIVITY AND INNOVATION
Virtually every organization and group, regardless of their functions and goals, desires to improve their productivity and number of innovations. For research-based pharmaceutical companies, the productivity of their research and development (R and D) function is vital for corporate prosperity and, almost always, for its long-term survival. Measuring and tracking a company’s productivity and innovations enable senior managers to assess whether the funds they commit to R and D are well spent and are ensuring the future success (and even existence) of their company. It is also relevant for senior R and D managers to measure, analyze, and report on the productivity and innovativeness
of the groups they manage when justifications are required to support their annual budget requests or to defend the effectiveness of R and D. Evaluations of a group’s productivity or innovativeness may also be used as a means to identify problem areas and as a guide to improve the effectiveness of drug discovery and development activities.
of the groups they manage when justifications are required to support their annual budget requests or to defend the effectiveness of R and D. Evaluations of a group’s productivity or innovativeness may also be used as a means to identify problem areas and as a guide to improve the effectiveness of drug discovery and development activities.
Productivity and innovation are commonly used terms in the pharmaceutical industry that everyone understands, or do they? Dictionary definitions of these concepts cannot be directly applied to drugs or the pharmaceutical industry without making numerous assumptions and decisions about what factors to consider and how they should be applied. One of the reasons for this situation is that both words (productivity and innovation) have several definitions with various shades of meaning. As a result, both terms are defined operationally in this chapter, and it is acknowledged that other definitions could be proposed. Unfortunately, most discussions of productivity and innovations in the scientific and medical literature do not clearly specify the precise definitions used.
Definition and Description of Productivity
Chambers’ 20th Century Dictionary defines productivity as “the rate or efficiency of work, esp[ecially] in industrial production.” Webster’s New International Dictionary of the English Language includes the phrase “yielding or furnishing results, profits, or benefits.” These definitions indicate that productivity is a rate or ratio. It is usually viewed as an output divided by some type of input.
In the manufacturing area, the most traditional definition of productivity is output per labor hour. In marketing, finance, and other disciplines, the definition of productivity generally emphasizes how well the company earns money and is able to accomplish the same or an increased number of tasks with fewer people, less time, and/or less money. The most important aspect of productivity in R and D is the number of commercially viable new drugs that were discovered, developed, and marketed. This number may be expressed per dollar spent on R and D or by another parameter (e.g., per 1,000 people-years of work). A productive company will be able to demonstrate this quality through the number of new drugs it “produces.”
The productivity of R and D is evaluated in many different disciplines and areas (e.g., the number of patients who complete a given trial per month or number of key strokes of data entered per hour by a data entry operator). The term productivity includes consideration of the efficiency of an activity or operation, whereas the term innovation does not.
Definition and Description of Innovation
The definition of innovation in numerous dictionaries is (a) something new or different and (b) introduction of new things or methods. Although there may be many innovations relating to the process of drug development (e.g., an improved method of conducting any one of thousands of activities), in this chapter the term is restricted to the discovery and development of new drugs.
The process of innovation was described as having four separate functions or components (Scherer 1990). These functions may be modified to fit the drug industry:
Invention. Conceptualization of a new drug, at least in a rudimentary form, plus the demonstration that the compound or drug possesses either biological or clinical activity of interest.
Investment. This involves the risking of funds and commitment of resources for a compound or drug’s development.
Entrepreneurship. Management’s decision to pursue a compound or drug’s development and the commitment to efforts required for efficiently organizing and carrying out the development.
Development. This involves the lengthy and complex steps required to bring an investigational compound to market. If a drug is already marketed, then new development activities may be necessary to attain regulatory approval for the newly discovered activity.
This description indicates that innovation involves much more than just drug discovery. The primary quantitative measure of innovation, however, is the number of new drugs or drug forms (e.g., dosage forms or creative packaging) developed. This number is often expressed on a per-year basis. It is unclear whether a drug must be marketed for a company to count it as an innovation. A drug that is a new chemical entity and is sold in country A but is rejected by regulatory authorities in country B would generally be considered as an innovation in country A, but not in country B. A different end point than drug marketing should be used to define an interesting compound (in the preclinical period) as an innovation. A logical time to make this distinction would be to use the point at which a compound becomes a drug. This has been defined in this book as the moment when the first human is given a compound in a clinical trial. On the other hand, it is also possible to define a chemical compound as an innovation at the time that a company’s management has made the commitment to invest money and to pursue the drug’s development. This is usually the time when a compound is elevated to project status. Whichever of these three time points is chosen as a criterion of defining a new chemical entity (NCE) as an innovation (i.e., commitment to development, first human exposure, or initiation of marketing), that criterion should be consistently applied. Other time points are possible to use in determining when a drug can rightfully be called an innovation.
Although the preceding discussion was written as if each drug represents a single innovation, there are several reasons why this is not always true:
There may be innovations in different therapeutic indications for a single drug in which new and important uses are discovered. These discoveries may occur almost anywhere along the drug discovery and development pipeline, from preclinical studies to the testing of clinical theories to serendipitous events observed by practicing physicians. A major innovation is defined as one where there is little or no precedent in currently used drugs for that indication, and the new drug represents an improvement in therapy.
Innovations may occur in marketing and production, in addition to R and D. Novel manufacturing processes may allow a drug to be produced that otherwise would be impossible to manufacture for one reason or another (e.g., raw supplies are unavailable anywhere in the world) or the drug’s manufacture would be too expensive to be competitive in the market.
Medicine is practiced differently around the world and the choice of drugs used, even in industrialized counties, varies greatly. A number of well-known and highly used drugs in
one country may be little known and little used just across the country’s border. Thus, an innovation in one country will not always be considered as an innovation in other countries.
Numerous creative ideas contribute to the development of virtually all drugs. These “innovative” ideas are not innovations in the sense that the overall drug is defined in this section as a single innovation. Nonetheless, creative ideas leading to advances in a drug’s development (e.g., discovering a new route of chemical synthesis) could be defined and considered as innovations.
Productivity and Innovation versus Value
Many definitions and descriptions of productivity and innovation do not consider the value of the drugs developed. Value may be measured and/or judged in both commercial and in medical terms. Numerous aspects of these values, as well as scientific value, are described in Chapter 52 and Section 1. A company may wish to judge its R and D group, not solely by the number of drugs under development and how well they are being developed, but also by their value. Methods and parameters that include both commercial and medical value should be considered.
An R and D group may work hard and efficiently in developing drugs, but through no fault of its own, most or all their important investigational drug projects may have to be terminated. This could occur for many reasons, such as toxicity in animals, adverse events in humans, or lack of sufficient efficacy in patients. An R and D group in this situation might still be rated as highly productive and innovative, depending on the specific definitions used. It is also possible that the number of their innovations would be zero, and their productivity would also be zero, if the definitions used required a drug to be marketed.
Drugs may have a high medical value but low commercial value and never repay their development and marketing costs. It may be debated as to whether these drugs contribute to a company’s productivity. On the one hand, they clearly do because of all the benefits that patients will receive, but on the other hand, a company that is highly productive but only markets this type of drug will eventually go out of business.
GENERAL METHODS TO MEASURE PRODUCTIVITY
Levels at Which Productivity Is Measured
It is usually important to measure productivity at multiple levels, including that of an individual scientist, a group of scientists, an individual research department, a project team, all research departments, all development departments, all R and D personnel, an entire company site, and/or an entire multinational company. Some methods that are appropriate for one level are not suitable for others.
The productivity of individual scientists is usually measured as part of their annual performance review. This chapter is concerned with judging the productivity of the entire drug research or drug development effort, not of the individuals who plan, conduct, and supervise this effort.
Establishing a System to Evaluate Productivity
One approach to evaluating productivity is to divide the evaluation into the following questions:
What is to be assessed?
What general tools and techniques are available to evaluate productivity?
What are the pros and cons of each method for the specific tasks intended?
Which tools and techniques are best to use and which parameters are best to measure?
How many parameters should be measured and is it worth combining the results into an overall index?
Will greater precision in measurement make any difference in actions taken to improve productivity?
What comparison data are available?
How will these evaluations be conducted?
Who will conduct this evaluation and interpret the results?
Who will review and approve the results?
How will the results be applied to the company (e.g., to improve productivity)?
A number of additional questions and issues should be considered before setting up the exercise as follows:
Are there preconceived end points or results that the evaluation is supposed to demonstrate or attain? If so, are these results hidden or clearly stated? Are there mechanisms to prevent this type of bias from influencing the evaluation?
Does the staff in charge of the evaluation have vested interests in having the outcome fit a predetermined pattern? How may these vested interests be controlled so that they do not bias the outcomes?
To what depth will the measure(s) be used to evaluate productivity (i.e., is a superficial or an in-depth analysis being conducted)?
Is this activity a one-time analysis or is it intended to become a periodic (i.e., ongoing) activity?
Making “Snapshot” Evaluations at a Single Point in Time versus Developing Trends Over a Period of Time
Comparing the effectiveness or productivity of an R and D function with similar groups in other companies often compares activities and situations at a single point in time. This is analogous to evaluating a photograph of the R and D function at a specific moment in time. It is also important to judge changes over a period of years to determine whether a company’s performance is improving, deteriorating, or remaining constant. Evaluating changes over time could be viewed as analogous to studying a movie version of R and D or whatever function is being assessed.
Who Will Conduct This Evaluation?
There are basically three answers to this question. The analysis may be conducted primarily (or solely) by (a) internal company staff, (b) external consultants, or (c) by a joint effort of both. There are a number of consulting groups with experience in reviewing all or part of a pharmaceutical company’s R and D organization. External consultants will require significant assistance
in time and effort from the company’s staff, even when the consultants perform all of the analyses and evaluations themselves.
in time and effort from the company’s staff, even when the consultants perform all of the analyses and evaluations themselves.
In the situation where both company staff and outside consultants are involved in performing the analyses, several scenarios may be followed. The first is where members of the evaluation group are chosen from both the company and consultants to work as a unified team, perhaps being instructed in the general approach to follow by the consultants. Another possible scenario is for the consultants to adopt a more passive role and serve as facilitators helping guide the company staff and keeping them on track as they gather information and conduct the analyses in-house. The opposite situation, in which company personnel serve as facilitators for the consultants, is also possible, but less likely to occur.
SPECIFIC PARAMETERS TO MEASURE PRODUCTIVITY
What Components of Research and Development May Be Assessed to Measure Productivity?
A company may evaluate the productivity of a highly specific part of their R and D division or they may wish to conduct a sweeping evaluation of the entire R and D function. In either case, each of the components that is to be assessed must be identified. If a highly specific part of R and D is to be evaluated, it may consist of a single component. If the entire division is to be assessed, it may be desirable to choose some global measures or to examine separate components. Data on any parameter obtained at different times may be plotted to evaluate trends.
Parameters to Measure Overall Research and Development Productivity
R and D productivity may be globally measured using parameters such as (a) the number of Investigational New Drug Applications (INDs) filed, (b) the number of projects started, (c) the commercial value of projects started, (d) the number of New Drug Applications (NDAs) and other marketing applications (e.g., Biologic License Applications, Premarket Approval Applications) filed, (e) the number of marketing applications approved, (f) the speed of NDA approval compared with other companies, and (g) how closely the total group met their annual goals in terms of the first six parameters listed. The first six parameters may be expressed on a per-year basis and the trends may be analyzed over time. In addition, these parameters may be evaluated on a per-R and D-person basis (per year) to make the comparison a bit more fair from year to year because the company’s total R and D effort is likely to change from year to year in the number of people employed and the money spent. Only professional staff above a certain level could be considered. A three-year (or other) moving average could be used to smooth out differences further that often occur from year to year. The money spent per person per year on the three major functions of R and D groups (drug discovery, drug development, and market-oriented activities, including line-extensions) could be determined separately or together. See Chapters 52 and 111 for examples of figures that illustrate R and D expenditures.
A number of other objective parameters that may be used to measure productivity are listed in Table 56.1.
A variety of grids or scattergrams could be produced to illustrate the status of investigational drug projects. Grids from two or more years could easily be compared to determine whether the movement of projects was in a desirable direction and was adequate in amount (see Fig. 52.12). A highly gross and imperfect measure of productivity would be the number of months taken by a regulatory authority to approve NDAs or Product License Applications (Fig. 56.1).
The number of months taken for a drug or the average number of months for all drugs approved in a given year or group of years could be compared with the company’s previous performance. These values could also be compared with those of the pharmaceutical industry. Numerous caveats are required for this comparison because the time taken by a regulatory authority to approve drugs depends on many factors. These include the type of drug, the medical need for the drug, and the specific reviewers involved. This measure would be related to a company’s productivity if there is a correlation between the quality of the data submitted and the speed of regulatory approval. The author believes that this correlation is relatively high for initial NDA applications. This measure is less indicative of productivity when evaluating supplemental NDAs.
Another criterion of overall productivity would be degree and quality of the liaison between R and D and other functional areas in a company. Feedback from other groups that frequently interact with R and D (e.g., marketing and production) would be important to judge how well the liaison was being accomplished. It would be ideal to measure both the number and value of good ideas and good decisions made as an assessment of productivity. Unfortunately, this cannot be accurately done.
Parameters to Measure Various Aspects of Research and Development Productivity
In addition to evaluating the productivity of the overall research and development function previously described, it may be relevant to measure the productivity of its component parts. Various parts (e.g., drug discovery, medical group, and technical development) or departments may be evaluated with some or all of the following parameters. Numerous caveats will be necessary in using each of these measures:
The number of therapeutic areas or disease areas being researched
The specific choices of therapeutic areas being explored for new drug discoveries in terms of medical need and commercial value
The number of compounds and drugs or NCEs in the project system
The speed and efficiency with which drugs are moving through the drug development pipeline. Evaluate the number of months from project formation to IND submission, the duration of Phase 1, Phase 2, Phase 3, and total development time to NDA submission.
The time from IND filing to NDA submission. This may be evaluated on a moving three-year average and compared to the company’s previous performance as well as the industry average.
The percent of regulatory submissions (e.g., NDAs, Biologic License Applications, Product License Applications) that are in each category of therapeutic importance (e.g., 1A, 1B, and 1C; priority or regular)
Table 56.1 Parameters for measuring productivity of research and development activities a,b
A.
Regulatory parameters
1.
Number of new products reaching the market in X years
2.
Number of new regulatory submissions for NCEs in the United States and/or several specific countries
3.
Number of countries approving specific drugs
4.
Speed of regulatory approval
B.
Commercial parameters
1.
Amount of sales and/or profits generated from products that reached the market within the past three yearsc
2.
Percent of sales and/or profits generated from products that reached the market within the past three yearsc
3.
Number and value of drugs licensed-in within the past three years c
4.
Number and value of drugs licensed-out within the past three years and how the company benefits
5.
Potential commercial value of all investigational drugs (e.g., total of estimated third year sales times the probability of marketing the drug) d
6.
Potential commercial value of projects established within the past year
7.
Number and value of line-extensions currently being developed
8.
Average cost to complete each stage of development e (e.g., preclinical, Phase 1, Phase 2, Phase 3)
9.
Average research cost to discover an NCE that reaches project status
10.
Ratio of new drug sales over X years divided by R and D expenditures over that same period
C.
R and D parameters
1.
Medical value of all projects with investigational drugs
2.
Medical value of projects established within the past year
3.
Number of new projects established within the past year
4.
Number of compounds reaching human trials within the past year
5.
Rapidity with which unsuccessful projects have been terminated within the past three years (i.e., the number of months from project formation to termination)
6.
Degree of congruence between projected goals and actual accomplishments
D.
Personnel parameters
1.
Number of senior staff who left the company within the past three years
2.
Evidence of positive staff morale
3.
Number of publications by staff
4.
Image of the R and D departments
5.
Number of staff with academic appointments
a Effectiveness or productivity would be assessed by comparing the value of a parameter with those obtained for an equal preceding period or with values obtained in previous years. Trends would be of particular importance. If the parameters had not been previously measured, then a new analysis of older data may be conducted. All of these values could be presented in tabular or graphical form. Two or more of these parameters could also be combined.
b Not all of these parameters are considered worthwhile measures of productivity. The use of some would depend on the specific situation.
c A different number of years could be used.
d These numbers may be reduced by multiplying by the percent of compounds or drugs put in the project system that historically reached the market (i.e., success rate).
e Financial comparisons should be made using dollars of comparable value to correct for inflation. Other corrections may also be used, but must be clearly defined.
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
