Instrument and Method Assessment



Instrument and Method Assessment



Objectives


At the end of this chapter, the reader should be able to do the following:


1. Define the true negatives, true positives, false negatives, and false positives for a method when determining the diagnostic value of that method.


2. Calculate the diagnostic sensitivity, specificity, efficiency, and positive and negative predictive value of a method.


3. Use the Student’s t-test to compare the difference between both paired and unpaired data sets.


4. Analyze the precision of methods by the use of the F test.


5. Define the terms used in linear regression analysis.


6. Calculate, using linear regression analysis, the slope, intercept, and degree of bias between two methods.


7. Calculate the standard error of the estimate associated with linear regression analysis.


8. Determine the coefficient of correlation associated with linear regression analysis.




DETERMINATION OF THE DIAGNOSTIC VALUE OF A METHOD


Diagnostic Sensitivity and Specificity


The goal of the clinical laboratory is to produce accurate and precise results that aid physicians in the diagnosis and treatment of disease states and processes. In addition, the laboratory tests should be able to distinguish between patients who have a disease and those who do not have the disease. Figure 15–1 is a graph of two sets of patients. The group on the left does not have the disease, whereas the group on the right has the disease. The best laboratory test will have no overlap between groups (i.e., a gray area in which the physician would not be able to diagnose if the patient had the disease). Figure 15–2 demonstrates a test that has an overlap between groups.


image
FIGURE 15–1 No overlap between groups.

image
FIGURE 15–2 Overlap between groups.

Diagnostic sensitivity is the probability that only patients with the disease will test positive for the disease. Diagnostic specificity is the probability that patients who do not have the disease will test negative for the disease. The best test will have 100% sensitivity and specificity.


The sensitivity and specificity of a method can be calculated by the following formulas:


Sensitivity=True PositivesTrue Positives+False Negatives×100Specificity=True NegativesTrue Negatives+False Positives×100


image

where: True Positives (TP) = Number of individuals who actually have the disease and test positive


False Positives (FP) = Number of individuals who do not have the disease but test positive


True Negatives (TN) = Number of individuals who do not have the disease and test negative


False Negatives (FN) = Number of individuals who have the disease but test negative





Example 15–1

A new kit was developed to detect the presence of Group A Streptococcus in throat swabs. The manufacturer tested the kit on 600 pediatric patients with presumptive diagnoses of Group A Streptococcus. Four hundred and seventy five patients tested positive for Group A Streptococcus with the kit. Of these, 475,460 were verified by culture to be positive for Group A Streptococcus. One hundred-twenty five children tested negative by the kit; of these, 20 were determined to be positive for Group A Streptococcus by culture. What is the diagnostic sensitivity and specificity of this new kit?


To calculate the diagnostic sensitivity and specificity of this kit, first determine the number of true positives, true negatives, false positives, and false negatives. If 475 children tested positive by the kit, but only 460 of them were confirmed, then the number of true positives is 460 and the number of false positives is 15. Because 125 children tested negative by the kit but 20 were actually positive, then the number of true negatives is 105 and the number of false negatives is 20. Therefore, by using the formulas, the diagnostic sensitivity and specificity of this kit can be determined.


image

Therefore, this method has a diagnostic sensitivity of 95.8% and a diagnostic specificity of 87.5%.



Additional Examples




image

The diagnostic sensitivity of this new kit is 89.9%, and its diagnostic specificity is 99.6%.



image

The diagnostic sensitivity of this new tumor marker is 94.1%, and its specificity is 99.4%.



Efficiency


In addition to diagnostic sensitivity and specificity, the efficiency of the test can be calculated. The efficiency of a test is the number of patients correctly diagnosed for the disease or not having the disease: true positives or true negatives. The following calculation can be used to determine the efficiency of a laboratory test:


Efficiency=TP+TNTP+FP+FN+TN×100


image




Example 15–2

Using the data from Example 15–1, what is the efficiency of the new Group A Streptococcus kit?


Analysis of the new kit revealed the following:


True positives = 460


False positives = 15


True negatives = 105


False negatives = 20


Using the formula to determine the efficiency of a test, the following equation is derived:


image

Therefore, the efficiency of the new Streptococcus kit is 94.2%.




Predictive Value


The last calculation that is used to assess the diagnostic value of a test method is the predictive value of the test method. There are two types of predictive value. The first is the positive predictive value of a method. The positive predictive value is the ability of the method to correctly determine the presence of a disease in those patients who have the disease. That is, patients who are positive for the disease will test positive by the test method.


The negative predictive value is the ability of a method to correctly determine the absence of a disease in those patients who do not have the disease. These patients should test negative by the test method designed to detect the disease.


The positive predictive value is calculated with the following formula:


Positive Predictive Value=True PositiveTrue Positive+False Positive×100


image

The negative predictive value is calculated with the following formula:


Negative Predictive Value=True NegativeTrue Negative+False Negative×100


image





Example 15–3

A new assay was developed that promised to be able to detect prostate cancer in men before the development of any clinical symptoms. The manufacturer claimed that the new assay was more sensitive and specific than any existing tests for prostate cancer. In a large clinical trial, blood samples from 10,000 men who were asymptomatic for prostate cancer were tested using this new assay. Two thousand men tested positive for this assay and 8000 tested negative. Of the 2000 positives, 1900 later developed prostate cancer. Of the 8000 negatives, 30 later developed prostate cancer. What is the positive predictive value for this test?


To calculate positive predictive value, the following formula is used:


Positive Predictive Value=True PositiveTrue Positive+False Positive×100


image

The number of true positives for this test is 1900, while there were 100 false positives. Therefore, the positive predictive value is as follows:


image

The positive predictive value of 95.0% means that if a patient tested positive with this new method, there is a 95% chance that they actually have prostate cancer.



Example 15–4

Using the data from Example 15–3, what would be the negative predictive value of this new prostate cancer test?


The negative predictive value is calculated by the following formula:


Negative Predictive Value=True NegativeTrue Negative+False Negative×100


image

Of the 10,000 men tested, 8000 tested negative. However, 30 of the men were false negatives, as they later developed prostate cancer. Therefore, there were 7970 true negatives. Using the formula, the negative predictive value is as follows:


image

A negative predictive value of 99.6% means that if a patient tests negative that there is only a 0.4% chance that they actually have prostate cancer.



Additional Examples




image Calculate the negative and positive predictive value for the following colon cancer test: Out of 6600 adults with a family history of colon cancer, 6245 tested negative. Of these, 12 developed colon cancer. Of the 355 individuals who tested positive, 340 developed colon cancer.


    There are 340 true positives, 15 false positives, 6233 true negatives, and 12 false negatives. Substituting the values into the formulas for negative and positive predictive values yields the following:


    Positive predictive value = 340/(340 + 15) = 0.958 × 100 = 95. 8%


    Negative predictive value = 6233/(6233 + 12) = 0.998 × 100 = 99.8%


image Calculate the positive and negative predictive value given the following data:


    True positive = 855 False positive = 3


    True negative = 340 False negative = 2


    The Positive predictive value = (855/855 + 3) = 0.996 × 100 = 99.6%


    The Negative predictive value = (340/340 + 2) = 0.994 × 100 = 99.4%



CLIA REQUIREMENTS FOR INSTRUMENT/METHOD QUALITY ASSURANCE


On January 23, 2003, the final rules for CLIA’88 were released that put the emphasis on quality back on the laboratory itself rather than on the manufacturer of the laboratory’s instruments. The original CLIA rules that were passed in 1988 and released in 1992 allowed the laboratory to use the manufacturer’s data for method validation, etc. The new rule mandated method quality assurance studies be performed by the laboratory instead of relying on the manufacturer’s data, such as method validation, accuracy, and precision studies. Except for personnel requirements, the 2003 final rules also changed the test categories of waived, moderate, and high complexity into waived and nonwaived categories. The final rules became effective after April 24, 2003.


The final rules stated that laboratories were not required to verify or establish performance specifications for any test system used by the laboratory before April 24, 2003. However, the laboratory was, and is, responsible to verify or establish performance specifications for any test system brought in after April 24, 2003. The rule further clarified what the laboratory must do to verify or establish performance specifications for unmodified test systems that were cleared or approved by the FDA and those test systems that were developed in-house.


For unmodified FDA approved or cleared systems the laboratory must:



For those tests that are developed in-house by the laboratory, or modified, or not cleared or approved by the FDA (including methods developed in-house and standardized methods such as text book procedures, Gram stain, or potassium hydroxide preparations), or uses a test system in which performance specifications are not provided by the manufacturer the laboratory must, before reporting patient test results, establish for each test system the performance specifications for the following:



In addition, Section 493.1281 of CLIA regulations states that “if a laboratory performs the same test using different methodologies or instruments, or performs the same test at multiple testing sites, the laboratory must have a system that twice a year evaluates and defines the relationship between test results using the different methodologies, instruments, or testing sites.” The same statistical tests that are used to determine accuracy can be used by the laboratory to meet this requirement. The Clinical Laboratory Standards Institute (CLSI) has a protocol to perform method comparisons.



Determining Accuracy


Accuracy is defined as the agreement between measurement and the true value. Accuracy studies are designed to detect systematic bias between two methods. A laboratory can determine the accuracy of a test method in a number of ways. The most common is to analyze 20 to 30 samples that span the reportable range of the method on both the new test method and a reference method. How can the laboratory determine if the results from an old analyzer and a new replacement analyzer are the same, or that the glucose result obtained in the emergency room satellite clinical laboratory would be the same result if analyzed on a different instrument in the main laboratory?


There are a number of inferential statistics that can be calculated to answer this question. Most laboratories today use computer programs specifically designed for the clinical laboratory to calculate these statistics. The information that follows is based on data that is normally distributed and has a sample number or “n” value of at least 30. Information on using population statistics or nonparametric sample distributions can be found in many statistics textbooks.



Student’s t-Test


Student’s t-test is useful to determine whether there is a statistical difference (or bias) between the two methods. The bias can be either positive (the new method’s results are higher than the reference method) or negative (the new method’s results are lower than the reference method.)


Student’s t-test can be an indicator of the accuracy of a new method. Before describing the actual calculations involved, some basic information must first be given. In general, one method is considered the “reference” method and the other method is the “test” method. Two hypotheses can be formed: the null (Ho) and the alternate (Ha). The null hypothesis states that there is no statistically significant difference between methods (Method A is the same as Method B), and the alternate hypothesis states that there is a statistically significant difference between methods (Method A is not the same as Method B). Based on the statistical difference, the null hypothesis is either rejected or accepted. When the null hypothesis is accepted, in statistical terminology, it is a failure to reject the null hypothesis.


To correctly reject or fail to reject the null hypothesis, a probability level (or significance level) of rejection or acceptance has to be established. The probability level depends on the degree of certainty that is required. For example, if a 95% probability level is used for acceptance of the null hypothesis, the significance level of rejection is 5%. The significance level is often referred to as a fraction of 100; that is, 5% would be expressed as P = 0.05. There is a greater degree of certainty if a 99% probability level is used because then there is only a 1% significance level (P = 0.01). The probability level used will depend on the needs of each laboratory. The probability level determines the degree of certainty that is required. What is not yet established is the numerical limit or cut-off point at which the significance level is reached. The critical region establishes this limit. In a Gaussian distribution, the values are equally distributed around the mean. Statistically, if there is a 95% probability level, there is a 5% chance that a result will fall into the critical region.


Whether there are one or two critical regions depends on whether the test is one-tailed or two-tailed. A one-tailed test is demonstrated in Figure 15–3. In this type of test, Method A compared to Method B has results that are either statistically greater than or less than Method B. For example, the null hypothesis may state that Method A results are greater than Method B or Method A results are less than Method B. In a one-tailed test, the entire 5% rejection region is on one side of the distribution.


image
FIGURE 15–3 One-tailed test.

In comparison, a two-tailed test has two possible outcomes. Statistically, it may be proved that Method A is different from Method B. Method A may be greater or less than Method B as shown in Figure 15–4. In two-tailed tests, the 5% significance level is split between both tails of the distribution. Therefore, there is a 2.5% probability that a result will fall in each critical region.


image
FIGURE 15–4 Two-tailed test.

Statistically, it has been determined that the actual numerical cut-off point for the 5% confidence limit for a two-tailed test with a sample size of at least 30 is the mean +/− 1.96 SD as shown in Figure 15–5. If a calculated t value is greater than 1.96 SD, it will fall into the critical region, and the null hypothesis will be rejected. If the calculated t value is less than 1.96 SD, statistically, there is a failure to reject the null hypothesis, and, therefore, the null hypothesis can be accepted.


image
FIGURE 15–5 Two-tailed test.

The last piece of basic information to be discussed is the concept of degrees of freedom. The amount that any number can vary is dependent on the amount of restrictions placed on that number. In a group of 10 numbers (n = 10) with a mean of 50, nine of the numbers have no restrictions. The tenth number is restricted because, based on the values of the previous nine numbers, the value of the tenth number is fixed. Therefore, the degrees of freedom in this example are 9 or n − 1.


There are two different calculations associated with Student’s t-test depending on if the data are paired or unpaired. For most clinical laboratory applications, the formula for paired t-test is used. Student’s t-test is the ratio of the difference of two means to the standard deviation of that difference.



Paired t-Test


The formula for the paired Student’s t-test is as follows:


Pairedt-test=d¯standarderrorofthemean(SE)


image

Where:d¯=difference between means divided bynd¯=∑X¯1−X¯2nSE=Σd2−[∑(X¯1−X¯2)]2÷nn−1nd2=∑(X¯1−X¯2)2


image

Combined, the formula for the paired Student’s t-test becomes:


Pairedt−test=∑(X¯1−X¯2)nΣd2−[∑(X¯1−X¯2)]2÷n]n−1÷n


image



Example 15–5

A laboratory wants to purchase a new chemistry analyzer that has a higher throughput than the current analyzer. The laboratory was able to temporarily obtain the instrument to conduct method comparison studies. As part of this study, 30 patient samples were split into two aliquots. Each aliquot was analyzed for glucose on both analyzers. Is there a statistically significant difference (P = 0.05) between analyzers?




































































































































Aliquot GLUCOSE RESULTS (MG/DL)
Analyzer A (current analyzer) Analyzer B (new analyzer)
1. 85 86
2. 198 195
3. 110 113
4. 140 142
5. 245 210
6. 103 99
7. 87 78
8. 96 99
9. 375 364
10. 68 64
11. 210 215
12. 105 103
13. 117 112
14. 184 193
15. 180 168
16. 157 153
17. 293 287
18. 83 81
19. 72 79
20. 102 112
21. 76 65
22. 89 93
23. 371 401
24. 125 120
25. 310 290
26. 138 153
27. 68 71
28. 90 96
29. 546 472
30. 193 184


image


To answer this question, it is helpful to phrase it as the null and alternate hypothesis.


Ho = there is no statistically significant difference between glucose results from both analyzers


Ha = there is a statistically significant difference between glucose results from both analyzers


To calculate the paired t-test value, first obtain the difference between samples (d) and the difference between means (d2).
































































































































































































Aliquot Analyzer A Analyzer B d d2
1. 85 86 −1 1
2. 198 195 3 9
3. 110 113 −3 9
4. 140 142 −2 4
5. 245 210 35 1225
6. 103 99 4 16
7. 87 78 9 81
8. 96 99 −3 9
9. 375 364 11 121
10. 68 64 4 16
11. 210 215 −5 25
12. 105 103 2 4
13. 117 112 5 25
14. 184 193 −9 81
15. 180 168 12 144
16. 157 153 4 16
17. 293 287 6 36
18. 83 81 2 4
19. 72 79 −7 49
20. 102 112 −10 100
21. 76 65 11 121
22. 89 93 −4 16
23. 371 401 −30 900
24. 125 120 5 25
25. 310 290 20 400
26. 138 153 −15 225
27. 68 71 −3 9
28. 90 96 −6 36
29. 546 472 74 5476
30. 193 184 9 81


image



∑=118=d∑=9264=d2


image

d¯=11830d¯=3.9


image

Next, substitute into the Student’s t-test formula the data from the problem.


Pairedt-testvalue=∑(X¯1−X¯2)nΣd2−[∑(X¯1−X¯2)]2÷n]n−1÷nPairedt-testvalue=118309264−[1182÷30]29÷30


image

Pairedt-testvalue=3.99264−464.129÷30Pairedt-testvalue=3.9303.430Pairedt-testvalue=3.917.45.5Pairedt-testvalue=3.93.2Pairedt-testvalue=1.219


image


The last step is to determine if the calculated t value exceeds the value obtained from the Student’s t table in Appendix 15–1. This is a two-tailed test because we are testing to determine if the two methods are different. We are not testing to determine if one method is greater than or less than the other. Using the Student’s t table, find the column under “Area” in two tails at the P = 0.05 significance level. As we have 30 samples, the degrees of freedom are 29. Next, find the number 29 under the column “df.” Look across the row until it intersects the 0.05 significance column. The number 2.045 is our tabulated t value. Our calculated t value is 1.219. Any calculated t value greater than 2.045 would fall into the rejection portion of the t distribution. As 1.49 <2.045, we fail to reject the null hypothesis: that is, we accept the null hypothesis. Therefore, there is no statistical difference between the glucose results obtained on the new and the old analyzer.



Additional Examples




























































































Aliquot Reference (core lab analyzer) Emergency room analyzer
1. 4.5 4.6
2. 4.3 4.3
3. 4.8 4.7
4. 5.6 5.4
5. 5.7 5.7
6. 4.1 4.2
7. 5.5 5.5
8. 6.1 5.9
9. 5.8 5.6
10. 6.2 6.3
11. 5.9 5.7
12. 3.4 3.3
13. 3.9 4.1
14. 4.2 4.5
15. 4.7 4.9
16. 6.0 6.2
17. 5.8 5.2
18. 3.9 4.1
19. 4.8 4.6
20. 4.2 4.2


image




















































Aliquot Current analyzer New analyzer
1. 15.2 16.0
2. 16.5 17.0
3. 14.1 14.0
4. 17.3 17.4
5. 10.2 8.9
6. 11.3 11.6
7. 9.7 9.4
8. 10.0 10.3
9. 13.4 13.2
10. 12.5 12.7


image




Unpaired t-Test


The paired t-test is useful when duplicate aliquots of the same samples are analyzed on two different analyzers to determine if there is a difference between analyzers. The paired t-test has some limitations in its application. Generally the same technologist analyzes the samples on both instruments to minimize any variations in testing. The sample size is identical for both analyzers as well. Sometimes a different situation occurs in which two different groups are compared to determine if there is a difference between them. For example, if 15 men and 20 women who suffered from hay fever received a new drug that promised to reduce the number of eosinophils in peripheral blood, the researchers would want to be able to combine the results obtained to determine if the new drug did in fact reduce the number of eosinophils. The unpaired Student’s t-test may allow them to combine the data. The null hypothesis would be that there is no statistically significant difference between the effect of the new drug on the eosinophil counts on men and women. The alternate hypothesis would be that there is a difference between the effect of the new drug on men and women. As most clinical laboratories, if using Student’s t-test, will use the paired t-test, the calculation of the unpaired Student’s t-test will not be discussed. For those laboratorians who use the unpaired Student’s t-test, information on performing the test is found in many biostatistics books.



Linear Regression Analysis by the Method of Least Squares


Student’s t-test is a useful but limited statistical tool when comparing methods. It may indicate that there is indeed a statistical difference between methods, but it cannot give information on the cause of the difference. One statistical tool that is used often in the clinical laboratory for method comparison is linear regression analysis. As with Student’s t-test, one method is considered the reference method, and the other, newer method, is considered the test method. In linear regression, analysis samples are tested by both methods. Then the results obtained from each method are plotted on linear graph paper. The results for the reference method are plotted on the x axis, and the results of the test method are plotted on the y axis. If, for each sample tested, the results for both test and reference method were identical, there would be perfect correlation between methods and a perfectly linear line as demonstrated in Figure 15–6.


image
FIGURE 15–6 Linear regression demonstrating perfect correlation between methods.

Linear regression analysis uses the following equation to determine the placement of the regression line:


yc=mx+b


image

where:



The slope of the line (m) is calculated by the following formula:


m=n(Σxy)−(Σx)(Σy)n(Σx2)−(Σx)2


image

The y intercept (b) is calculated by the following formula:


b=y¯−(mx¯)


image


There are some assumptions made when performing linear regression analysis. It is assumed that the reference method is accurate and precise, that random error is held to a minimum, that each method is performing to its best, and that there are no other factors that may interfere in the analysis of either method that might lead to erroneous conclusions. Most regression analysis is performed using computer software such as Excel. The manually performed examples that follow are provided to help explain how the regression formula is calculated.




Example 15–6

A new method for total calcium was developed, and linear regression analysis was performed between the new method and the reference calcium method. Using the CLSI guidelines, 40 patient samples were analyzed by both methods. The concentration of the patient samples used in this study encompassed the full analytical range of both methods. Using the regression analysis formula, what is the slope and calculated y intercept? Note: Not all data points are shown, but all necessary information is given.



Using the formula for the slope of the line and substituting into it the values from the problem yields the following equation:


m=40(3117.68)−(352.7)(351.8)40(3126.87)−(352.7)2m=124707.20−124079.86125074.8−124397.29m=627.34677.51m=+0.926


image

Related posts:

  1. Immunohematology Laboratory
  2. Quality Assurance and Quality Control in the Clinical Laboratory
  3. Dilutions and Titers
  4. Calculations Associated with Solutions

Stay updated, free articles. Join our Telegram channel
Tags:
Nov 18, 2017 | Posted by in PHARMACY | Comments Off on Instrument and Method Assessment

Full access? Get Clinical Tree
Get Clinical Tree app for offline access