Measuring prognostic performance

There are two useful statistical considerations when examining prognostic markers: effect sizes and prognostic performance. Effect sizes are measures of association, such as relative risk and odds or hazard ratio (Chapter 3). They indicate whether the relationship between a marker and a disorder is small, moderate or large. Many researchers only provide effect sizes when they report prognostic studies, which is the first step, but these do not really tell us how good a prognostic marker is at forecasting outcomes. Both measures are ultimately required.

Prognostic performance is examined using two quantitative parameters: detection rate (DR, also called sensitivity or true positive rate) and false-positive rate (FPR, which is 1 − specificity). These are illustrated in Figure 9.1.

Unlike effect sizes, which are calculated by analysing all individuals together, DR and FPR always consider them as two separate groups: people with the outcome (affected) and those without (unaffected).

It is important to have both the DR and FPR. A high DR (e.g. 85%) is meaningless without knowing whether the FPR is low or high (e.g. 5 or 50%). A diagnostic marker or test should have a DR close to 100% and FPR close to 0%. There is no gold standard for what makes a good prognostic marker, in terms of size of DR and FPR. Figure 9.1 shows an excellent marker (high DR and very low FPR), but many other markers have lower DRs (50–70%) and higher FPRs (e.g. 5–20%).

Marker performance can be considered in relation to the seriousness of the disorder, and what happens to people who have positive test results. For example, markers for predicting Down’s syndrome in pregnancy have a DR of 70–85%, but this comes from specifying a low FPR (< 5%), because women who are test positive are offered an invasive and expensive diagnostic test (e.g. amniocentesis), which has a risk of miscarriage. In contrast, mean corpuscular haemoglobin is used to screen for β-thalassaemia in pregnancy and because the aim is to detect almost all cases (DR close to 100%), the FPR can be relatively high (up to 37%) The subsequent test is only another blood (diagnostic) test that, although relatively expensive, is not harmful.

The most commonly used terms in the literature are sensitivity and specificity, although DR is a better term than sensitivity, which has other definitions in medicine (e.g. lower limit of assay measurement). FPR is more relevant than specificity, because it designates the individuals who go on to have further investigations and/or interventions.

DR and FPR can be combined to produce a likelihood ratio (LR = DR/FPR), which is a way of quantifying the ‘power’ of the marker. LR = 1 is a useless marker. The larger the value, the higher the DR in relation to the FPR (e.g. a good marker would have DR = 80% and FPR = 5%, which produces a large LR of 16).

Two characteristics of prognostic markers or models are worth noting:

1. Markers do not identify all individuals with the outcome of interest (cases are always missed), that is, DR would never be 100%.
   
2. Markers will always pick up individuals without the outcome (‘error’ and these individuals could have unnecessary further investigations or interventions): that is the FPR would always exceed 0.

Although DR and FPR are important parameters, it is also necessary to see how well the marker works in the population of interest. This is achieved by examining the odds of being affected given a positive result (OAPR) or positive predictive value (PPV); see example 9.1 (page 177).

When several prognostic markers are examined together, they should be combined into a model using multivariable linear, logistic, or Cox regressions depending on whether the outcome measure involves ‘taking measurements on people’, ‘counting people’, or measuring the time until an event occurs.

Effect sizes such as relative risk, odds ratio, and hazard ratio measure the relationship between an exposure (prognostic marker) and outcome. This is conceptually different from the prognostic performance of the marker, which examines how well it predicts the outcome. It is not commonly recognised that a prognostic marker needs to be very strongly associated with the disorder (i.e. high or low odds or hazard ratios) to have a good performance [8]. For example, maternal serum alpha-fetoprotein (AFP) is a prognostic marker for neural tube defects in pregnancy (and used for prenatal screening). The odds ratio between the highest and lowest quintile of AFP is 246 (a very large effect), and the corresponding DR = 91% and FPR = 5% indicates good performance. In comparison, although the association between serum cholesterol and ischaemic heart disease is established and appears to be a large effect (odds ratio of 2.7 between the highest and lowest quintiles among UK men), the DR is only 15% for a FPR of 5%; that is, cholesterol is not an effective prognostic marker, because it has a poor performance [8]. Having a moderate or reasonably strong association does not necessarily mean that the marker is good at predicting outcomes.

Training and validation datasets (examining several markers)

When using a single prognostic marker, one dataset is often sufficient when the cut-points used to estimate DR and FPR are already pre-established and have not been developed by the researchers. For example, in Figure 9.1, the retinal photograph can only be normal or abnormal, and for a continuous marker like blood pressure, fixed values such as ≥ 120, ≥ 130, ≥ 140 mmHg, etc. can be used. Neither of these is driven by the actual dataset. However, if there is uncertainty over the generalisability of the study findings, then it might be appropriate to examine the marker in another (independent) dataset.

When examining several prognostic markers, each will probably have different units of measurement, so they need to be combined to produce a single measure, and then a cut-off point can be applied, as if dealing with a single marker. This combined measure and cut-off point come from a statistical model, and the model needs to be developed by the researchers. This could be done using a large study that is split into two, or two separate studies of similar individuals, so that the prognostic model is developed with one dataset (training dataset) and tested on another (validation dataset). Using the same dataset to both develop and test the model will give biased (optimistic) estimates of performance. The validation dataset can also be used to check whether the model is sufficiently reliable by comparing estimated (predicted) with observed risk values.

Splitting a dataset (half training and half validation or, two-thirds training and one-third validation) could be performed randomly, but then the two groups should be similar to each other, and this could overestimate prognostic performance. It is better to split the dataset using time (e.g. the first two-thirds of registered participants in the study form the training set, and the last third make up the validation set). This has the advantage of attempting to validate the model prospectively, which is similar to what would be done in practice (i.e. a model developed now, to be applied to future individuals). The number of events is as important as study size, so it may be better to divide the data according to events (see Example 3), to ensure that sufficient numbers of events occur in each dataset to produce reliable results. Having a completely independent dataset (as the validation data) is ideal because it will address generalisability, but it can often be difficult to find one that has similar individuals to the one used to develop the model and that has all the factors measured.

Other methods are cross-validation or bootstrapping (especially if the dataset is not large enough to be split). With cross-validation, one individual is removed, and the model is developed using the remaining individuals and then tested/validated on the removed individual, who is classified as marker positive or not. This is repeated for every individual in the dataset, and DR and FPR are obtained by counting the number of positives in the affected and unaffected groups. With bootstrapping, if there are, for example, 435 individuals in the study, a random sample of 435 is selected, after replacing each person (so an individual could be included several times in the sample or not at all). The model is developed on this sample and then tested on the original dataset. The process is repeated at least 100 times, with the average model performance estimated.

Another approach involves using the dataset to produce statistical parameters (e.g. means, standard deviations and correlation coefficients), that are incorporated into a model which produces DR, FPR and OAPR [9]. An advantage of these methods, over splitting the dataset, is that all of the data values are used to develop and test the model.