the Evaluation of Automated MRI Brain Segmentations: Technical and Conceptual Tools

Slice Thickness 0.6 mm

Spacing Between Slices 0.6 mm

Pixel Spacing

Repetition Time 8000

Echo Time 282.89

All dataset volumes are altered by the presence of glial tumors, which are heterogeneous in terms of position, dimension, intensity and shape. A team of five medical experts was asked to segment axial, sagittal and coronal slices of these volume data by employing an image annotator normally in use in clinical practice and by offering standard image viewing facilities. Figure 1 shows an example of slice-by-slice manual segmentation of glial tumor areas provided by 5 experts along the axial plane and superimposed on the original MRI slice.

Fig. 1

Slice-by-slice manual segmentations of low grade glioma brain tumor performed by 5 medical experts

MRI segmentation was performed with the purpose of determining the size of pathological tissues and their spatial distribution in two or three dimensions according to the nature of the data. Metrics adopted in the present analysis for size estimation error and spatial distribution error are described below.

Size Estimation Error

Let be

the size estimation of the region (surface or volume) extracted from the axial, sagittal and coronal plane segmentation respectively, performed by the i-th expert. The intra- and inter- size estimation errors along the plane p with $p \in \{1,2,3\}$ and the i-th expert are computed as follows:

$\begin{aligned} \mathit{intraSizeErr}^{i}_{p}&=\frac{S^i_p - \frac{1}{N_{seg}}\sum_{j=1}^{N_{seg}}S^i_j}{ \frac{1}{N_{seg}}\sum_{j=1}^{N_{seg}}S^i_j}; \nonumber\\ \mathit{interSizeErr}^{i}_{p} &=\frac{S^i_p - \frac{1}{N_{exp}}\sum_{j=1}^{N_{exp}}S^j_p}{ \frac{1}{N_{exp}}\sum_{j=1}^{N_{exp}}S^j_p}\end{aligned}$

(1)

where N _segis the number of segmentations performed by the same expert on the same volume and N _expis the total number of experts.

Spatial Distribution Error

Let be

the 2D or 3D masks obtained from the segmentations along axial, sagittal and coronal plane respectively, performed by the i-th expert. The intra- and inter- spatial distribution errors, evaluated in terms of Jaccard Distance [13] are computed as follows:

$\begin{aligned} J_{p,t}^{i} =1- \frac{M^{i}{p} \cap M^{i}{t}}{M^{i}{p} \cup M^{t}{i}}; J_p^{i,j} = 1 - \frac{M^i{p} \cap M^j{p}}{M^i{p} \cup M^j{p}};\end{aligned}$

(2)

where i and j are indexes related to the experts and p and t related to the segmentation planes.

Fig. 2

2D intra-variability analysis conducted on each expert on one MRI volume: (a) Surface estimation error (b) 2D Spatial distribution error

2.1.1 2D Variability Analysis

Figure 2a shows the mean of the intra-size estimation error $intraSizeErr^{i}_{p}$ as computed varying the segmentation plane p and referring to each slice presenting a tumor of one MRI volume in the data set as operated by each varying expert.

Figure 2b shows the mean of the spatial distribution error $J_{p,t}^{i}$ , as computed varying all the possible pairs of planes $p,t$

and referring to each slice presenting a tumor of one MRI volume as operated by each varying expert.

The intra-variability measures confirm consistently an acceptable level of reproducibility for slices including the central part of the tumor area, with values lower than 15 and 20 % for the surface estimation error and for the Jaccard distance respectively. The intra-variability increases considerably in the slices which include the marginal part of the tumor with peaks of 103 % in surface estimation error and 92 % in spatial distribution error. This result can be interpreted mainly in light of two facts that the boundary masks are smaller and an error computed on few pixels results in a large percentage error; secondly that the slices are difficult to segment considering the high level of infiltration in the healthy tissue.

Fig. 3

2D inter-variability analysis conducted on 4 MRI volumes: (a)Mean of surface estimation errors (b) Mean of 2D spatial distribution errors

Figure 3a shows the mean of the inter-size estimation error $interSizeErr^{i}_{p}$ as computed varying the expert i and referring to both each volume in the data set and each segmentation along the axial plane.

Figure 3b shows the mean of the spatial distribution error $J_{p}^{i,j}$ as computed by each varying pair of experts $i,j$

and referring to each volume in the data set and to each segmentation along the axial plane.

Both the inter- variability measures adopted confirm a high level of variability when segmenting both central and boundary slices with peaks exceeding 50 % and with definitely unacceptable results in the boundary slices.

2.1.2 3D Variability Analysis

Table 1 reports the results of intra-variability analysis both in terms of volume estimation error and of 3D spatial distribution for 2 cases of the dataset. The analysis of the volume estimation shows an acceptable level of variability. The Jaccard Distances indicate instead a high level of variability in spatial distribution. The inconsistency of the two metrics comes from the compensation of errors in volume estimation.

Table 1

3D intra-variability analysis conducted on 4 MRI volumes

	Case 1		Cace 2
	Volume	3D jaccard	Volume	3D Jaccard
	estimation error (%)	Distance (%)	Estimation error (%)	Distance (%)
Expert 1	0.32	23.33	1.20	15.03
Expert 2	4.58	24.67	2.46	24.33
Expert 3	6.46	24.33	6.09	24.33
Expert 4	1.68	22.00	4.83	16.33
Expert 5	7.79	25.00	9.10	20.00