Table 2 Voxel-level NSCLC classification performance. Mean values and 95% confidence intervals are provided. Calibration metrics were not computable for GradCAM, integrated gradients and occlusion sensitivity methods. The clinician preference rate was calculated as the frequency with which clinicians preferred the method in 100 test instances, excluding instances in which they considered no model informative. DenseNet predictions were not included in the clinician preference test
Partition | Method | Precision | Recall | Dice | ECE | AUPR | Clinician preference rate |
|---|---|---|---|---|---|---|---|
Validation | WSUnet | 0.77 [0.75–0.8] | 0.33 [0.29–0.36] | 0.43 [0.39–0.46] | 0.02 [0.01–0.02] | 0.53 [0.49–0.56] | NA |
Validation | sCNN GradCAM (16, 16, 64) | NA | NA | NA | NA | 0.28 [0.25–0.3] | NA |
Validation | sCNN GradCAM (8, 8, 128) | NA | NA | NA | NA | 0.27 [0.25–0.29] | NA |
Validation | sCNN integrated gradients | NA | NA | NA | NA | 0.1 [0.09–0.1] | NA |
Validation | sCNN occlusion sensitivity | NA | NA | NA | NA | 0.04 [0.03–0.04] | NA |
Validation | DenseNet GradCAM (16, 16, 64) | NA | NA | NA | NA | 0.19 [0.17–0.21] | NA |
Validation | DenseNet GradCAM (8, 8, 128) | NA | NA | NA | NA | 0.23 [0.21–0.25] | NA |
Validation | DenseNet occlusion sensitivity | NA | NA | NA | NA | 0.03 [0.02–0.03] | NA |
Test | WSUnet | 0.78 [0.76–0.81] | 0.24 [0.22–0.25] | 0.33 [0.32–0.35] | 0.01 [0.01–0.02] | 0.4 [0.38–0.41] | 0.72 [0.68–0.77] |
Test | sCNN GradCAM (16, 16, 64) | NA | NA | NA | NA | 0.36 [0.34–0.37] | 0.2 [0.16–0.24] |
Test | sCNN GradCAM (8, 8, 128) | NA | NA | NA | NA | 0.23 [0.21–0.24] | 0.05 [0.03–0.08] |
Test | sCNN integrated gradients | NA | NA | NA | NA | 0.11 [0.1–0.11] | 0.01 [0.0–0.03] |
Test | sCNN occlusion sensitivity | NA | NA | NA | NA | 0.03 [0.03–0.03] | 0.0 [0.0–0.01] |
Test | DenseNet GradCAM (16, 16, 64) | NA | NA | NA | NA | 0.13 [0.12–0.14] | NA |
Test | DenseNet GradCAM (8, 8, 128) | NA | NA | NA | NA | 0.23 [0.22–0.25] | NA |
Test | DenseNet occlusion sensitivity | NA | NA | NA | NA | 0.02 [0.02–0.02] | NA |