Comments on “Comparative Study With New Accuracy Metrics for Target Volume Contouring in PET Image Guided Radiation Therapy”

Phantom measurements with glass inserts in a hot background are still used frequently for calibration and performance assessment of volume delineation algorithms in PET. Taking as an example the recent paper by Shepherd et al.: Comparative Study with New Accuracy Metrics for Target Volume Contouring in PET Image Guided Radiation Therapy (IEEE Trans Med Imaging. 2012 Jun 4. [Epub ahead of print]), we demonstrate that this is not a valid approach due to the discontinuity introduced in the background by the cold walls of the glass inserts. We, moreover, emphasize that in order to define a sensible ground truth for performance assessment of contouring algorithms in patient data it is necessary to average over a sizable number of experienced observers and lesions in order to compensate for the substantial inter-observer variability.