Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard

Abhinav Jha just sent me the following intriguing email

Hello Igor,

My name is Abhinav Jha, Research Fellow, Department of Radiology, Johns Hopkins University. I have visited your blog on many occasions. It is a great source of information and I would like to thank you for all your efforts.

I write this mail to let you know about one of my publications on task-based evaluation of image analysis algorithms in the absence of a gold standard. This work is not directly related to compressive sensing (CS), but I like to view CS as task-specific imaging (on similar lines as Dr. Mark Neifelds approach). In this context, it becomes essential to evaluate CS systems and algorithms based on the task for which the image has been acquired, and that is where my work comes in. What I suggest is a paradigm to evaluate image analysis algorithms based on the task for which the image has been acquired. Also, often in these systems, we do not know of the gold standard, and therefore, the evaluation task becomes complicated. The technique that my paper suggests can work in the absence of a gold standard. Here is a general summary of the paper

With the development of task-specific imaging systems and algorithms, there is a requirement to evaluate these systems/algorithms based on the task for which they were designed. The standard evaluation methodologies are often incapable in performing this task-based evaluation. In this paper, we suggest a framework to perform the task-based evaluation of image analysis algorithms, where we specifically target the problem of evaluating segmentation algorithms. Evaluation of these image-analysis algorithms may require a gold standard to compare against, but that is often unavailable. The evaluation technique that we suggest takes this issue into account.

Please let me know if you need any other documents.

Thanks,

Abhinav

Website: www.optics.arizona.edu/ajha

Thanks Abhinav

Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard by Abhinav K. Jha, Matthew A. Kupinski, Jeffrey J. Rodrıguez, Renu M. Stephen and Alison T. Stopeck

In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.

Join the CompressiveSensing subreddit or the Google+ Community and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.