Evaluation of attention map generating algorithms. (a) Input images from three classes used for evaluation (top row). Numbers in parentheses indicate the class label of the dataset. Lesions in the image were manually annotated (bottom row). (b)-(e) Attention map generating methods applied to each image and displayed as a heatmap over the input. See Materials and Methods for details. (b) Perturbation-based visualization. (c) Gradient-based visualization. (d) Grad-CAM visualization. (e) Reference-based visualization. For Grad-CAM and explanation map, the layers of which the gradient and the intermediate output were used are indicated.