is it possible to use the provided cams with own models?

[cherepanovic]

Hello torch-cam providers :)

it is possible for example to train an own CNN and then use the provided CAM methods? If yes, in which way, what should be considered?

Thanks!

[frgfm]

Hello there @cherepanovic 👋

This is entirely possible, the whole project was meant for people willing to check class activation maps on their custom CNN architecture 👍

To do this, I'd suggest checking the example in the README: https://github.com/frgfm/torch-cam#retrieving-the-class-activation-map
You'll only have to change the model definition from model = resnet18(pretrained=True).eval() to your custom architecture definition (+ loading your trained parameters)

Let me know if it's still confusing 👍

[khawar-islam]

@frgfm, Please give me a example of our own CNN model. I have saved my model without dictionary and I tried a lot to apply class activation map but still it didn't work out.

model_path = '/media/cvpr/CM_1/tutorials/mobile_vit/best.pt'
model = torch.load(model_path)
# model = mobilevit_xs()
# model.load_state_dict(checkpoint)
# model.eval()
cam_extractor = GradCAM(model)
# Get your input
img = read_image(img_path)
# Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

Traceback (most recent call last):
  File "/media/cvpr/CM_1/tutorials/pytorch_class_activation.py", line 16, in <module>
    model = torch.load(model_path)
  File "/home/cvpr/anaconda3/envs/tutorials/lib/python3.8/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/cvpr/anaconda3/envs/tutorials/lib/python3.8/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'mobilevit_pytorch'

[frgfm]

Hi @khawar-islam 👋

Thanks for your question, but I cannot help you with this. As shown in the error traceback, the problem arises on this line

model = torch.load(model_path)

which has nothing to do with TorchCAM 😅
So I'd suggest to make sure your model properly loads before using TorchCAM :)

[khawar-islam]

@frgfm thank you, I have solved above problem and I am using GradCAM for class activation map but i think it didn't work correctly. I have attached a link to visualize image. The accuracy of my model is 82%

Working Code

import matplotlib.pyplot as plt
import torch
from torch.nn.functional import softmax, interpolate
from torchvision.io.image import read_image
from torchvision.models import resnet18
from torchvision.transforms.functional import normalize, resize, to_pil_image

from torchcam.methods import SmoothGradCAMpp, LayerCAM, GradCAM
from torchcam.utils import overlay_mask

from mobile_vit.mobilevit_pytorch import mobilevit_xs

img_path = "/media/cvpr/CM_1/test/1/image_06734.jpg"
model_path = '/media/cvpr/CM_1/tutorials/mobile_vit/best.pth'

model = mobilevit_xs()
model.load_state_dict(torch.load(model_path), strict=False)
model.eval()

cam_extractor = GradCAM(model)
# Get your input
img = read_image(img_path)
# Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

# Preprocess your data and feed it to the model
out = model(input_tensor.unsqueeze(0))
# Retrieve the CAM by passing the class index and the model output
cams = cam_extractor(out.squeeze(0).argmax().item(), out)

# Notice that there is one CAM per target layer (here only 1)
for cam in cams:
    print(cam.shape)

# The raw CAM
for name, cam in zip(cam_extractor.target_names, cams):
    plt.imshow(cam.numpy())
    plt.axis('off')
    plt.title(name)
    plt.show()

# Overlayed on the image
for name, cam in zip(cam_extractor.target_names, cams):
    result = overlay_mask(to_pil_image(img), to_pil_image(cam, mode='F'), alpha=0.5)
    plt.imshow(result)
    plt.axis('off')
    plt.title(name)
    plt.show()

Results
https://ibb.co/TkRvhcg

[frgfm]

I'm still working on improving the compatibility with Vision Transformer so that would be the reason :)
To explain a bit the challenge:

• the raw CAM is computed to have the same shape as a given layer's output feature map
• but with Transformer layer, the spatial shape is significantly modified

I've already made progress (it's easy to make it work for one architecture, the problem is when you want it to work with almost everything 😅 )

[khawar-islam]

@frgfm thank you for work and time. Yes, "the problem is when you want it to work with almost everything" you are right