cannot find spatial branch in model code

[inderpreet-adapdix]

Hi,

I was looking into model implementation and found that Fusion_block in seaformer.py file is not returning anything in its forward pass

SeaFormer/seaformer-cls/seaformer.py

Lines 338 to 368 in db38fe7

	class Fusion_block(nn.Module):
	def __init__(
	self,
	inp: int,
	oup: int,
	fuse_type='bi',
	norm_cfg=dict(type='BN', requires_grad=True),
	activations=None,
	) -> None:
	super(InjectionMultiSum, self).__init__()
	self.norm_cfg = norm_cfg
	self.local_embedding = ConvModule(inp, oup, kernel_size=1, norm_cfg=self.norm_cfg, act_cfg=None)
	self.global_act = ConvModule(oup, oup, kernel_size=1, norm_cfg=self.norm_cfg, act_cfg=None)
	self.act = h_sigmoid()
	self.fuse_type = fuse_type
	def forward(self, x_l, x_g):
	'''
	x_g: global features
	x_l: local features
	'''
	B, C, H, W = x_l.shape
	B, C_c, H_c, W_c = x_g.shape
	local_feat = self.local_embedding(x_l)
	global_act = self.global_act(x_g)
	sig_act = F.interpolate(self.act(global_act), size=(H, W), mode='bilinear', align_corners=False)
	global_feat = F.interpolate(global_feat, size=(H, W), mode='bilinear', align_corners=False)
	out_detail = local_feat * sig_act

also there seems to be error in its implementation:

• global_feat is referenced before assignment,
• name InjectionMultiSum is used as the name of class while initializing super class

While debugging the forward pass for seaformer-small model I could not find the use of the Fusion block and model weights for fusion block are also not present in checkpoint. It seems like model implementation only has shared stem and context branch.

Can you please help me with these issues if I m missing something?

Thanks

[wwqq]

Hi, @inderpreet-adapdix Fusion_block (Spatial branch) is not used in the classification task. We've deleted this class. It is referenced in the segmentation code.

SeaFormer/seaformer-seg/mmseg/models/decode_heads/light_head.py

Line 69 in f64a6b5

for i in range(len(embed_dims)):

[inderpreetsingh01]

Thanks for the reply @wwqq, I am actually using CPU and was able to get inference done on CPU for a classification model, I am not sure how to do it for the segmentation model, can you help me with it I want to evaluate seaformer-small for segmentation on CPU.

[wwqq]

The convert file is uploaded. https://github.com/fudan-zvg/SeaFormer/blob/main/convert2onnx.py
run: python3 convert2onnx.py <config-file> --input-img <img-dir> --shape 512 512 --checkpoint <model-ckpt> to convert model to ONNX.
test inference speed in mobile devices, please refer to TopFormer

[inderpreetsingh01]

Hi @wwqq I followed the link you mentioned, I am doing inference for seaformer small model in jupyter notebook, I am using model SeaFormer-S_512x512_4x8_160k, I have initialized the model using config present in seaformer_small.py and loaded the weights. I am passing it an image from ADE20K dataset after resizing it to (512, 512) and result I am getting is of (64, 64). I have done bilinear interpolation to convert into size (512, 512) and mapped each class to a RGB color, the output image I am getting doesn't seem to be good and it seems image is not getting segmented. I have updated Conv2d_BN to use nn.BatchNorm2d instead of build_layer_norm.
Here is the link to notebook https://github.com/inderpreetsingh01/models/blob/master/Seaformer_segmentation_inference.ipynb
It would be helpful if you can help me with this, if I am doing something wrong here?

Thanks

[wwqq]

Hi @inderpreetsingh01
1.Logits need to be upsampled first and then argmax.

2.Label_colors need to generate 150 classes not 125. You can import it from the checkpoint directly.

3.Same here, nc=150. And rgb_image should transpose not reshape.

Final vis result:

[inderpreetsingh01]

Thanks a lot, @wwqq for the clear explanation. I have one general doubt why the output seems a bit noisy, is there any way to improve it for the same checkpoint?

[speedinghzl]

Hi @inderpreetsingh01, thanks for your interest in our work.
I saw you use the simple transform as the image preprocess, it is mismatched with the preprocess used in the Seaformer training step. The segmentation result could be better if you align the preprocess, including normalization, rgb order(not sure).

[inderpreetsingh01]

Hi @speedinghzl thanks for reply, I have normalized the image now using mean=[123.675, 116.28, 103.53] and std=[58.395, 57.12, 57.375], also I was using model in training mode I converted it to eval mode and below are the output I got.

ADE_val_00001001.jpg

ADE_val_00001005.jpg

It seems better than the previous one and less noisy.
Can you please confirm if is this the expected output of the seaformer small model?

[wwqq]

Hi, @inderpreetsingh01 Yes, they are the expected outputs.