FusedBiasAct Native Pytorch Implementation

[mandelapatrick]

Hi! Great job on this repo - well done! I am trying to crate a native Pytorch implementation of the C++ function fused.fused_bias_act and I was wonder if you can help me. My attempt is below.
Mostly I am not clear what bias, refer, act, grad and scale refer to in the C++ implementation.

def fused_bias_act(inp, bias, refer, act, grad, negative_slope, scale):
noise = bias + refer
fused_act = torch.nn.functional.leaky_relu(inp + noise, negative_slope=negative_slope, inplace=False) * scale
return fused_act

[rosinality]

It will be like this:

def fused_bias_act(inp, bias, negative_slope=0.2, scale=2 ** 0.5):
    fused_act = torch.nn.functional.leaky_relu(inp + bias, negative_slope=negative_slope, inplace=False) * scale

    return fused_act

[mandelapatrick]

Thanks so much for your response! I am having one issue though. When doing a forward pass through the mode, inp=torch.Size([16, 512, 4, 4]) and bias=torch.Size([512]) during the forward pass of StyledConv on line 357, so it can't do the addition. Any suggestions to fix this?

[rosinality]

You will need to use .view to add axes to bias vector.

[Animadversio]

@rosinality Currently I used this .view statement

def fused_leaky_relu(input, bias, negative_slope=0.2, scale=2 ** 0.5):
    return scale * F.leaky_relu(input + bias.view((1, -1)+(1,)*(len(input.shape)-2)), negative_slope=negative_slope)

Not sure if this is the most efficient one but it works on my side!
The (len(input.shape)-2)) is added since sometimes the input is 2d sometimes it's 4d

[silence-tang]

Thanks so much for your answers. It helps me a lot!
I have a question that will the PyTorch implementation of fused_leaky_relu affect the performance of the model or the results of the model negatively?