Question about truncation (Problem with dimensionality)

[Palamabron]

Hi, I'm trying to implement TQC to play game called Trackmania using custom version of Tmrl which is available here. While trying to implement your code I encountered this problem, which I cannot solve due to lack of my knowledge of this subject.
(...) backup = r + (1 - d) * self.gamma * (sorted_z_part - alpha * next_log_pi) RuntimeError: The size of tensor a (46) must match the size of tensor b (128) at non-singleton dimension 1
And this is my fragment of code with tensor sizes in the comments:

# in this case batch size=128 and 2*[quantiles - top_quantiles_to_drop] = 46
with torch.no_grad():
            new_next_action, next_log_pi = self.model.actor(o2)  # Tensor(batch x 3), Tensor(batch)
            # Compute and cut quantiles at the next state
            next_q1 = self.model_target.q1(o2, new_next_action)  # Tensor(batch x quantiles)
            next_q2 = self.model_target.q2(o2, new_next_action)  # Tensor(batch x quantiles)
            next_z = torch.stack((next_q1, next_q2), dim=1)  # Tensor(batch x nets x quantiles)
            sorted_z, _ = torch.sort(next_z.reshape(batch_size, -1))
            sorted_z_part = sorted_z[:, :self.quantiles_total - self.top_quantiles_to_drop] # Tensor(batch x 2*[quantiles - top_quantiles_to_drop])
            # alpha: Tensor (1,)
            backup = r + (1 - d) * self.gamma * (sorted_z_part - alpha * next_log_pi)

While 2*[quantiles - top_quantiles_to_drop] > 1 my code cannot run and I don't how to fix it. Thanks for any help in advance. :)

[brickerino]

Check all the dimensions. If you are unsure what they should be, try running a Mujoco env.

On Wed, 30 Aug 2023 at 00:15 Jakub Szulc ***@***.***> wrote: Hi, I'm trying to implement TQC to play game called Trackmania using custom version of Tmrl <https://github.com/trackmania-rl/tmrl/tree/master> which is available here <https://github.com/Pheoxis/AITrackmania/tree/main>. While trying to implement your code I encountered this problem, which I cannot solve due to lack of my knowledge of this subject. (...) backup = r + (1 - d) * self.gamma * (sorted_z_part - alpha * next_log_pi) RuntimeError: The size of tensor a (46) must match the size of tensor b (128) at non-singleton dimension 1 And this is my fragment of code with tensor sizes in the comments: # in this case batch size=128 and 2*[quantiles - top_quantiles_to_drop] = 46 with torch.no_grad(): new_next_action, next_log_pi = self.model.actor(o2) # Tensor(batch x 3), Tensor(batch) # Compute and cut quantiles at the next state next_q1 = self.model_target.q1(o2, new_next_action) # Tensor(batch x quantiles) next_q2 = self.model_target.q2(o2, new_next_action) # Tensor(batch x quantiles) next_z = torch.stack((next_q1, next_q2), dim=1) # Tensor(batch x nets x quantiles) sorted_z, _ = torch.sort(next_z.reshape(batch_size, -1)) sorted_z_part = sorted_z[:, :self.quantiles_total - self.top_quantiles_to_drop] # Tensor(batch x 2*[quantiles - top_quantiles_to_drop]) # alpha: Tensor (1,) backup = r + (1 - d) * self.gamma * (sorted_z_part - alpha * next_log_pi) While 2*[quantiles - top_quantiles_to_drop] > 1 my code cannot run and I don't how to fix it. Thanks for any help in advance. :) — Reply to this email directly, view it on GitHub <#7>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFY4TBXFYWGF4HUB4I3QHTXXZEW3ANCNFSM6AAAAAA4DPKGOY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Best regards, Arseny Kuznetsov Mob. +79852771410