("Reference beats are empty.") mir_eval warning #5
Comments
|
Hello, Can you copy the complete console log, or at least a reference to the point of the code that is causing the problem I will in any case still try to guess where your problem is coming from. Let me know if this works! I would in any case not change something on the rotary encoding file, as I think this is just masking the issue instead of solving it. |
|
Full console log: ''' Seed set to 0 While executing %copy_ : [num_users=0] = call_method[target=copy_](args = (%l_self_buffers_cached_freqs_seq_len_, 32), kwargs = {}) The above exception was the direct cause of the following exception: Traceback (most recent call last): While executing %copy_ : [num_users=0] = call_method[target=copy_](args = (%l_self_buffers_cached_freqs_seq_len_, 32), kwargs = {}) Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information You can suppress this exception and fall back to eager by setting: Seed set to 1 While executing %copy_ : [num_users=0] = call_method[target=copy_](args = (%l_self_buffers_cached_freqs_seq_len_, 32), kwargs = {}) The above exception was the direct cause of the following exception: Traceback (most recent call last): While executing %copy_ : [num_users=0] = call_method[target=copy_](args = (%l_self_buffers_cached_freqs_seq_len_, 32), kwargs = {}) Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information You can suppress this exception and fall back to eager by setting: Seed set to 2 While executing %copy_ : [num_users=0] = call_method[target=copy_](args = (%l_self_buffers_cached_freqs_seq_len_, 32), kwargs = {}) The above exception was the direct cause of the following exception: Traceback (most recent call last): While executing %copy_ : [num_users=0] = call_method[target=copy_](args = (%l_self_buffers_cached_freqs_seq_len_, 32), kwargs = {}) Thanks for your attention! Right now, in my annotations/spectrograms folders, I have both the harmonix and the npz folders. The problem is that an integer (32) is being used where a tensor is expected, causing a type mismatch when RotaryEmbedding() is called in beat_track.py. I tried different library versions, but it doesn't seem to be a version conflict. |
|
Thank you! Could you have copied them to the wrong directory? |
|
Oh sorry, i meant the annotations and the spectrograms folders, separately. |
|
I see. Could you try to train without compiling the model? So maybe we have a more useful log. |
|
note that the train keeps running after that, but I've noticed that the reference beats are arriving empty to the mir_eval library. This seems to cause the metrics to drop significantly, and I suspect it might be due to a version discrepancy. Thank you so much for your assistance and understanding. I really appreciate it! |
|
No problem, I'm happy to help :) I'm using mir_eval 0.7, but mir_eval is not the problem. At the beginning of the training, the model still predicts random outputs, so it's normal that no beats are predicted for the first epochs. This is solved after some training. I'm more preoccupied with the cast problem you mentioned before. I think it is something related to the fact that you are only using 1 dataset instead of all of them. But I don't have time to test the code this week. |
|
Hey, only saw this now!
It is normal that mir_eval warns about empty predictions in the beginning, but it should not warn about empty references. Do you only get the
It seems that this is a legit finding that only manifests when compiling with a more recent PyTorch version. Apparently PyTorch is lenient about this in Python, but strict in Dynamo. Note that this is in lucidrains' rotary-embedding-torch module, not in our code. Looking at the source code, I wonder why
I'm attaching the |
|
Hey, using the command |
|
Good to hear that it works now! I'd still encourage you to send a pull request to rotary-embedding-torch since you found the bug there. The easiest option is to use the pencil button on the top right at https://github.com/lucidrains/rotary-embedding-torch/blob/main/rotary_embedding_torch/rotary_embedding_torch.py; this would create a fork in which you can edit the file, save as a commit and then send a pull request. With the changes outlined above, you will be able to compile the model. |
|
Sent a pull request to lucidrains now, as we need it ourselves: lucidrains/rotary-embedding-torch#38 |
I'm executing the first train example using only the harmonix dataset, but i've encoutered the following error:
RuntimeError: aten::copy() Expected a value of type 'Tensor' for argument 'src' but instead found type 'int'.
Position: 1
Value: 32
Declaration: aten::copy(Tensor self, Tensor src, bool non_blocking=False) -> Tensor
Cast error details: Unable to cast 32 to Tensor
investigating it, i found that changing this line(315) on the rottary_embedding_torch.py file the problem was "solved":
from the original:
self.cached_freqs_seq_len.copy_(seq_len)
to this:
self.cached_freqs_seq_len.copy_(torch.tensor(seq_len, dtype=torch.long, device=self.cached_freqs_seq_len.device))
But actually, it only allowed the train to continue running, but with the wrong tensor value.
I used the beat annotations on the beat_this repository and generated the spectrograms by the pre-processs script, it all seemed to work well in the pre-process step.
When i used the "fix" for the rottary_embedding_torch.py, i've encountered the following warning: ("Reference beats are empty.") mir_eval warning and investigating more deep into it, i found that in fact sometimes the estimated beats/ reference beats become empty at some point of the run
The text was updated successfully, but these errors were encountered: