Unable to upload a training config to run instance

[PaulaNeuro]

I’m trying to upload a training configuration file that was created on macOS to SLEAP on a Windows computer. I'm not sure the file uploads successfully, it doesn’t show up under the inference settings. When I attempt to run inference, I receive the error: Unable to run: cannot run inference with untrained models. Could someone guide me on how to properly upload and use this training config file for inference in SLEAP on Windows? I’d appreciate any help or tips.

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[gitttt-1234]

Hi @PaulaNeuro,

Are you able to choose the right training_config.json in the drop down?

If training_config.json is loaded properly, then the issue could be that the GUI is not able to find the trained models\ folder (This is the dir where ckpts for each model type are saved. Each model type has a separate folder in which the checkpoints, configs and the output files would be saved after training). Could you check if the training_config.json is in the models\<your-run-ckpt-name> folder?

Thanks,

Divya

[PaulaNeuro]

Thank you for your answer Divya, It worked!
But now I'm having another issue, when I run the inference I get this:

I tried the exact same thing in a mac (where the training was originally done) instead of a windows and it worked there, we had the same folder organization and files. I don't know what's happening.
I would really appreciate some help
Thank you!

[gitttt-1234]

Hi @PaulaNeuro,

Just to confirm, you have only one frame on your test data? and you have both your checkpoints and config file inside models\<your-run-name> folder? If yes for both, then it seems like the inference runs successfully but the model is not able to predict anything from the data. Would you mind sharing the logs so that we could check on our end?
You can find the logs in the terminal where you started the GUI.

Thanks,

Divya

[PaulaNeuro]

Hi @gitttt-1234,

Thank you again for your answer! I forgot to run the inference in the entire video, I have a total of 15044 frames, but the result is still the same, no predictions. I don't really know what the checkpoints are, I thought they would be in the same file as the training config? These are the logs from the terminal:

System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True

Finished inference at: 2024-09-20 11:32:57.186568
Total runtime: 94.38584589958191 secs
Predicted frames: 0/15044
Provenance:
{
'model_paths': [
'C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/models/Final_model/training_config.json'
],
'predictor': 'SingleInstancePredictor',
'sleap_version': '1.3.1',
'platform': 'Windows-10-10.0.22621-SP0',
'command': 'C:\Users\pxdia\anaconda3\envs\sleap\Scripts\sleap-track C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/labels.v001.slp --video.index 0 --frames 0,-15043 -m C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/models/Final_model/training_config.json --tracking.tracker flow --tracking.pre_cull_to_target 0 --tracking.pre_cull_iou_threshold 0.8 --tracking.similarity instance --tracking.match hungarian --tracking.track_window 5 --tracking.save_shifted_instances 0 --tracking.post_connect_single_breaks 0 -o C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen\predictions\labels.v001.slp.240920_113118.predictions.slp --verbosity json --no-empty-frames',
'data_path': 'C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/labels.v001.slp',
'output_path': 'C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen\predictions\labels.v001.slp.240920_113118.predictions.slp',
'total_elapsed': 94.38584589958191,
'start_timestamp': '2024-09-20 11:31:22.800722',
'finish_timestamp': '2024-09-20 11:32:57.186568'
}

Saved output: C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen\predictions\labels.v001.slp.240920_113118.predictions.slp

Process return code: 0

I really appreciate your help!
Paula.

[gitttt-1234]

Hi @PaulaNeuro ,

Sorry , the best_model.h5 (or any .h5 file) is the checkpoint file. Would you mind sharing your test video and the models folder as a zip file here so that we can debug from our side?

Thanks,

Divya

[PaulaNeuro]

Oh, then yes, I have it in the same folder as the config file.

I'm attaching the models folder and the test video as a link because it was too big.
model folder.zip
https://rutgersconnect-my.sharepoint.com/:u:/g/personal/pd580_gsbs_rutgers_edu/EeESddRPYW9Imy1pFDzuKzMBiN381vrwKIxcyKox9lTGvA?e=NHXFpd

Let me know if this works,
Thank you!

Paula

[gitttt-1234]

Hi @PaulaNeuro,

From your logs, I see that you're running inference on a .slp file: labels.v001.slp but I don't see this in the list of files you shared. Are you trying to run inference on the HL268_Novel-L.mp4? And just to confirm you were able to get the predictions for the same input video file and ckpts in mac?

'command': 'C:\Users\pxdia\anaconda3\envs\sleap\Scripts\sleap-track C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/labels.v001.slp --video.index 0 --frames 0,-15043 -m C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/models/Final_model/training_config.json --tracking.tracker flow --tracking.pre_cull_to_target 0 --tracking.pre_cull_iou_threshold 0.8 --tracking.similarity instance --tracking.match hungarian --tracking.track_window 5 --tracking.save_shifted_instances 0 --tracking.post_connect_single_breaks 0 -o C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen\predictions\labels.v001.slp.240920_113118.predictions.slp --verbosity json --no-empty-frames',
'data_path': 'C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen/labels.v001.slp',
'output_path': 'C:/Users/pxdia/OneDrive/Documents/Paula/Entire trained folder/2-cup_infgen\predictions\labels.v001.slp.240920_113118.predictions.slp',

[PaulaNeuro]

Hi @gitttt-1234!
Oh yes, should I send the .slp file to you? Yes, I am trying to run it on the HL268_Novel-L.mp4 video.
And also, yes, I can get predictions for the same video and checkpoints on a Mac where the model was originally trained.

Thank you so much for your help!

Paula.

[gitttt-1234]

Hi @PaulaNeuro,

Would you mind sharing some of the source videos used for the training and validation slp files to compare it with the test video you shared? If you're trying to run inference only on the video, then we don't need the .slp file. Thanks!

Thanks,

Divya

[PaulaNeuro]

Hi @gitttt-1234!

Here it is,

2cup_proj_1.v020.zip
https://rutgersconnect-my.sharepoint.com/:v:/g/personal/pd580_gsbs_rutgers_edu/EeOg6CHQTm9Ntf_smaLqNtIBod91YRNf9Ytv4R8XnLP2DQ?nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJPbmVEcml2ZUZvckJ1c2luZXNzIiwicmVmZXJyYWxBcHBQbGF0Zm9ybSI6IldlYiIsInJlZmVycmFsTW9kZSI6InZpZXciLCJyZWZlcnJhbFZpZXciOiJNeUZpbGVzTGlua0NvcHkifX0&e=Mx7w1g

Let me know if you need anything else,

Paula.

[gitttt-1234]

Hi @PaulaNeuro,

Thank you for providing all the data! We did look into your training files and noticed a couple of things: your training OKS mAP appears to be very low (your model needs more learning to get better performance). Another reason why you're not able to get any inference on your test video is because of domain shift, where a model fails to predict on a new image which is significantly different from the images it was trained on.

These are some of the sample images from your videos. There's a sufficient change in illumination and the camera angle across both the images. This can also be seen from the distribution of pixels in the density plot.

I have linked a previous issue which addresses this problem: #1105. There are a couple of fixes provided in the discussion. Let us know if it works for you!

Thanks,

Divya