metrics.npz file to be generated after training a SLEAP model

[ozgurcemozgen06]

Hi all,

I've been utilizing SLEAP for my M.Sc thesis on "Multi-Animal Keypoint and Pose Estimation using Deep Learning Algorithms". During my last usages(training) of SLEAP on some dataset, it does not create "metrics.npz" file under models directory. In my previous usages, no such thing happened but now it starts to not create that metrics file so I cannot reach the evaluation results of the trained model. I directly used "sleap-train" command inside the terminal and am wondering if I am missing something as an argument. What might be the reason for this situation? I would be so glad if you may guide me about it.

With my kindest regards,
Özgür

[roomrys]

Hi @ozgurcemozgen06,

The first reason why the metrics would not be output would be if your training_config.json has outputs.save_outputs set to False effectively passing over the evaluate function that outputs the metrics:

sleap/sleap/nn/training.py

Lines 947 to 955 in b5aab0e

	# Run post-training actions.
	if self.config.outputs.save_outputs:
	if (
	self.config.outputs.save_visualizations
	and self.config.outputs.delete_viz_images
	):
	self.cleanup()
	self.evaluate()

However, if you are using the CLI, then we should be setting this to True for you:

sleap/sleap/nn/training.py

Lines 1925 to 1926 in b5aab0e

	# Override config settings for CLI-based training.
	job_config.outputs.save_outputs = True

The second reason why the metrics would not be output is if we ran into an error in the evaluate_model function, either ValueError: Unrecognized model type (which I would also be surprised by):

sleap/sleap/nn/evals.py

Lines 675 to 708 in b5aab0e

	# Setup predictor for evaluation.
	head_config = cfg.model.heads.which_oneof()
	if isinstance(head_config, CentroidsHeadConfig):
	predictor = TopDownPredictor(
	centroid_config=cfg,
	centroid_model=model,
	confmap_config=None,
	confmap_model=None,
	)
	elif isinstance(head_config, CenteredInstanceConfmapsHeadConfig):
	predictor = TopDownPredictor(
	centroid_config=None,
	centroid_model=None,
	confmap_config=cfg,
	confmap_model=model,
	)
	elif isinstance(head_config, MultiInstanceConfig):
	predictor = BottomUpPredictor(bottomup_config=cfg, bottomup_model=model)
	elif isinstance(head_config, SingleInstanceConfmapsHeadConfig):
	predictor = SingleInstancePredictor(confmap_config=cfg, confmap_model=model)
	elif isinstance(head_config, MultiClassBottomUpConfig):
	predictor = BottomUpMultiClassPredictor(
	config=cfg,
	model=model,
	)
	elif isinstance(head_config, MultiClassTopDownConfig):
	predictor = TopDownMultiClassPredictor(
	centroid_config=None,
	centroid_model=None,
	confmap_config=cfg,
	confmap_model=model,
	)
	else:
	raise ValueError("Unrecognized model type:", head_config)

or Failed to compute metrics (which might be the most likely):

sleap/sleap/nn/evals.py

Lines 713 to 718 in b5aab0e

	# Compute metrics.
	try:
	metrics = evaluate(labels_reader.labels, labels_pr)
	except:
	logger.warning("Failed to compute metrics.")
	metrics = None

Did you happen to get any messages in the terminal that might help us determine the cause?

Thanks,
Liezl

[ozgurcemozgen06]

Hi @roomrys

At one of my trials, the training just stopped at iteration 7, without any cause and in my last trial I got a "killed" message just after getting "Creating dataset for training". It seems like it cannot complete training process

Thank you for your support,
Özgür

[ozgurcemozgen06]

Hi @roomrys,

When I run SLEAP in my local computer, i am getting a different error saying
ValueError: Cannot load file containing pickled data when allow_pickle=False

Thanks a lot,
Özgür

[roomrys]

Hi @ozgurcemozgen06,

That sounds like it might be a completely different issue. I have a feeling it has to do with the video loading, but can't be certain without the Traceback from your terminal. Do you mind providing the traceback - maybe we can start a new discussion for that topic?

Thanks,
Liezl

[ozgurcemozgen06]

Hi @roomrys

Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /home/students/oezgen/sleaponmarss/labels.v001.slp
Traceback (most recent call last):
  File "/home/students/oezgen/miniconda/envs/sleap/bin/sleap-train", line 33, in <module>
    sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-train')())
  File "/home/students/oezgen/sleap/sleap/nn/training.py", line 2013, in main
    trainer = create_trainer_using_cli(args=args)
  File "/home/students/oezgen/sleap/sleap/nn/training.py", line 2005, in create_trainer_using_cli
    video_search_paths=args.video_paths,
  File "/home/students/oezgen/sleap/sleap/nn/training.py", line 675, in from_config
    with_track_only=is_id_model,
  File "/home/students/oezgen/sleap/sleap/nn/training.py", line 152, in from_config
    with_track_only=with_track_only,
  File "/home/students/oezgen/sleap/sleap/nn/training.py", line 170, in from_labels
    training, search_paths=video_search_paths
  File "/home/students/oezgen/sleap/sleap/io/dataset.py", line 2731, in load_file
    return Labels.load_file(filename, search_paths, match_to=match_to)
  File "/home/students/oezgen/sleap/sleap/io/dataset.py", line 1966, in load_file
    filename, for_object="labels", video_search=video_search, *args, **kwargs
  File "/home/students/oezgen/sleap/sleap/io/format/main.py", line 113, in read
    return disp.read(filename, *args, **kwargs)
  File "/home/students/oezgen/sleap/sleap/io/format/dispatch.py", line 56, in read
    return adaptor.read(file, *args, **kwargs)
  File "/home/students/oezgen/sleap/sleap/io/format/hdf5.py", line 139, in read
    labels = cls.read_headers(file, video_search, match_to)
  File "/home/students/oezgen/sleap/sleap/io/format/hdf5.py", line 124, in read_headers
    labels = labels_json.LabelsJsonAdaptor.from_json_data(dicts, match_to=match_to)
  File "/home/students/oezgen/sleap/sleap/io/format/labels_json.py", line 386, in from_json_data
    videos = Video.cattr().structure(dicts["videos"], List[Video])
  File "/home/students/oezgen/miniconda/envs/sleap/lib/python3.7/site-packages/cattr/converters.py", line 201, in structure
    return self._structure_func.dispatch(cl)(obj, cl)
  File "/home/students/oezgen/miniconda/envs/sleap/lib/python3.7/site-packages/cattr/converters.py", line 336, in _structure_list
    for e in obj
  File "/home/students/oezgen/miniconda/envs/sleap/lib/python3.7/site-packages/cattr/converters.py", line 336, in <listcomp>
    for e in obj
  File "/home/students/oezgen/miniconda/envs/sleap/lib/python3.7/site-packages/cattr/converters.py", line 323, in structure_attrs_fromdict
    dispatch(type_)(val, type_) if type_ is not None else val
  File "/home/students/oezgen/miniconda/envs/sleap/lib/python3.7/site-packages/cattr/converters.py", line 407, in _structure_union
    return self._structure_func.dispatch(cl)(obj, cl)
  File "/home/students/oezgen/sleap/sleap/io/video.py", line 1541, in fixup_video
    return Video.make_specific_backend(cl, x)
  File "/home/students/oezgen/sleap/sleap/io/video.py", line 1523, in make_specific_backend
    return backend_class(**attribute_kwargs)
  File "<attrs generated init sleap.io.video.NumpyVideo>", line 3, in __init__
  File "/home/students/oezgen/sleap/sleap/io/video.py", line 538, in __attrs_post_init__
    self.__data = np.load(self.filename)
  File "/home/students/oezgen/miniconda/envs/sleap/lib/python3.7/site-packages/numpy/lib/npyio.py", line 445, in load
    raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False

This is what pops up in my terminal as I run sleap-train command

Thanks,
Özgür

[roomrys]

Hi @ozgurcemozgen06,

I opened PR #1327 for the above Traceback - if tests pass it should be merged soon.

Some background on the Traceback above

We have experience the above intermittently on SingleImageVideos backends (if your "video" consists of a series of images). I wonder if your video is also a series of images instead of an mp4 or avi? Basically, we use a library cattr that handles restructuring information in the slp back into the data classes used by SLEAP. We were allowing cattr to infer the data class by looking at its attributes, but occasionally cattr would infer incorrectly and attempt to construct the wrong video backend.

Thanks,
Liezl

[ozgurcemozgen06]

Hi @roomrys ,

I do not use video input, I have 15,000 images of frames of interacting mice

Thanks,
Ozgur