Have the ability to edit .pkg.slp files while keeping embedded images and metadata

[auesro]

Bug description

When moving a slp file to another computer, some slp project files require to link each of the videos in the dataset while others do not.

Expected behaviour

Either all or none of the slp projects should require the videos linked in the Videos tab. Or at least to clarify somewhere how we can use the slp file itself to provide those paths.

Actual behaviour

Some slp files ask for the videos to be re-linked while others show the path to the actual slp file for each of the videos originally linked in the Videos tab (see screenshots). This is something specially weird because when I try to use the slp path itself for more than 1 video it shows an error message (see second screenshot).

Your personal set up

• OS:
  Linux-5.4.0-131-generic-x86_64-with-debian-bullseye-sid (Linux Mint 20.2)
• Version(s):
  SLEAP: 1.2.8
  TensorFlow: 2.6.3
  Numpy: 1.19.5
  Python: 3.7.12
• SLEAP installation method (listed here):
  • Conda from package
  • Conda from source
  • pip package
  • M1 Macs

Environment packages

# packages in environment at /home/auesro/mambaforge/envs/Sleap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   0.15.0                   pypi_0    pypi
aom                       3.3.0                h27087fc_1    conda-forge
astunparse                1.6.3                    pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
backports-zoneinfo        0.2.1                    pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4                    pypi_0    pypi
cattrs                    1.1.1                    pypi_0    pypi
certifi                   2021.10.8                pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
clang                     5.0                      pypi_0    pypi
colorama                  0.4.5                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
cuda-nvcc                 11.3.58              h2467b9f_0    nvidia
cudatoolkit               11.3.1               ha36c431_9    nvidia
cudnn                     8.2.1.32             h86fa8c9_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
efficientnet              1.0.0                    pypi_0    pypi
ffmpeg                    5.0.1                h594f047_1    conda-forge
flatbuffers               1.12                     pypi_0    pypi
fonttools                 4.37.1                   pypi_0    pypi
freetype                  2.10.4               h0708190_1    conda-forge
gast                      0.4.0                    pypi_0    pypi
geos                      3.9.1                h9c3ff4c_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               1.35.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.44.0                   pypi_0    pypi
h5py                      3.1.0           nompi_py37h1e651dc_100    conda-forge
hdf5                      1.10.6          nompi_h3c11f04_101    conda-forge
hdmf                      3.4.2                    pypi_0    pypi
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.3                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imageio                   2.15.0                   pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        4.11.1                   pypi_0    pypi
importlib-resources       5.9.0                    pypi_0    pypi
jbig                      2.1               h7f98852_2003    conda-forge
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9e                   h166bdaf_1    conda-forge
jsmin                     3.0.1                    pypi_0    pypi
jsonpickle                1.2                      pypi_0    pypi
jsonschema                4.16.0                   pypi_0    pypi
keras                     2.6.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0                8_openblas    conda-forge
libcblas                  3.9.0                8_openblas    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libdrm                    2.4.109              h7f98852_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            7.5.0               h14aa051_20    conda-forge
libgfortran4              7.5.0               h14aa051_20    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0                8_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.12          pthreads_hb3c22a3_1    conda-forge
libpciaccess              0.16                 h516909a_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_16    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libva                     2.14.0               h7f98852_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.13               h22db469_0    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markdown                  3.3.6                    pypi_0    pypi
matplotlib                3.5.3                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ndx-pose                  0.1.1                    pypi_0    pypi
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  2.6.3                    pypi_0    pypi
numpy                     1.19.5           py37h3e96413_3    conda-forge
oauthlib                  3.2.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
opencv-python             4.5.5.62                 pypi_0    pypi
opencv-python-headless    4.5.5.62                 pypi_0    pypi
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   3.0.5                h166bdaf_2    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
pillow                    8.4.0            py37h0f21c89_0    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
protobuf                  3.19.4                   pypi_0    pypi
psutil                    5.9.2                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pykalman                  0.9.5                    pypi_0    pypi
pynwb                     2.1.0                    pypi_0    pypi
pyparsing                 3.0.6                    pypi_0    pypi
pyrsistent                0.18.1                   pypi_0    pypi
pyside2                   5.14.1                   pypi_0    pypi
python                    3.7.12          hf930737_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-rapidjson          1.8                      pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pytz-deprecation-shim     0.1.0.post0              pypi_0    pypi
pywavelets                1.3.0                    pypi_0    pypi
pyzmq                     23.2.1                   pypi_0    pypi
qimage2ndarray            1.9.0                    pypi_0    pypi
qtpy                      2.2.1              pyhd8ed1ab_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rich                      10.16.1                  pypi_0    pypi
ruamel-yaml               0.17.21                  pypi_0    pypi
ruamel-yaml-clib          0.2.6                    pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scikit-video              1.1.11                   pypi_0    pypi
scipy                     1.5.3            py37h8911b10_0    conda-forge
seaborn                   0.12.0                   pypi_0    pypi
segmentation-models       1.0.1                    pypi_0    pypi
setuptools                59.8.0           py37h89c1867_1    conda-forge
setuptools-scm            6.4.2                    pypi_0    pypi
shapely                   1.7.1            py37h48c49eb_5    conda-forge
shiboken2                 5.14.1                   pypi_0    pypi
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sleap                     1.2.8                    pypi_0    pypi
sqlite                    3.38.2               h4ff8645_0    conda-forge
svt-av1                   0.9.1                h27087fc_0    conda-forge
tensorboard               2.6.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.6.3                    pypi_0    pypi
tensorflow-estimator      2.6.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2021.11.2                pypi_0    pypi
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1                    pypi_0    pypi
typing-extensions         3.10.0.2                 pypi_0    pypi
tzdata                    2022.2                   pypi_0    pypi
tzlocal                   4.2                      pypi_0    pypi
urllib3                   1.26.8                   pypi_0    pypi
werkzeug                  2.0.3                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.12.1                   pypi_0    pypi
x264                      1!161.3030           h7f98852_1    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.7.0                    pypi_0    pypi
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge

Screenshots

[roomrys]

Incorrect analysis below, see future comments for actual problem.

The .slp files that link their own file location for the video paths should only be happening in .pkg.slp files which contain the "video" information (really just a collection of training images).

I did not take this into account when adding the pop-up requiring unique video names for replacing videos in #787.

A couple of proposed fixes would be to:

1. allow non-unique video paths for .pkg.slp files
2. automatically find videos for .pkg.slp's (should be the path to the .pkg.slp)
3. reference videos in better way: distinguish actual object not just the filename (this allows duplicate videos to exist, although not sure why anyone would want duplicates?)

I am in favor of 2.

[auesro]

Hi @roomrys
Thanks for the info.
As you can see in the screenshot, it is a pkg.slp file, we exported from a different computer (Windows 10) and moved it to a Linux machine (where I took those screenshots).
I have been playing a bit with this, several observations:

-When you use Predict->Export Labels Package..., you get a .pkg.slp file that uses self-reference as video paths.
-When you use Save or Save as... from a pkg.slp file, you get a copy.slp file that has lost all paths to original videos and is not possible to open it without re-linking the original videos.

Given this, why not allow the Save function to keep the info from the pkg.slp file regarding self-reference for the video paths? To me, it has been useful to have the pkg.slp file to test different models and parameters in a different (more powerful) computer, without the need to have the original videos. And it would be even more useful if I could make corrections to the labeled frames (adjust some frames, add nodes or edges, etc), be able to save the file and re-train without the need to have the videos linked (anyway I am using already a bunch of frames).

Cheers,

A

[roomrys]

Hi @auesro,

You are correct; I overlooked the extension earlier - these are indeed normal .slp files with the video paths referencing the original .pkg.slp, but expecting direct links to the video(s). The original (and current) purpose of .pkg.slp file is to allow remote training (i.e. transportation, not editing).

I could speculate more than a few reasons for why you would want to edit the .pkg.slp, but it might be better to ask: why do you prefer to edit the .pkg.slp instead of the original .slp? Also, what would you want to be able to edit (i.e. just allow adding more instances)?

Thanks,
Liezl

P.S. I removed the bug label and added the enhancement label since it is not an error, but a feature we have yet to add. The TODO would be to raise an error or warning message when trying to save the .pkg.slp warning the user of the unexpected behavior: creation of a normal .slp. I would rather open a PR to add a save feature for .pkg.slps (this will be larger: need to disable certain features in GUI s.a. adding additional videos) than add a PR for the warning message.

[auesro]

Hi @roomrys

Thanks for the detailed explanaiton.

In our personal case, we are working together with @isabelperezf in the same project. She has labeled more than 1000 frames from 20 different source videos using her computer. Now I wanted to test several different models and parameters (my computer is faster) so I took the pkg.slp file. And for this purpose it worked perfectly.
But then I wanted to correct some instances, at some point also to add a couple new nodes/edges, etc. We consider we have a rich enough dataset and we are just optimizing it. In addition, the idea was to have someone else label exactly the same frames in order to get an idea of human-induced variability. Not having to deal with copying of the videos from computer to computer is a big advantage in my opinion. The only time when I would require to have the videos handy is when predicting/labeling new frames.
I believe an "easy" option would be to add the option to incorporate the videos to the pkg.slp when exporting from the GUI (just as you can select what frames to export). It would result in a bigger file, sure, but being optional it would only affect those wanting to do it. Otherwise, having the option to edit the pkg.slp file and saving it still as a pkg.slp file would be ideal, just not sure how hard that would be to achieve.

Hope it is kind of clear, if not please, let me know!

Great job both at the software development and "customer" support front!

Cheers,

A

[talmo]

Hi @auesro,

Thanks for the detailed description of the use case.

As @roomrys outlined above, we do not currently support editing .pkg.slp files while retaining the embedded images. This is definitely something we could add, but it'll be a bit non-trivial largely because of how we abstract our serialization and data structures. Namely, we currently re-save all of the data in the labels file, which in the case of .pkg.slp files would require all of the images to be re-exported -- not ideal.

The corrections use case is a tricky one. What's practical would be to be able to edit the skeleton and instances, but not add/remove labeled frames, but we still need to refactor some stuff to deal with the re-saving issue mentioned above.

The use case of having multiple people label the same images (without having to keep the videos together) makes a lot of sense and is definitely one which we would like to support.

We'll add this to our roadmap, but in the meantime, I'll mention a couple of pieces of functionality that are relevant here:

• You can choose which frames will be embedded by adding them as Suggestions and then exporting them explicitly (Predictions -> Export package -> Labels + suggestions). This will include frames that are not labeled as well as the ones that are.
• If you can pull out the images that you want to have multiple people reannotate, you can import them as a SingleImageVideo and only keep a folder of the images together with the .slp project file together, rather than the whole video.

The latter might be the best workaround for now. If you're interested, let us know and we can try and give you a notebook example showing how to set something like this up.

Cheers,

Talmo

[auesro]

Thanks for the input, @talmo, I will consider the options and get back to you
Cheers,
A