Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate Sample Suggestions with less than number of sampled frames #773

Closed
1 of 4 tasks
eberrigan opened this issue Jun 2, 2022 · 7 comments
Closed
1 of 4 tasks
Assignees
Labels
bug Something isn't working

Comments

@eberrigan
Copy link
Contributor

eberrigan commented Jun 2, 2022

Bug description

Some of my videos only have 1 or 2 frames but most have 72. I would like to generate sample suggestions on everything with more than the number of samples I would like.

Expected behaviour

I generate (20) sample suggestions using the stride method. Only videos with more than 20 frames would have the sample suggestions generated.

Actual behaviour

Initializing sample suggestions window appears and value error is shown on command prompt. Nothing happens.

Your personal set up

Versions:
SLEAP: 1.2.1
TensorFlow: 2.8.0
Numpy: 1.21.5
Python: 3.7.13
OS: Windows-10-10.0.19041-SP0

System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True

Environment packages
# packages in environment at C:\Users\Elizabeth\.conda\envs\sleap_develop:
#
# Name                    Version                   Build  Channel
absl-py                   1.0.0                    pypi_0    pypi
alabaster                 0.7.12                   pypi_0    pypi
anyio                     3.5.0                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
argon2-cffi               21.3.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
atomicwrites              1.4.0                    pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
babel                     2.9.1                    pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
backports-zoneinfo        0.2.1                    pypi_0    pypi
beautifulsoup4            4.10.0                   pypi_0    pypi
black                     21.6b0                   pypi_0    pypi
bleach                    4.1.0                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0
cached-property           1.5.2                      py_0
cachetools                5.0.0                    pypi_0    pypi
cattrs                    1.1.1                    pypi_0    pypi
certifi                   2021.10.8        py37haa95532_2
cffi                      1.15.0                   pypi_0    pypi
cfgv                      3.3.1                    pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
click                     8.0.4                    pypi_0    pypi
colorama                  0.4.4                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
coverage                  6.3.2                    pypi_0    pypi
cuda-nvcc                 11.3.58              hb8d16a4_0    nvidia
cudatoolkit               11.3.1               h59b6b97_2
cudnn                     8.2.1                cuda11.3_0
cycler                    0.11.0                   pypi_0    pypi
debugpy                   1.6.0                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
deprecated                1.2.13                   pypi_0    pypi
distlib                   0.3.4                    pypi_0    pypi
docutils                  0.17.1                   pypi_0    pypi
efficientnet              1.0.0                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
ffmpeg                    4.2.2                he774522_0
filelock                  3.6.0                    pypi_0    pypi
flatbuffers               2.0                      pypi_0    pypi
fonttools                 4.31.2                   pypi_0    pypi
freetype                  2.10.4               hd328e21_0
gast                      0.5.3                    pypi_0    pypi
geos                      3.8.0                h33f27b4_0
gitdb                     4.0.9                    pypi_0    pypi
gitpython                 3.1.27                   pypi_0    pypi
google-auth               2.6.2                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
greenlet                  1.1.2                    pypi_0    pypi
grpcio                    1.44.0                   pypi_0    pypi
h5py                      3.6.0           nompi_py37h2502845_100    conda-forge
hdf5                      1.12.1          nompi_h2a0e4a3_103    conda-forge
icu                       69.1                 h0e60522_0    conda-forge
identify                  2.4.12                   pypi_0    pypi
idna                      3.3                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imageio                   2.15.0                   pypi_0    pypi
imagesize                 1.3.0                    pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        4.11.3                   pypi_0    pypi
importlib-resources       5.6.0                    pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
intel-openmp              2022.0.0          haa95532_3663
ipykernel                 6.11.0                   pypi_0    pypi
ipython                   7.32.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                7.7.0                    pypi_0    pypi
jedi                      0.17.2                   pypi_0    pypi
jinja2                    3.1.1                    pypi_0    pypi
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9e                   h8ffe710_0    conda-forge
jsmin                     3.0.1                    pypi_0    pypi
json5                     0.9.6                    pypi_0    pypi
jsonpickle                1.2                      pypi_0    pypi
jsonschema                4.4.0                    pypi_0    pypi
jupyter-cache             0.4.3                    pypi_0    pypi
jupyter-client            7.2.1                    pypi_0    pypi
jupyter-core              4.9.2                    pypi_0    pypi
jupyter-server            1.13.5                   pypi_0    pypi
jupyter-server-mathjax    0.2.5                    pypi_0    pypi
jupyter-sphinx            0.3.2                    pypi_0    pypi
jupyterlab                3.3.2                    pypi_0    pypi
jupyterlab-pygments       0.1.2                    pypi_0    pypi
jupyterlab-server         2.12.0                   pypi_0    pypi
jupyterlab-widgets        1.1.0                    pypi_0    pypi
keras                     2.8.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keyring                   23.5.0                   pypi_0    pypi
kiwisolver                1.4.2                    pypi_0    pypi
libblas                   3.8.0                    20_mkl    conda-forge
libcblas                  3.8.0                    20_mkl    conda-forge
libclang                  13.0.0                   pypi_0    pypi
libcurl                   7.80.0               h86230a5_0
libiconv                  1.15                 h1df5818_7
liblapack                 3.8.0                    20_mkl    conda-forge
libpng                    1.6.37               h2a8f88b_0
libssh2                   1.9.0                h7a1dbc1_1
libtiff                   4.2.0                hd0e1b90_0
libwebp                   1.2.2                h2bbff1b_0
libxml2                   2.9.12               h0ad7f3c_0
libxslt                   1.1.34               he774522_0
libzlib                   1.2.11            h8ffe710_1014    conda-forge
linkify-it-py             1.0.3                    pypi_0    pypi
livereload                2.6.3                    pypi_0    pypi
lz4-c                     1.9.3                h2bbff1b_1
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markdown                  3.3.6                    pypi_0    pypi
markdown-it-py            1.1.0                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.1                    pypi_0    pypi
matplotlib-inline         0.1.3                    pypi_0    pypi
mdit-py-plugins           0.2.8                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
mkl                       2020.2                      256
msys2-conda-epoch         20160418                      1
mypy-extensions           0.4.3                    pypi_0    pypi
myst-nb                   0.13.2                   pypi_0    pypi
myst-parser               0.15.2                   pypi_0    pypi
nbclassic                 0.3.7                    pypi_0    pypi
nbclient                  0.5.13                   pypi_0    pypi
nbconvert                 6.4.5                    pypi_0    pypi
nbdime                    3.1.1                    pypi_0    pypi
nbformat                  5.1.3                    pypi_0    pypi
nest-asyncio              1.5.5                    pypi_0    pypi
networkx                  2.6.3                    pypi_0    pypi
nodeenv                   1.6.0                    pypi_0    pypi
notebook                  6.4.10                   pypi_0    pypi
notebook-shim             0.1.0                    pypi_0    pypi
numpy                     1.21.5           py37h5fa1a60_0    conda-forge
oauthlib                  3.2.0                    pypi_0    pypi
olefile                   0.46                     py37_0
opencv-python             4.5.5.64                 pypi_0    pypi
opencv-python-headless    4.5.5.62                 pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3                     pypi_0    pypi
pandas                    1.3.5            py37h9386db6_0    conda-forge
pandocfilters             1.5.0                    pypi_0    pypi
parso                     0.7.1                    pypi_0    pypi
pathspec                  0.9.0                    pypi_0    pypi
patsy                     0.5.2                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    8.4.0            py37hd45dc43_0
pip                       21.2.4           py37haa95532_0
pkginfo                   1.8.2                    pypi_0    pypi
platformdirs              2.5.1                    pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
pre-commit                2.17.0                   pypi_0    pypi
prometheus-client         0.13.1                   pypi_0    pypi
prompt-toolkit            3.0.28                   pypi_0    pypi
protobuf                  3.20.0                   pypi_0    pypi
psutil                    5.9.0                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pydata-sphinx-theme       0.8.1                    pypi_0    pypi
pygithub                  1.55                     pypi_0    pypi
pygments                  2.11.2                   pypi_0    pypi
pyjwt                     2.3.0                    pypi_0    pypi
pykalman                  0.9.5                    pypi_0    pypi
pynacl                    1.5.0                    pypi_0    pypi
pyparsing                 3.0.7                    pypi_0    pypi
pyrsistent                0.18.1                   pypi_0    pypi
pyside2                   5.14.1                   pypi_0    pypi
pytest                    7.1.1                    pypi_0    pypi
pytest-cov                3.0.0                    pypi_0    pypi
pytest-qt                 4.0.2                    pypi_0    pypi
pytest-xvfb               2.0.0                    pypi_0    pypi
python                    3.7.13               h6244533_0
python-dateutil           2.8.2              pyhd3eb1b0_0
python-rapidjson          1.6                      pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2021.3             pyhd3eb1b0_0
pytz-deprecation-shim     0.1.0.post0              pypi_0    pypi
pyvirtualdisplay          3.0                      pypi_0    pypi
pywavelets                1.3.0                    pypi_0    pypi
pywin32                   303                      pypi_0    pypi
pywin32-ctypes            0.2.0                    pypi_0    pypi
pywinpty                  1.1.6                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     22.3.0                   pypi_0    pypi
qimage2ndarray            1.8.3                    pypi_0    pypi
qt                        5.12.9               h556501e_6    conda-forge
readme-renderer           34.0                     pypi_0    pypi
regex                     2022.3.15                pypi_0    pypi
requests                  2.27.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
requests-toolbelt         0.9.1                    pypi_0    pypi
rfc3986                   2.0.0                    pypi_0    pypi
rich                      10.16.1                  pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
scikit-image              0.19.2                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scikit-video              1.1.11                   pypi_0    pypi
scipy                     1.7.3            py37hb6553fb_0    conda-forge
seaborn                   0.11.2                   pypi_0    pypi
segmentation-models       1.0.1                    pypi_0    pypi
send2trash                1.8.0                    pypi_0    pypi
setuptools                61.3.1                   pypi_0    pypi
shapely                   1.7.1            py37h06580b3_0
shiboken2                 5.14.1                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1
sleap                     1.2.1                     dev_0    <develop>
smmap                     5.0.0                    pypi_0    pypi
sniffio                   1.2.0                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
soupsieve                 2.3.1                    pypi_0    pypi
sphinx                    4.5.0                    pypi_0    pypi
sphinx-autobuild          2021.3.14                pypi_0    pypi
sphinx-book-theme         0.3.2                    pypi_0    pypi
sphinx-copybutton         0.5.0                    pypi_0    pypi
sphinx-togglebutton       0.3.1                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
sqlalchemy                1.4.34                   pypi_0    pypi
sqlite                    3.38.2               h2bbff1b_0
statsmodels               0.13.2                   pypi_0    pypi
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.8.0                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.24.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.13.3                   pypi_0    pypi
testpath                  0.6.0                    pypi_0    pypi
tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2021.11.2                pypi_0    pypi
tk                        8.6.11               h2bbff1b_0
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
tornado                   6.1                      pypi_0    pypi
tqdm                      4.63.1                   pypi_0    pypi
traitlets                 5.1.1                    pypi_0    pypi
twine                     3.3.0                    pypi_0    pypi
typed-ast                 1.5.2                    pypi_0    pypi
typing-extensions         4.1.1                    pypi_0    pypi
tzdata                    2022.1                   pypi_0    pypi
tzlocal                   4.2                      pypi_0    pypi
uc-micro-py               1.0.1                    pypi_0    pypi
urllib3                   1.26.9                   pypi_0    pypi
vc                        14.2                 h21ff451_1
virtualenv                20.14.0                  pypi_0    pypi
vs2015_runtime            14.27.29016          h5e58377_2
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.3.2                    pypi_0    pypi
werkzeug                  2.1.1                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0
widgetsnbextension        3.6.0                    pypi_0    pypi
wincertstore              0.2              py37haa95532_2
wrapt                     1.14.0                   pypi_0    pypi
xz                        5.2.5                h62dcd97_0
zipp                      3.7.0                    pypi_0    pypi
zlib                      1.2.11            h8ffe710_1014    conda-forge
zstd                      1.4.9                h19a0ad4_0
Logs
Traceback (most recent call last):
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\app.py", line 874, in wrapped_function
    action(*args)
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\commands.py", line 541, in generateSuggestions
    self.execute(GenerateSuggestions, **params)
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\commands.py", line 236, in execute
    command().execute(context=self, params=kwargs)
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\commands.py", line 133, in execute
    self.do_with_signal(context, params)
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\commands.py", line 157, in do_with_signal
    cls.do_action(context, params)
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\commands.py", line 2024, in do_action
    labels=context.labels, params=params
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\suggestions.py", line 67, in suggest
    return method_functions[method](labels=labels, **params)
  File "c:\users\elizabeth\desktop\sleap_develop\sleap\gui\suggestions.py", line 84, in basic_sample_suggestion_method
    range(0, video.frames, video.frames // per_video)
ValueError: range() arg 3 must not be zero

Screenshots

image
image

How to reproduce

  1. Go to 'Generate Suggestions'
  2. Click on 'Method'
  3. Scroll down to 'Stride'
  4. Click 'Generate Suggestions'
  5. Initializing sample suggestions shows and value error is given in the command prompt
@eberrigan eberrigan added the bug Something isn't working label Jun 2, 2022
@roomrys
Copy link
Collaborator

roomrys commented Jun 2, 2022

Hi @eberrigan,

Do you mind giving us a bit more info on the problem?

Thanks,
Liezl

@eberrigan
Copy link
Contributor Author

Sorry, the contents didn't save for some reason! Updated above.

@roomrys
Copy link
Collaborator

roomrys commented Jun 6, 2022

Problem Analysis

Currently when using the stride method for generating suggestions, SLEAP will loop through each video and generate a list of indices to suggest using the following logic:

vid_suggestions = list(
                    range(0, video.frames, video.frames // per_video)
                )[:per_video]

However, when perform the floor division video.frames // per_video, we get a zero for arg 3 of range() if the video is shorter than the specified stride (stored in per_video).

While we are focused on this section of the code, we can also ensure that the method for generating random samples, which is currently implemented as:

vid_suggestions = random.sample(range(video.frames), per_video)

either uses the per_video value set by the user or the number of frames in the video (if video length < per_video). Otherwise, a value error is currently returned:

ValueError: Sample larger than population or is negative

Proposed Solution(s)

Note: Sol'n 2 is likely best as it maintains the current functionality of suggestions via "stride".

  1. Add a quick check if video.frames > per_video before generating vid_suggestions.
  2. Ensure that arg 3 of range() is > 0 (equal to 1 even if video.frames // per_video == 0)
  3. New option to predict on per_project number of frames spaced evenly throughout entire project.

Test

Tests should go in test/gui/test_suggestions.py and test:

  • "stride" suggestions works on all project videos even if "stride" length is greater than video length
  • At least one sample is taken from each video if "stride" is used - regardless of video length

Relevant files/functions

  • L84 VideoFrameSuggestions.basic_sample_suggestion_method() from sleap/gui/suggestions.py
  • L20 sleap\config\suggestions.yaml
  • L1086 MainWindow._create_dock_windows from sleap\gui\app.py
  • tests\gui\test_suggestions.py

@eberrigan
Copy link
Contributor Author

eberrigan commented Jun 8, 2022 via email

@davidasamy
Copy link
Collaborator

davidasamy commented Jun 10, 2022

Hi @eberrigan,

We have patched your issue in the development version of SLEAP. If you would like to use this version of SLEAP with the new changes please use the conda from source installation.

Thanks,
David

@davidasamy davidasamy added the fixed in future release Fix or feature is merged into develop and will be available in future release. label Jun 10, 2022
@roomrys
Copy link
Collaborator

roomrys commented Jun 29, 2022

Hi @eberrigan ,

The new release of SLEAP v1.2.4 is now available for installation and includes this fix.

Thanks,
Liezl

@roomrys roomrys closed this as completed Jun 29, 2022
@roomrys roomrys removed the fixed in future release Fix or feature is merged into develop and will be available in future release. label Jun 29, 2022
@eberrigan
Copy link
Contributor Author

eberrigan commented Oct 11, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants