Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot check actual batch size at wandb when using autobatch #6008

Closed
1 task done
Seyoung9304 opened this issue Dec 16, 2021 · 6 comments · Fixed by #6039
Closed
1 task done

Cannot check actual batch size at wandb when using autobatch #6008

Seyoung9304 opened this issue Dec 16, 2021 · 6 comments · Fixed by #6039
Labels
question Further information is requested TODO High priority items

Comments

@Seyoung9304
Copy link

Search before asking

Question

Hi! :)

I found out that when we use autobatch with --batch-size -1 option, -1 is logged at wandb table.

Is there a way to log the actual batch size used for learning in wandb?

image

Additional

No response

@Seyoung9304 Seyoung9304 added the question Further information is requested label Dec 16, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Dec 16, 2021

👋 Hello @Seyoung9304, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at [email protected].

Requirements

Python>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

$ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Status

CI CPU testing

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

@glenn-jocher
Copy link
Member

@AyushExel do you know if there's a way to update the batch size field in wandb after creation? Or could we delay the wandb logger creation until after AutoBatch is computed here? We can't move this up because it depends on imgsz, which in turn requires model to be defined to get the model stride.

yolov5/train.py

Lines 134 to 141 in c1249a4

# Image size
gs = max(int(model.stride.max()), 32) # grid size (max stride)
imgsz = check_img_size(opt.imgsz, gs, floor=gs * 2) # verify imgsz is gs-multiple
# Batch size
if RANK == -1 and batch_size == -1: # single-GPU only, estimate best batch size
batch_size = check_train_batch_size(model, imgsz)

@AyushExel
Copy link
Contributor

@glenn-jocher yes we can update the batch size in in wandb after the run is created as well. It should be a simple one line fix but I'm away from my system this week :( . I can add this feature on monday

@glenn-jocher
Copy link
Member

@AyushExel awesome, thanks for the fast response!

@Seyoung9304 we should be able to push a fix for this next week. Thank you for your patience!

@glenn-jocher glenn-jocher added the TODO High priority items label Dec 16, 2021
@Seyoung9304
Copy link
Author

@AyushExel @glenn-jocher Thank you so much for fast response! 👍 Have a nice day 😀

@glenn-jocher
Copy link
Member

glenn-jocher commented Dec 23, 2021

@Seyoung9304 good news 😃! Your original issue may now be fixed ✅ in PR #6039 by @AyushExel. To receive this update:

  • Gitgit pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
  • PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
  • Notebooks – View updated notebooks Open In Colab Open In Kaggle
  • Dockersudo docker pull ultralytics/yolov5:latest to update your image Docker Pulls

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested TODO High priority items
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants