Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inference on K40m GPU #58

Closed
goyalsaransh97 opened this issue May 22, 2019 · 5 comments
Closed

Inference on K40m GPU #58

goyalsaransh97 opened this issue May 22, 2019 · 5 comments

Comments

@goyalsaransh97
Copy link

I tried running inference for pre-trained pythia mdoel on K40m. It didn't start for quite some time and then the ETA was oscillating around 10-15 hours.
So I enabled multi-GPU training using dataparallel flag. But now its not starting, I've waited for around 30 minutes before stopping it. I got the following errors on stopping:
"
File "/mnt/data_g/saransh/anaconda3/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
".
I've tried re-running it but the problem persists. Earlier I tried it on a V100, it was working fine on that.
Could you suggest something?
@apsdehal
Copy link
Contributor

apsdehal commented May 22, 2019 •
edited
Loading

Hard to say without testing on k40m. But for now can you try setting training_parameters.num_workers to 0?

@goyalsaransh97
Copy link
Author

The problem persists after adding the numworkers flag.

@apsdehal
Copy link
Contributor

Following https://bugs.launchpad.net/designate/+bug/1782647, it seems like a bug with 3.7, can you try with 3.6?

@apsdehal
Copy link
Contributor

Related: pytorch/pytorch#8388

@goyalsaransh97
Copy link
Author

Thanks for your help.

@ChenyuGAO-CS ChenyuGAO-CS mentioned this issue Jul 3, 2019
Closed
apsdehal added a commit that referenced this issue May 8, 2020
@apsdehal
[feature,fix] Exclude list, png, jpeg, cls_prob for feature extraction (
15ab3b2 
#58)

- Fixes support for exclude list and allows it to be passed as args
- Add support for feature extraction from png and jpeg files
- Add "cls_prob", class probabilities field back to output_dict
apsdehal added a commit that referenced this issue May 8, 2020
@apsdehal
[feature,fix] Exclude list, png, jpeg, cls_prob for feature extraction (
0728eee 
#58)

- Fixes support for exclude list and allows it to be passed as args
- Add support for feature extraction from png and jpeg files
- Add "cls_prob", class probabilities field back to output_dict
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@apsdehal @goyalsaransh97