为什么训练起来特别慢

[kingdeewang]

在chat目录下运行py train.py，几天才跑了一个epoch是啥原因？

[qhduan]

没用GPU或者cuda配置不对就特别的慢

[shizhediao]

2018-04-11 21:24:01.266522: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-04-11 21:24:01.745824: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_INVALID_DEVICE
2018-04-11 21:24:01.745901: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: LCWM-GPU-20
2018-04-11 21:24:01.745914: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: LCWM-GPU-20
2018-04-11 21:24:01.745961: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 367.48.0
2018-04-11 21:24:01.746002: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 367.48 Sat Sep 3 18:21:08 PDT 2016
GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
"""
2018-04-11 21:24:01.746033: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 367.48.0
2018-04-11 21:24:01.746046: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 367.48.0
epoch 1 loss=6.931528 lr=0.001000: 0%| | 17/104630 [03:37<328:16:09, 11.30s/it]

请问显示这样的信息，请问是正常使用了GPU训练吗。。。
我看有个error是不是错了。。。
感觉还是特别慢

[qhduan]

2018-04-11 21:24:01.745824: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_INVALID_DEVICE

会有这样提示一般都不行啦

[shizhediao]

谢谢！！！
请问如果用GTX1080训练，大概需要多长时间呢？

[qhduan]

参数不一样区别很大

[yaleimeng]

demo里面默认的dgk_shooter_min.conv太大了，可以换一个小的。体积越小，训练越快

[charlesXu86]

我想问一下，你们训练的时候没有多线程报错的问题吗

[NexusLee]

@charlesXu86 有 老哥