Cloud Training Never Start/End

[GuppyMIS]

Search before asking

• I have searched the HUB issues and found no similar bug report.

HUB Component

Training

Bug

Hi,

Yesterday I trained my dataset using Cloud Training using Nvidia L40S, but it seems until now the training not yet finished/started.

My account already being deducted but my model never shown up.

Please advise

Environment

-Ultralytics Pro Account
-Ultralytics Cloud Training
-Instances: Nvidia L40s

Minimal Reproducible Example

No response

Additional

No response

[UltralyticsAssistant]

👋 Hello @GuppyMIS, thank you for bringing your issue to our attention regarding the Ultralytics HUB 🚀! We're sorry to hear that you're encountering difficulties with the Cloud Training.

To get you started on troubleshooting the issue, please have a look at our HUB Docs which provide comprehensive guidance:

• Models: Training and Exporting. Explore how to train YOLOv5 and YOLOv8 models, tackle common issues, and export them for deployment.
• Inference API. Gain insight into how to utilize the Inference API for your trained models.

Given that this appears to be a 🐛 Bug Report, we kindly ask you to provide a minimum reproducible example along with any screenshots or logs that illustrate the issue. You can learn more about creating an MRE in our guide.

An Ultralytics engineer will review your issue and assist you shortly. We strive to address all queries as quickly as possible. Thank you for your patience and collaboration in resolving this! 🙌

[pderrenger]

@GuppyMIS hi there! Sorry to hear you're experiencing issues with Cloud Training 🤔

This seems like an unusual situation with your training session. Let me help you troubleshoot this:

1. First, please verify if you can see the model listed under your Models page in HUB, even if it shows as pending or incomplete.

2. According to our Cloud Training documentation, the initialization process can sometimes take a while depending on GPU resource availability, but should not be indefinite. However, your situation appears to be an edge case since your account was already charged.

3. Since this involves billing and potential technical issues with the training infrastructure, I recommend:

   a. Taking a screenshot of your billing section showing the deduction
   b. Opening a support ticket through GitHub Issues with:

   • The exact time you started the training
   • Your model ID or name
   • The screenshots of both the training status and billing
   • The dataset size and number of epochs configured

4. In the meantime, you can try starting a new training session with a smaller number of epochs first to verify if the issue persists.

The Ultralytics team monitors these issues closely and will help resolve any billing discrepancies or technical issues you've encountered.

For future training sessions, you might want to check out our Cloud Training guide which includes best practices and troubleshooting tips for smooth training experiences.

Let me know if you need any clarification or have additional questions! 🙂

[yogendrasinghx]

@GuppyMIS We sincerely apologize for the inconvenience caused and thank you for bringing this issue to our attention.

I have forwarded this issue to our team to investigate and resolve it promptly and requested the team to initiate a refund for the deducted amount.

To help us investigate and resolve the problem efficiently, could you please share the Model ID?

You can find it in the URL when accessing the model in the HUB Web, for example: https://hub.ultralytics.com/models/7wzkDSKNMcwkPTs8ZVJC.

Model ID: 7wzkDSKNMcwkPTs8ZVJC

Providing the Model ID will allow us to locate your account and gain a better understanding of the issue.

We truly appreciate your patience and understanding in this matter. I’ll keep you updated as soon as we have more information.

Thank you again for your cooperation.