Improve Retry Stratergy while interacting with ML Commons Predict API

[navneet1v]

Description

The current while interacting with ML Commons is very simple, we retry 3 times and then fail the request. This is very naive and doesn't scale well with remote models that can have quotas or rate limiting enabled.

https://github.com/opensearch-project/neural-search/blob/main/src/main/java/org/opensearch/neuralsearch/ml/MLCommonsClientAccessor.java#L116-L134

Solution

Come up with a better retry strategy while interacting with ML Commons Predict API to improve the user experience. Some of the strategies can be:

1. Exponential backoff
2. Token Bucket Retries
   etc.