[BUG] Unable to send backup to custom s3 storage url

[kapitani4]

Error Description.
Unable to send backup to company's internal s3 storage
when running the backup command, the utility does not parse the address of the required s3 storage from the configuration

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: s3.ru.amazonaws.com: Name or service not known
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: s3.ru.amazonaws.com
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: s3.ru.amazonaws.com
Caused by: java.net.UnknownHostException: s3.ru.amazonaws.com

at the same time running aws s3 ls gives correct results, i.e. the settings in the system are correct

Replay
The utility is installed from the rpm package
run with the command
esop backup --jmx-service={{ip}}:7199 --jmx-user={{user}} --jmx-password={{password}} --storage-location s3://{{my_bucket}} --data-dir /opt/ssd1 --data-dir /opt/ssd2/ --data-dir /opt/ssd3 --data-dir /opt/ssd4 --data-dir /opt/ssd5 --data-dir /opt/ssd6

The URL is specified in $AWS_ENDPOINT and in $AWS_ENDPOINT_URL, as well as in .aws/config
[default]
region = ru
output = json
endpoint_url = https://{{specific_host}}
services = s3-specific

[profile s3-prod]
endpoint_url = https://{{specific_host}}
services = s3-specific

[services s3-specific]
s3 =
endpoint_url = https://{{specific_host}}

Expected Behaviour
The utility connects to the correct specified s3 storage and sends the backup there

System and versions (please fill in the following information):.

• RED OS MUROM (7.3.4), Cassandra version 4.0.14, Java version 1.8.0_422, Esop version 3.0.0.

Additional Context
Add any other information about the problem here.

[smiklosovic]

@kapitani4 I think this was removed by

5b6c622

The reason was that it was increasingly difficult to maintain all possible s3 providers (and test that it works!) etc (ceph, minio, oracle ...) If you use something internal, that tells me you are doing something custom / have some specific s3 implementation where this is necessary.

Looking into the current code, we are reading endpoint here

https://github.com/instaclustr/esop/blob/master/src/main/java/com/instaclustr/esop/s3/S3ConfigurationResolver.java#L47

but we are not doing anything with it further here

https://github.com/instaclustr/esop/blob/master/src/main/java/com/instaclustr/esop/s3/v2/S3ClientsFactory.java#L117

so we would need to somehow make it happen there.

Could you checkout 5b96a28 (one commit before the deletion happened) and try if that works for you? We were supporting that back then like visible here

esop/src/main/java/com/instaclustr/esop/s3/v1/TransferManagerFactory.java

Line 63 in 5b96a28

builder.withEndpointConfiguration(new EndpointConfiguration(s3Conf.awsEndpoint, s3Conf.awsRegion.toLowerCase()));

You need to set env properties for that too. I dont think parsing it from that file is supported.