On RemoveObject, nil error is returned when 5xx is encountered

[ericychoi]

I've noticed that when the S3 server (I am using Rados Gateway for Ceph) returns 5xx, removeObject gives nil error. Anyone else seen this issue?

[harshavardhana]

I've noticed that when the S3 server (I am using Rados Gateway for Ceph) returns 5xx, removeObject gives nil error. Anyone else seen this issue?

RemoveObject() never returns an error in accordance with - http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html

Ceph returning 500 for this API is not correct.

[harshavardhana]

minio-go can on the other hand send an error if received from the server regardless of the API behavior.

[ericychoi]

Isn't 503 a valid error even with S3?
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

This says 503 is returned when service request rate needs to be lower.

[harshavardhana]

This says 503 is returned when service request rate needs to be lower.

It is a valid error in S3 but DELETE API never returns any error other than of course signature related issues etc.

I don't know conceptually here why RadosGW is returning 5xx can you put a ".TraceOn()" and see the HTTP traffic for the XML that is sent? are you saying that 503 error is similar to "service request rate" on RadosGW here?

[harshavardhana]

Closing.. please re-open if you have .TraceOn data we can further investigate.

[ericychoi]

So let me step back and explain what I did to test this. I think I have a solution for this but obviously I am not in the best position to judge it

As I said, we are using minio-go client to talk to RadosGW, and we were testing a case where RadosGW may return a 5xx error by using hijacking the DELETE call at a proxy level. We did this by a custom tool, hijacking the response on the fly.

We found that on 500 from DELETE, the client would retry (which is correct) for MAX RETRY times, but it eventually gives up, returning back to the client. But the client doesn't get an error in this case at all, it just gets nil, nil.

Given that 5xx is already treated as a retriable error, wouldn't you rather return back an error to the client so that it can take charge of retrying at a later time?

I created this PR to temporarily address this on our side: #554, but obviously this isn't complete as it's missing tests; let me know what you think of it please. If you think this approach is sensible, I will complete tests. Otherwise we can rethink my approach here.

[ericychoi]

PR #555 created as an alternative approach