OpenSSL::SSL::SSLErrorWaitReadable: read would block

[lwoggardner]

I am seeing these errors making calls to different DynamoDB client objects from multiple threads - perhaps they are still sharing a connection because they are using the same endpoint.

This is on 2.3.7 and using Ruby 2.1.5. I will try and create a repeatable case on other ruby versions.

Could a simple fix be to include this as a retriable error?

OpenSSL::SSL::SSLErrorWaitReadable: read would block
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/patches.rb:29:in block in new_transport_request' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/patches.rb:26:incatch'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/patches.rb:26:in new_transport_request' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/connection_pool.rb:337:inrequest'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/handler.rb:62:in block in transmit' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/handler.rb:111:inblock in session'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/connection_pool.rb:108:in session_for' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/handler.rb:109:insession'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/handler.rb:61:in transmit' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/net_http/handler.rb:35:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/plugins/content_length.rb:12:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/dynamodb_crc32_validation.rb:27:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/json/error_handler.rb:8:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/request_signer.rb:84:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/retry_errors.rb:87:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/json/handler.rb:11:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/user_agent.rb:12:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/plugins/endpoint.rb:41:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/param_validator.rb:21:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/logging.rb:39:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/plugins/raise_response_errors.rb:14:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/dynamodb_simple_attributes.rb:112:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/aws-sdk-core/plugins/param_converter.rb:20:in call' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/plugins/response_target.rb:21:incall'
/opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/request.rb:70:in send_request' /opt/bambooagent/.gem/ruby/2.1/gems/aws-sdk-core-2.3.7/lib/seahorse/client/base.rb:207:inblock (2 levels) in define_operation_methods'

[trevorrowe]

I have not seen this error reported previously. Client objects should be thread safe. Clients share a pool of Net::HTTP connections. Connections are removed from the pool while in use and returned once the request is completed.

It would be helpful if you could provide a minimal test case that demonstrates this issue under concurrency so we can track down the root cause.

[awood45]

While waiting on the repro case, research has also indicated (I think, need to confirm) that this exception hasn't always been defined in OpenSSL. To avoid breaking some customers, we'd want to ensure that we don't crash out when the constant isn't defined. (This is primarily a note to self.)

[lwoggardner]

The other point to note is that we are making these calls through a Squid Proxy (via CONNECT, there is no SSL termination on the proxy).

I have horribly hacked around it with this...
Aws::Plugins::RetryErrors::ErrorInspector::NETWORKING_ERRORS << 'SSLErrorWaitReadable'

which has unblocked me (I think). May take a little while to generate the reproducible case, but I'll try and do that in the next week or two.

[trevorrowe]

Have you stopped seeing these errors after adding this to the list of retryable errors?

[lwoggardner]

I think I've worked this out.

Adding the error to the retryable list did not stop the errors appearing in my logs.

However, trying to explicitly rescue it didn't work either.

So I'm pretty sure that it was being raised and handled by net/http or aws-sdk already, but when I raise my final exception the OpenSSL error is still in $! and is thus associated as the "cause" of my exception, and then getting logged when my program finished - leading me down this merry garden path.

I haven't proved this yet either, but I'm pretty sure it was a non issue to begin with.