Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic reconnection when RMQ is not available when sending the first message. #1038

Closed
dmkost opened this issue Jun 28, 2019 · 6 comments · Fixed by #1040 or #1043
Closed

Automatic reconnection when RMQ is not available when sending the first message. #1038

dmkost opened this issue Jun 28, 2019 · 6 comments · Fixed by #1040 or #1043

Comments

@dmkost
Copy link

dmkost commented Jun 28, 2019

Hello. Please help me with the problem.

The problem is related to automatic reconnection.
I use Spring Boot 2.1.3, but it happens in version 2.1.6 and RabbitMQ with spring-boot-starter-amqp

Bin RabbitTemplate looks like this:

public final class ExtendedRabbitTemplate extends RabbitTemplate {

    public ExtendedRabbitTemplate(final ConnectionFactory connectionFactory) {
        super(connectionFactory);
    }

    @Override
    protected final void replyTimedOut(final String correlationId) {
        throw new RpcTimeoutException(String.format("Timeout on receiving reply to message with correlation id = %s", correlationId));
    }

}
    @Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
        RabbitTemplate template = new ExtendedRabbitTemplate(connectionFactory);
        template.setMessageConverter(producerJackson2MessageConverter());
        template.setReplyTimeout(properties.getRpcMessagesReplyTimeout());
        return template;
    }

We use the following method to support rpc:
Снимок экрана 2019-06-28 в 15 32 09
We want to use Direct ReplyTo with pseudo queues.

The scenario in which the normal behavior RabbitTemplate

  1. Initial state: Internet is available (access to the RMQ), the application is not running.
  2. Run the application.
  3. We send several messages and get answers. All is well.
  4. Turning off the Internet (access to the RMQ).
  5. We send several messages and get an exception after a timeout. Everything is also OK.

Scenario in which abnormal behavior is detected:

  1. Initial state: the Internet is not available(access to the RMQ), the application is not running
  2. Run the application.
  3. We send several messages and get exceptions org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused.
  4. Turn on the Internet (access to the RMQ)
  5. We send several messages and receive exceptions after a timeout. Therefore, no reconnection occurs.

The error is as follows:
Снимок экрана 2019-06-28 в 15 15 38

I tried to figure out what was happening and found the following. The selected code block is executed 1 time at the first call. If the access was to the rabbit, then the first condition in the chain is satisfied. If we get an exception, then we get into the second condition block. Everything happens because at the first message(RQM is not available) we get replyAddress = null and usingFastReplyTo = false.
Снимок экрана 2019-06-28 в 15 10 40

I hope this is enough.

@garyrussell
Copy link
Contributor

You are correct - thanks for the analysis.

garyrussell added a commit to garyrussell/spring-amqp that referenced this issue Jun 28, 2019
Fixes spring-projects#1038

Don't set `evaluatedFastReplyTo` if we didn't actually evaluate it because
the broker is down on the first request.

**cherry-pick to all 2.x; backport to 1.7.x**
artembilan pushed a commit that referenced this issue Jun 28, 2019
Fixes #1038

Don't set `evaluatedFastReplyTo` if we didn't actually evaluate it because
the broker is down on the first request.

**cherry-pick to all 2.x; backport to 1.7.x**
artembilan pushed a commit that referenced this issue Jun 28, 2019
Fixes #1038

Don't set `evaluatedFastReplyTo` if we didn't actually evaluate it because
the broker is down on the first request.

**cherry-pick to all 2.x; backport to 1.7.x**

# Conflicts:
#	spring-rabbit/src/test/java/org/springframework/amqp/rabbit/core/RabbitTemplateTests.java
artembilan pushed a commit that referenced this issue Jun 28, 2019
Fixes #1038

Don't set `evaluatedFastReplyTo` if we didn't actually evaluate it because
the broker is down on the first request.

**cherry-pick to all 2.x; backport to 1.7.x**

# Conflicts:
#	spring-rabbit/src/test/java/org/springframework/amqp/rabbit/core/RabbitTemplateTests.java

# Conflicts:
#	spring-rabbit/src/test/java/org/springframework/amqp/rabbit/core/RabbitTemplateTests.java
artembilan pushed a commit that referenced this issue Jun 28, 2019
Fixes #1038

Don't set `evaluatedFastReplyTo` if we didn't actually evaluate it because
the broker is down on the first request.

**cherry-pick to all 2.x; backport to 1.7.x**

# Conflicts:
#	spring-rabbit/src/test/java/org/springframework/amqp/rabbit/core/RabbitTemplateTests.java

# Conflicts:
#	spring-rabbit/src/test/java/org/springframework/amqp/rabbit/core/RabbitTemplateTests.java

# Conflicts:
#	spring-rabbit/src/main/java/org/springframework/amqp/rabbit/core/RabbitTemplate.java
#	spring-rabbit/src/test/java/org/springframework/amqp/rabbit/core/RabbitTemplateTests.java
@OctoSenya
Copy link

Hi!
It seem's like the cause of problem is SocketException wrapped in AmqpIOException, not AmqpConnectException.

@garyrussell
Copy link
Contributor

What operating system/JDK? For me it gets a

Caused by: java.net.ConnectException: Connection refused (Connection refused)

@garyrussell garyrussell reopened this Jul 1, 2019
@garyrussell
Copy link
Contributor

The problem with using AmqpIOException is that a queue declaration failure gives that exception; we will have to dig down to the root cause.

@OctoSenya
Copy link

Ubuntu 18.04
openjdk version "1.8.0_212"
I'm getting this exception when trying to send message with internet disabled.
image

@garyrussell
Copy link
Contributor

OK

garyrussell added a commit to garyrussell/spring-amqp that referenced this issue Jul 1, 2019
Fixes spring-projects#1038

The previous fix to catch `AmqpConnectException` was incorrect - the
`ShutdownSignalException` for the passive queue declaration failure
is wrapped in an `AmqpConnectException` so we would have failed to
detect the failure.

Also, it has been reported that sometimes an `AmqpIOException` is thrown.

- Add `AmqpIOException` to the catch block
- Search for, and explitly check for the queue declaration failed exception
- For all other cases rethrow so we can test again
garyrussell added a commit to garyrussell/spring-amqp that referenced this issue Jul 1, 2019
Fixes spring-projects#1038

The previous fix to catch `AmqpConnectException` was incorrect - the
`ShutdownSignalException` for the passive queue declaration failure
is wrapped in an `AmqpConnectException` so we would have failed to
detect the failure.

Also, it has been reported that sometimes an `AmqpIOException` is thrown.

- Add `AmqpIOException` to the catch block
- Search for, and explitly check for the queue declaration failed exception
- For all other cases rethrow so we can test again

**cherry-pick to all supported**
garyrussell added a commit to garyrussell/spring-amqp that referenced this issue Jul 1, 2019
Fixes spring-projects#1038

The previous fix to catch `AmqpConnectException` was incorrect - the
`ShutdownSignalException` for the passive queue declaration failure
is wrapped in an `AmqpConnectException` so we would have failed to
detect the failure.

Also, it has been reported that sometimes an `AmqpIOException` is thrown.

- Add `AmqpIOException` to the catch block
- Search for, and explitly check for the queue declaration failed exception
- For all other cases rethrow so we can test again

**cherry-pick to all supported**
artembilan pushed a commit that referenced this issue Jul 1, 2019
Fixes #1038

The previous fix to catch `AmqpConnectException` was incorrect - the
`ShutdownSignalException` for the passive queue declaration failure
is wrapped in an `AmqpConnectException` so we would have failed to
detect the failure.

Also, it has been reported that sometimes an `AmqpIOException` is thrown.

- Add `AmqpIOException` to the catch block
- Search for, and explitly check for the queue declaration failed exception
- For all other cases rethrow so we can test again

**cherry-pick to all supported**

* * Fix log messages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment