Connection Recovery not triggered

[JanEggers]

Describe the bug

we see this at a customer after the broker is unavailable for a short time. after the broker is back the mqtt connections recover automatically but all amqp clients are still disconnected and receive this error when trying to send events. We have to restart these clients manually then connections are fine again.

we are using the latest 7.12 client with AutomaticRecoveryEnabled = true.
we also hook into ConnectionRecoveryErrorAsync and RecoverySucceededAsync with logging and I dont see any reconnect attempts there.

Reproduction steps

nothing yet, Im working on a repro. but locally everything is working as intended.

Expected behavior

the amqp client reconnects as soon as the broker is up again.

Additional context

No response

[JanEggers]

as a workaround we will dispose and recreate the connection when we get a AlreadyClosedException when sending messages

[lukebakken]

Hi @JanEggers, any chance you can test with a build of this client from main? I think I'll be able to get version 7.2 out by the end of the month, and this particular issue may already be fixed.

[JanEggers]

sry cannot reproduce still and im not going into prod with preview versions

[lukebakken]

OK, how many times have you seen this issue? I'm just trying to get a feel for how serious it may be.

[JanEggers]

we had to restart our services maybe once or twice a week. but since we do the manual reconnect now cnnection seem to be stable.

[lukebakken]

I'm re-opening this because I intend to take a look.

[giuliboy]

adding to the conversation, maybe fyi @lukebakken
the customer is performing nightly VM snapshots. we have seen this "fatal" "OS monotonic time stepped backwards!" log entry , after which RabbitMQ was restarted

This is performed every night, but we only had this issue 3-4 times now - NOT every day.

[michaelklishin]

@giuliboy this log snippet is not particularly relevant for this question. The OS monotonic time tracking is not a part of RabbitMQ. See rabbitmq/rabbitmq-server#8940.

In the context of connection recovery, this scenario is no different from any other node restart. Please provide evidence relevant to this client's recovery, in particular on 7.x versions. The author of the issue himself states that you cannot reproduce this behavior locally. There are quite a few integration tests specifically for connection recovery, so you can be sure it's not something obvious.