Anand Sukumaran

Startup Founder, Software Engineer, Abstract thinker
Co-founder & CTO @ Engagespot (Techstars NYC '24)

Fix missed heartbeats from client - RabbitMQ | Python Pika

Recently I come across with a strange issue in RabbitMQ. The RabbitMQ consumer written in Pika, a python based AMQP client was getting killed. When I checked the logs, it says “Missed heartbeats from client”.

I had enabled heartbeats interval in the connection

pika.BlockingConnection(
  pika.ConnectionParameters(
    host='localhost',
    heartbeat=10
  )
)

How does the heartbeat timer works?

The server sends a ping to the client script at particular intervals. If the client doesn’t respond to the heartbeat message, the broker thinks that the client process has been killed, and thus it terminates the TCP connection.

The problem with pika.BlockingConnection

The problem with BlockingConnection is that, it blocks the main thread when it process the incoming message.

For example, if your RabbitMQ client’s incoming message handler takes few milliseconds to finish a task, and coincidentally if the client also receives a heartbeat message from the broker, it is unable to respond. Because the main thread is being blocked by the incoming message handler (even if it’s for few seconds).

Based on what I understood from the documentation, if two consequent heartbeat timers are missed, the broker thinks that the client processed has died.

So if you’re unlucky, you’ll face this issue atleast once in the lifetime of your client process, and then the broker will close the connection.

Solution

Well, theoritically, the solution is to avoid blocking the main thread so the heartbeat timers are never acknowledged by your client process. But since BlockingConnection is used, it is not possible. So, the solution is to manually process the heartbeat timer inside your message handler.

connection.process_data_events()

Just have this function call inside your blocking message handler, and then it’ll process any heartbeat pings from the broker and therefore your client will never miss any hearbeats!