Anand Sukumaran Nair
Startup Founder, Software Engineer, Abstract thinker
Co-founder, Engineering @ Engagespot
BRPOPLPUSH ReplyError: UNBLOCKED force unblock from blocking operation, instance state changed - Solving Node bull queue issue with RedisJan 21, 2023
I had a Node.js app hosted in a single EC2 instance using a docker-compose file. Strange thing was that the server’s disk space always reach 100% and the Redis stops working.
After investigating, I found the culprit to be the docker container’s log file. It grows to consume 100% of the server disk space no matter how many times I delete and restart the container.
And the APIs were showing some error that says EVALSHA
After checking the log file, I found this particular error being consistently logged.
BRPOPLPUSH ReplyError: UNBLOCKED force unblock from blocking operation, instance state changed Couldn’t find any resource online to solve this. Later I figured out that, this happens when the Redis configuration changes from master to slave after starting the container.
I was surprised at how this happened. It kept happening.
Out of curiosity, I checked the security group of this EC2 instance in the AWS Console. And there you go! The redis port 6379 was open to the public!
I blocked the port and the issue has been solved forever, finally!
TL-DR: Never ever make your Redis instance available to the public!