rabbitmq showing wrong disk free limit in management console

Question

as the title says, I have a problem, that the rabbitmq shows (and thinks) that there is more space available, as I told him.

I'm running 2 instances of rabbitmq 3.8.8 with erlang 23.0 in 2 RHEL pods. To these pods, a dynamically provisioned PersistentVolume is bound of 2GB size on NFS. That means, that every pod shoud have 1GB of space for himself.

In the rabbitmq.conf I have the following:

vm_memory_high_watermark.relative = 0.9
total_memory_available_override_value = 1000MB
disk_free_limit.absolute = 1GB
management.load_definitions = /etc/rabbitmq/definitions.json

Also when I start Rabbitmq, I see in the log, that the configuration is read correctly:

2020-10-13 08:26:51.726 [info] <0.427.0> Memory high watermark set to 858 MiB (900000000 bytes) 
2020-10-13 08:26:51.811 [info] <0.439.0> Enabling free disk space monitoring
2020-10-13 08:26:51.811 [info] <0.439.0> Disk free limit set to 1000MB

the problem is, that rabbitMQ somehow thinks, that there is the whole NFS free space available - 54GB (as on the screenshot above). So I got into a problem, that over 200K messages were stuck in one of the queues, filled up those 2GB of PersistentVolume I gave him, but didn't stopped accepting messages, cause he thought, that there is more space available. Of course, the whole rabbitmq pod crashed, cause it couldn't write more messages to the NFS.

can you please guide me, how to set is correctly? Or do you know, why rabbitMQ doesnt respect the disk_free_limit.absolute value?

many thanks

Answer 1

rabbitmq-diagnostics environment | grep disk_free_limit

will display the actual effective configuration value.

On Linux, RabbitMQ will use either a configured absolute value, or compute how much disk space its data directory's partition has by running

df -kP /path/to/directory

which is not aware of Kubernetes quotas.

I don't have an NFS partition on Kubernetes to try but a basic test with the following rabbitmq.conf file

disk_free_limit.absolute = 3GB

does not reproduce; the configured value is used as expected. See 1.

Answer 2

Regarding your question : "why rabbitMQ doesnt respect the disk_free_limit.absolute value" - I think it does (even if it treats the free memory wrong of the k8s / pod).

the value is shown in the image you attached as '954 MiB low watermark' - that means that when you'll have only 1 GB of disk free space the broker will stop publishers from publishing and will only allow consumers to consume until there's more space available on disk.

so as long as the machine has more than 1 gb available it continues to accept messages.

perhaps since it wrongly reads that it has 54 GB of free space it crashes but the disk_free_limit.absolute value seems to be read right.