'RabbitMQ how to change ownership of an ha queue that did not migrate when node left cluster
I have a three-node RabbitMQ cluster (version 3.4.4 on Ubuntu 14.04.3) with approximately 80 queues, 40 of which are ha queues (policy {"ha-mode":"exactly","ha-params":3}). The three-node cluster was increased to 5 nodes. When it was at 5 nodes the queues rebalanced as expected, so that some queues moved ownership to the new cluster nodes.
A few hours later the new nodes were removed from the cluster via the following command sequence on the two new nodes:
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
The cluster_status on all three remaining nodes is consistent and shows only the three remaining nodes as expected.
Cluster status of node 'rabbit@prod-api-02' ...
[{nodes,[{disc,['rabbit@prod-api-01',
'rabbit@prod-api-02',
'rabbit@prod-api-03']}]},
{running_nodes,['rabbit@prod-api-03',
'rabbit@prod-api-01',
'rabbit@prod-api-02']},
{cluster_name,<<"[email protected]">>},
{partitions,[]}]
However, the cluster was left in a state where the three nodes of the cluster believe that two of the ha queues that moved ownership when the cluster was expanded are still owned by a node that is no longer part of the cluster, and it considers them to be in a 'down' state as a consequence. Further, it no longer shows the policy associated with the queues or the slave nodes.
rabbitmqctl list_queues name state policy pid slave_pids
production-process-queue down <[email protected]>
delete_unused_things_queue down <[email protected]>
Trying to delete the two queues returns an error saying the home node is down or inaccessible.
Has anyone experienced this type of problem before? Is there any way to change the ownership of these two queues back to one of the remaining cluster nodes, or a way to delete them? I do not need to preserve the queue content.
Solution 1:[1]
Had a similar problem.
It is possible to move the home node if you use the command forget_cluster_node for each of the two departed nodes. See https://www.rabbitmq.com/rabbitmqctl.8.html#forget_cluster_node.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Radu Marian |