Fuel for OpenStack

RabbitMQ cannot stop on controllers during re-assembling a cluster

Bug #1447184 reported by Dennis Dmitriev on 2015-04-22

This bug report is a duplicate of: Bug #1446241: RabbitMQ OCF timeout should be used without 'su' childs. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	New	High	Fuel Library (Deprecated)	Fuel for OpenStack 6.1

Bug Description

Steps to reproduce:
    1) Deploy HA cluster (Ubuntu, 3 controllers, 1 compute)
    2) Shutdown primary controller (slave-02_controller in the example http://paste.openstack.org/show/205088/)
    3) Run OSTF check

Expected result: OSTF Functional tests are passed, several tests that check count of services are failed.

Actual result: OSTF Functional tests that perform actions with an instance are failed, RabbitMQ replication test failed:
"Failed to connect to 5673 port on host 10.109.20.5 Please refer to OpenStack logs for more details."

On the node-2 and node-3 there are several processes that are trying to shutdown 'rabbitmq_server' every 6 minutes:
http://paste.openstack.org/show/205091/

Resources 'p_rabbitmq-server' become unmanaged on node-2 and node-3:

--------------------------------
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     p_rabbitmq-server (ocf::fuel:rabbitmq-server): Started node-2.test.domain.local (unmanaged) FAILED
     p_rabbitmq-server (ocf::fuel:rabbitmq-server): Started node-3.test.domain.local (unmanaged) FAILED
...
Failed actions:
    p_rabbitmq-server_stop_0 (node=node-2.test.domain.local, call=542, rc=1, status=Timed Out, last-rc-change=Wed Apr 22 13:33:13 2015
, queued=60003ms, exec=0ms
): unknown error
    p_rabbitmq-server_stop_0 (node=node-3.test.domain.local, call=515, rc=1, status=Timed Out, last-rc-change=Wed Apr 22 13:37:14 2015
, queued=60003ms, exec=0ms
): unknown error
-------------------------------

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-04-22:

fuel-rabbitmq-snapshot-2015-04-22_13-14-48.tar.xz Edit (157.9 MiB, application/octet-stream)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-04-22:

Update:
After manual killing 'beam' process with rabbitmq_server on one node (node-2), the same process on node-3 also was finished , and then pacemaker assemble rabbitmq cluster from node-2 and node-3 in several minutes.

Nastya Urlapova (aurlapova) on 2015-04-22

Changed in fuel:
importance:	Undecided → High

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-04-22:

[root@nailgun ~]# fuel --fuel-version
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
api: '1.0'
astute_sha: bf1751a4fe0d912325e3b4af629126a59c1b2b51
auth_required: true
build_id: 2015-04-20_16-08-39
build_number: '325'
feature_groups:
- mirantis
fuellib_sha: 147f7730d764dc778c208bc1707032e1e67a8061
fuelmain_sha: 991fa8feca4afcf05ce85a1e7ebd7c7503b3f212
nailgun_sha: 69fca3bada96a48ab19b969621e5128390b7052e
openstack_version: 2014.2-6.1
ostf_sha: df8db1f48f03b18126ce5ec65317a1eb83a5a95f
production: docker
python-fuelclient_sha: b4f1ddffd5263489090b65e662173e9e11aafd94
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: bf1751a4fe0d912325e3b4af629126a59c1b2b51
      build_id: 2015-04-20_16-08-39
      build_number: '325'
      feature_groups:
      - mirantis
      fuellib_sha: 147f7730d764dc778c208bc1707032e1e67a8061
      fuelmain_sha: 991fa8feca4afcf05ce85a1e7ebd7c7503b3f212
      nailgun_sha: 69fca3bada96a48ab19b969621e5128390b7052e
      openstack_version: 2014.2-6.1
      ostf_sha: df8db1f48f03b18126ce5ec65317a1eb83a5a95f
      production: docker
      python-fuelclient_sha: b4f1ddffd5263489090b65e662173e9e11aafd94
      release: '6.1'