Upgrading from stable to devel charm breaks clients

Bug #1480893 reported by Liam Young
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rabbitmq-server (Juju Charms Collection)
Fix Released
Critical
James Page

Bug Description

After upgrading the rabbitmq-server charm from the stable version to the next (devel) version clients could no longer connect.
The rabbit logs showed:
...
=ERROR REPORT==== 3-Aug-2015::12:05:51 ===
closing AMQP connection <0.13457.0> (10.5.39.129:40991 -> 10.5.39.128:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

=ERROR REPORT==== 3-Aug-2015::12:05:52 ===
closing AMQP connection <0.13461.0> (10.5.39.121:39746 -> 10.5.39.128:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

=ERROR REPORT==== 3-Aug-2015::12:05:52 ===
closing AMQP connection <0.13464.0> (10.5.39.127:43251 -> 10.5.39.128:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}
...

the password being used by the client was consistent with the password being sent by rabbit:

juju run --unit neutron-gateway/0 "relation-get -r amqp:45 - rabbitmq-server/0"
hostname: 10.5.39.128
password: YxMpxJtTc33g637prBdptgSpk7hthSjSjmYqm6pW32SHPm9yqWscfWwZW9XqwgWL
private-address: 10.5.39.128

 juju run --unit neutron-gateway/0 "grep rabbit /etc/neutron/neutron.conf"
rabbit_userid = neutron
rabbit_virtual_host = openstack
rabbit_password = YxMpxJtTc33g637prBdptgSpk7hthSjSjmYqm6pW32SHPm9yqWscfWwZW9XqwgWL
rabbit_host = 10.5.39.128
list_notifier_drivers = neutron.openstack.common.notifier.rabbit_notifier

changing the password via rabbitmqctl to the one being used by the client fixed the issue

Related branches

Liam Young (gnuoy)
Changed in rabbitmq-server (Juju Charms Collection):
importance: Undecided → Critical
milestone: none → 15.07
status: New → Confirmed
Revision history for this message
James Page (james-page) wrote :

Cluster and leader storage is inconsistent:

ubuntu@james-page-bastion:~/charms⟫ juju run --service rabbitmq-server "relation-get -r cluster:12 - rabbitmq-server/0"
ceilometer.passwd: SWRwZPpSr3sbZtBP4nk72n4jr838NkW6N5R88R3mBHzK7LYG42849f3cPGkz6Ztg
cinder.passwd: YhXCqgYz9MtMgKqnVX38FPCPfbHGZcC69KNmkPn2z5Vg37n3f5dfYZdfnXTVb4yV
heat.passwd: dZFf2zWHMpWHrK2JywJWqfpd5wVt6XwpV8LY88mqjJLysqbR8ML9mGq3Wnxw3X3r
nagios-rabbitmq-server-0.passwd: fgMPL6bqRYJjYFLcKtmYm433zW9qfbV93SWYngytF8sHRXY9FTjJg2HBybFKZFpz
neutron.passwd: 6fp58xNC54B5y46Mfk7JTpc3CKm3dgTgMhndCjNZnmHnNxfHHkkhfSmxFpfjnTBd
nova.passwd: 3fb57LtWMxYHkk6V8bCMF39LBTh7cbkjHy4nKmtJ9rWNyV5xdLx8F2ZxHRWHhMy8
private-address: 10.5.26.108

ubuntu@james-page-bastion:~/charms⟫ juju run --service rabbitmq-server "leader-get"
__leader_get_migrated_settings__: '["nova.passwd", "neutron.passwd"]'
ceilometer.passwd: hscCZxjpV42W4B8NSJRC3ZBk6VYgzCnNd8NNr62XZCcSVNHbJwk9G6tVrhgj4qJz
cinder.passwd: kKrpY27LjmV5PXLrkhd9Jy2TcRM8jCcwG2GPnz97SC5rd2KWC5ZK6CTn8ckLKZYY
heat.passwd: B3r2xkft5pVfSPYTt5FpgyWbSkrksfdTTmqHXdnRzVrKwCNBz8GZ9K84gHwLCtM8
neutron.passwd: 43ykPZYhjyhpkjSNJd73yNRnpjty89VZTmwZkz3Yscy5pn8nH25qJ9Nt4g9JNRCG
nova.passwd: tMRPBgXXzn2rwbSsHR96Ss9XymFypLSprZctJk6TbKck2rf2Zh9xPggzYGGZqYbR

Revision history for this message
James Page (james-page) wrote :

This also looks incomplete:

__leader_get_migrated_settings__: '["nova.passwd", "neutron.passwd"]'

James Page (james-page)
Changed in rabbitmq-server (Juju Charms Collection):
status: Confirmed → In Progress
assignee: nobody → James Page (james-page)
James Page (james-page)
Changed in rabbitmq-server (Juju Charms Collection):
status: In Progress → Fix Committed
James Page (james-page)
Changed in rabbitmq-server (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.