OpenStack Masakari Charm

[Charms 21.04] start hook fails after upgrade to Wallaby

Bug #1940627 reported by Radu Malica on 2021-08-20

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Ceph RBD Mirror Charm	Fix Released	Undecided	Unassigned	Ceph RBD Mirror Charm 21.04
Gnocchi Charm	Fix Released	High	Unassigned	Gnocchi Charm 21.04
MySQL InnoDB Cluster Charm	Fix Released	High	Unassigned	MySQL InnoDB Cluster Charm 21.04
MySQL Router Charm	Fix Released	High	Unassigned	MySQL Router Charm 21.04
OpenStack AODH Charm	Fix Released	High	Unassigned	OpenStack AODH Charm 21.04
OpenStack Barbican Charm	Fix Released	Undecided	Unassigned	OpenStack Barbican Charm 21.04
OpenStack Barbican-Vault Charm	Fix Released	Undecided	Unassigned
OpenStack Ceph-FS Charm	Fix Released	High	Unassigned	OpenStack Ceph-FS Charm 21.04
OpenStack Cinder Backup Swift Proxy Charm	Fix Released	Undecided	Unassigned	OpenStack Cinder Backup Swift Proxy Charm 21.04
OpenStack Cinder Pure Storage Charm	Fix Released	Undecided	Unassigned
OpenStack Designate Charm	Fix Released	Undecided	Unassigned	OpenStack Designate Charm 21.04
OpenStack Designate-Bind Charm	Fix Released	Undecided	Unassigned	OpenStack Designate-Bind Charm 21.04
OpenStack Ironic API Charm	Fix Released	High	Unassigned	OpenStack Ironic API Charm 21.04
OpenStack Ironic Conductor Charm	Fix Released	High	Aurelien Lourot	OpenStack Ironic Conductor Charm 21.04
OpenStack Keystone Kerberos Charm	Fix Released	Undecided	Unassigned	OpenStack Keystone Kerberos Charm 21.04
OpenStack Keystone LDAP integration	Fix Released	Undecided	Unassigned	OpenStack Keystone LDAP integration 21.04
OpenStack Manila Charm	Fix Released	Undecided	Unassigned	OpenStack Manila Charm 21.04
OpenStack Manila Generic Backend Charm	Fix Released	Undecided	Unassigned	OpenStack Manila Generic Backend Charm 21.04
OpenStack Manila-Ganesha Charm	Fix Released	High	Aurelien Lourot	OpenStack Manila-Ganesha Charm 21.04
OpenStack Masakari Charm	Fix Released	High	Unassigned	OpenStack Masakari Charm 21.04
OpenStack Masakari Monitors Charm	Fix Released	Undecided	Unassigned	OpenStack Masakari Monitors Charm 21.04
OpenStack Neutron API Arista Plugin Charm	Fix Released	Undecided	Unassigned	OpenStack Neutron API Arista Plugin Charm 21.04
OpenStack Neutron API Ironic Plugin Charm	Fix Released	High	Aurelien Lourot	OpenStack Neutron API Ironic Plugin Charm 21.04
OpenStack Neutron API OVN Plugin Charm	Fix Released	High	Aurelien Lourot	OpenStack Neutron API OVN Plugin Charm 21.04
OpenStack Neutron Dynamic Routing charm	Fix Released	Undecided	Unassigned	OpenStack Neutron Dynamic Routing charm 21.04
OpenStack Nova Cell Controller Charm	Fix Released	High	Unassigned	OpenStack Nova Cell Controller Charm 21.04
OpenStack Octavia Charm	Fix Released	High	Aurelien Lourot	OpenStack Octavia Charm 21.04
OpenStack Octavia Dashboard Charm	Fix Released	High	Aurelien Lourot	OpenStack Octavia Dashboard Charm 21.04
OpenStack Pacemaker Remote Charm	Fix Released	Undecided	Unassigned
OpenStack Placement Charm	Fix Released	Undecided	Unassigned	OpenStack Placement Charm 21.04
charm-octavia-diskimage-retrofit	Fix Released	High	Aurelien Lourot	charm-octavia-diskimage-retrofit 21.04
charm-ovn-central	Fix Released	High	Unassigned	charm-ovn-central 21.04
charm-ovn-chassis	Fix Released	Undecided	Unassigned	charm-ovn-chassis 21.04
charm-ovn-dedicated-chassis	Fix Released	High	Unassigned	charm-ovn-dedicated-chassis 21.04
vault-charm	Fix Released	Undecided	Unassigned	vault-charm 21.04

Bug Description

I am running openstack version Victoria on Focal Ubuntu deployed with Juju charms and i wanted to upgrade to Wallaby

Everything worked fine by the book until i ran the upgrade on Masakari app with the following commands:

juju config masakari action-managed-upgrade=True
juju config masakari openstack-origin=cloud:focal-wallaby

charm version is 11.0.0, revision 11 (latest):
masakari 11.0.0 error 3 masakari charmstore 11 ubuntu hook failed: "start"

The Masakari stack is having 3 instances with HA cluster deployed. The HA cluster upgraded successfully

masakari/6 error idle 4/lxd/4 10.118.0.138 15868/tcp hook failed: "start"
  masakari-hacluster/3 active idle 10.118.0.138 Unit is ready and clustered
masakari/7 error idle 5/lxd/1 10.118.0.137 15868/tcp hook failed: "start"
  masakari-hacluster/5 active idle 10.118.0.137 Unit is ready and clustered
masakari/8* error idle 3/lxd/6 10.118.0.136 15868/tcp hook failed: "start"
  masakari-hacluster/4* active idle 10.118.0.136 Unit is ready and clustered

From the logs on each instances i can see this error:

unit-masakari-7: 07:24:57 ERROR unit.masakari/7.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-masakari-7/charm/reactive/masakari_handlers.py", line 40, in render_config
    charm_class.upgrade_if_available(args)
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1142, in upgrade_if_available
    if self.openstack_upgrade_available(self.release_pkg):
  File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1109, in openstack_upgrade_available
    raise RuntimeError(
RuntimeError: In charms_openstack.charm.core.openstack_upgrade_available() cur_vers=None and avail_vers=2021.1, one of which is None. This usually implies that the openstack version is not present in the self.package_codenames or os_utils.PACKAGE_CODENAMES. Please re-visit and fix.

unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start Traceback (most recent call last):
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/charm/hooks/start", line 22, in <module>
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start main()
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start bus.dispatch(restricted=restricted_mode)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start _invoke(other_handlers)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start handler.invoke()
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start self._action(*args)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/charm/reactive/masakari_handlers.py", line 40, in render_config
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start charm_class.upgrade_if_available(args)
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1142, in upgrade_if_available
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start if self.openstack_upgrade_available(self.release_pkg):
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start File "/var/lib/juju/agents/unit-masakari-7/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1109, in openstack_upgrade_available
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start raise RuntimeError(
unit-masakari-7: 07:24:57 WARNING unit.masakari/7.start RuntimeError: In charms_openstack.charm.core.openstack_upgrade_available() cur_vers=None and avail_vers=2021.1, one of which is None. This usually implies that the openstack version is not present in the self.package_codenames or os_utils.PACKAGE_CODENAMES. Please re-visit and fix.
unit-masakari-7: 07:24:58 ERROR juju.worker.uniter.operation hook "start" (via explicit, bespoke hook script) failed: exit status 1

It seems the charm is unable to get the openstack release from the OS, which is the same LXD containers with Focal as all other openstack components.

From the error above i am assuming that Masakari didn't even upgrade to Wallaby, since the script cannot find the cur_vers of the Openstack (which was Victoria and running properly)

Thanks

See original description

Tags:

Radu Malica (radumalica) on 2021-08-20

description:

updated

Frode Nordahl (fnordahl) on 2021-08-23

Changed in charm-masakari:
status:	New → Triaged
importance:	Undecided → High
tags:	added: openstack-upgrade

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2021-08-23:

This is happening due to https://review.opendev.org/c/openstack/charms.openstack/+/787641 missing in the stable/21.04 charms.

Revision history for this message

Radu Malica (radumalica) wrote on 2021-08-23:

Thanks for the clarification, is there any workaround for this , or should I wait for the openstack-charmers to update the masakari charm ?

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2021-08-23:

This would be an issue for all the stable reactive charms at this point in time, the only available workaround would be to switch over to the -next charms that track the main branch (cs:~openstack-charmers-next/CHARM). There is of course risk involved in doing that as we are still in the development cycle and they have not received pre-release testing. So if this is for a production environment it would most likely be best to wait until the issue is resolved for all the reactive charms.

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2021-08-27:

https://review.opendev.org/c/openstack/charm-masakari/+/805810 has now landed, which must have triggered a rebuild and picked up https://review.opendev.org/c/openstack/charms.openstack/+/787641 , which solves this bug.

Now creating a 21.04 batch for all reactive charms: https://review.opendev.org/q/topic:wallaby-bundles-stable%252F21.04

Changed in charm-masakari:
status:	Triaged → Fix Released
milestone:	none → 21.04

Felipe Reyes (freyes) on 2021-08-31

Changed in charm-mysql-innodb-cluster:
status:	New → Fix Released
importance:	Undecided → High
milestone:	none → 21.04
Changed in charm-aodh:
status:	New → Fix Released
milestone:	none → 21.04
Changed in charm-gnocchi:
status:	New → Fix Released
milestone:	none → 21.04
Changed in charm-nova-cell-controller:
status:	New → Fix Released
milestone:	none → 21.04
Changed in charm-ovn-central:
status:	New → Fix Released
milestone:	none → 21.04

Felipe Reyes (freyes) on 2021-08-31

Changed in charm-mysql-router:
status:	New → Fix Released
milestone:	none → 21.04
Changed in charm-ovn-dedicated-chassis:
status:	New → Fix Released
milestone:	none → 21.04
Changed in charm-gnocchi:
importance:	Undecided → High
Changed in charm-aodh:
importance:	Undecided → High
Changed in charm-nova-cell-controller:
importance:	Undecided → High
Changed in charm-mysql-router:
importance:	Undecided → High
Changed in charm-ovn-central:
importance:	Undecided → High
Changed in charm-ovn-dedicated-chassis:
importance:	Undecided → High

Aurelien Lourot (aurelien-lourot) on 2021-09-01

Changed in charm-octavia-diskimage-retrofit:
status:	New → In Progress
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
Changed in charm-manila-ganesha:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
status:	New → In Progress
Changed in charm-octavia-dashboard:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
status:	New → In Progress
Changed in charm-barbican-vault:
status:	New → Fix Released

Aurelien Lourot (aurelien-lourot) on 2021-09-01

Changed in charm-manila:
milestone:	none → 21.10
status:	New → Fix Released
milestone:	21.10 → 21.04
Changed in charm-cinder-backup-swift-proxy:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-neutron-api-plugin-arista:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-ovn-chassis:
milestone:	none → 21.04
status:	New → Fix Released
summary:	- Masakari starting fail after upgrade to Wallaby + start hook fails after upgrade to Wallaby
Changed in vault-charm:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-manila-ganesha:
milestone:	none → 21.04
Changed in charm-octavia-dashboard:
milestone:	none → 21.04
Changed in charm-octavia-diskimage-retrofit:
milestone:	none → 21.04

Aurelien Lourot (aurelien-lourot) on 2021-09-01

Changed in charm-barbican:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-cinder-purestorage:
status:	New → Fix Released
Changed in charm-designate:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-designate-bind:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-keystone-ldap:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-manila-generic:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-masakari-monitors:
milestone:	none → 21.04
status:	New → Fix Released

Aurelien Lourot (aurelien-lourot) on 2021-09-01

Changed in charm-pacemaker-remote:
status:	New → Fix Released
Changed in charm-ceph-rbd-mirror:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-keystone-kerberos:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-neutron-dynamic-routing:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-ironic-api:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
milestone:	none → 21.04
status:	New → In Progress
summary:	- start hook fails after upgrade to Wallaby + [21.04] start hook fails after upgrade to Wallaby
summary:	- [21.04] start hook fails after upgrade to Wallaby + [Charms 21.04] start hook fails after upgrade to Wallaby

Aurelien Lourot (aurelien-lourot) on 2021-09-01

Changed in charm-ironic-conductor:
milestone:	none → 21.04
status:	New → In Progress
Changed in charm-neutron-api-plugin-ironic:
milestone:	none → 21.04
status:	New → In Progress
Changed in charm-placement:
milestone:	none → 21.04
status:	New → Fix Released
Changed in charm-manila-ganesha:
importance:	Undecided → High
Changed in charm-octavia:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
importance:	Undecided → High
milestone:	none → 21.04
status:	New → In Progress
Changed in charm-ironic-api:
importance:	Undecided → High
Changed in charm-ironic-conductor:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
importance:	Undecided → High
Changed in charm-neutron-api-plugin-ironic:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
importance:	Undecided → High
Changed in charm-octavia-dashboard:
importance:	Undecided → High
Changed in charm-octavia-diskimage-retrofit:
importance:	Undecided → High

Aurelien Lourot (aurelien-lourot) on 2021-09-01

Changed in charm-ceph-fs:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
importance:	Undecided → High
milestone:	none → 21.04
status:	New → In Progress
Changed in charm-neutron-api-plugin-ovn:
assignee:	nobody → Aurelien Lourot (aurelien-lourot)
importance:	Undecided → High
milestone:	none → 21.04
status:	New → In Progress

Aurelien Lourot (aurelien-lourot) on 2021-09-21

Changed in charm-ironic-api:
assignee:	Aurelien Lourot (aurelien-lourot) → nobody
status:	In Progress → Fix Released
Changed in charm-ceph-fs:
assignee:	Aurelien Lourot (aurelien-lourot) → nobody
status:	In Progress → Fix Released

Revision history for this message

Chris MacNaughton (chris.macnaughton) wrote on 2021-10-25:

The 21.20 release has now gone out so I'm marking the rest of the in-progress targets as fix-released!

Changed in charm-manila-ganesha:
status:	In Progress → Fix Released
Changed in charm-octavia:
status:	In Progress → Fix Released
Changed in charm-ironic-conductor:
status:	In Progress → Fix Released
Changed in charm-neutron-api-plugin-ironic:
status:	In Progress → Fix Released
Changed in charm-neutron-api-plugin-ovn:
status:	In Progress → Fix Released
Changed in charm-octavia-dashboard:
status:	In Progress → Fix Released
Changed in charm-octavia-diskimage-retrofit:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-12-16: Change abandoned on charm-octavia-dashboard (stable/21.04)

Change abandoned by "James Page <email address hidden>" on branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-octavia-dashboard/+/806227
Reason: Review targets obsolete stable branch

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.