[upgrade] versions N and N+1 are not compatible

According to guide [1], while upgrading OpenStack services, a service in given version N should be compatible with its N+1 version. This is important when not being able to upgrade the cloud all at once, or when you have several nodes running the same services.

I performed some testing upgrading from Ocata to Pike, and Pike to Queens, following the upgrade order in [2], and I encountered the following issues:

Ocata => Pike
=============

After neutron-gateway upgrade, nova-api-metadata could not talk to nova-cloud-controller services. It continuosly printed the error message below and new instances could not obtain metadata:

2019-04-11 19:04:39.560 7011 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.22 of Service is not supported

When running "curl 169.254.169.254" the error message was:

500 Internal Server Error

Remote metadata server experienced an internal server error.

After nova-cloud-controller was upgraded, the nova-api-metadata functionality was restored, but the errors below were observed, rendering the nova API mostly useless:

Remote error: UnsupportedVersion Endpoint does not support RPC version 4.4. Attempted method: select_destinations

+ openstack server add volume ins2_old v2_old
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.rpc.client.RemoteError'> (HTTP 500) (Request-ID: req-757f2fc1-a622-4501-9934-e7d66b2d84e9)

+ openstack server add volume ins1_old v1_old
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.rpc.client.RemoteError'> (HTTP 500) (Request-ID: req-91552a98-5f93-430a-92c5-60d1c95fe6e9)

$ openstack server list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_versionedobjects.exception.IncompatibleObjectVersion'> (HTTP 500) (Request-ID: req-d48dc209-ccde-4d4e-b00f-1a50637db30a)

Upgrading nova-compute restored functionality.

Pike => Queens
==============

Upgrading neutron-gateway did not cause nova-api-metadata problems when upgrading to Queens, but upgrading nova-cloud-controller did:

+ openstack server migrate ins2_old --live juju-335fd8-upgrade-27
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.rpc.client.RemoteError'> (HTTP 500) (Request-ID: req-998cbe26-af0b-40d5-b8fe-58a9a5dfdcb7)

+ openstack server add volume ins2_old v2_old
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.rpc.client.RemoteError'> (HTTP 500) (Request-ID: req-a49ef959-38da-4359-aef8-5fa0872a662c)

++ openstack server show ins1_new -c status -f value
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_versionedobjects.exception.IncompatibleObjectVersion'> (HTTP 500) (Request-ID: req-03f78531-1f6d-4ac5-b06b-c588fb15b037)

Remote error: UnsupportedVersion Endpoint does not support RPC version 5.0. Attempted method: check_can
_live_migrate_destination

2019-04-16 20:09:15.485 30853 ERROR nova.api.openstack.wsgi [req-1c96d08b-3182-4c89-b757-8304884ac772 946bbb70210e42d1b08798e006adef1b f2bf2b35d763401099510ee4b5f07ea4 - default default] Unexpected exception in API method: RemoteError: Remote error: UnsupportedVersion Endpoint does not support RPC version 5.0. Attempted method: reserve_block_device_name

Again, upgrading nova-compute restored cloud functionality.

=================================

Most of those problems seem related to the fact that the current charm implementations do not allow locking down the RPC version when performing upgrades. as suggested by [1].

[1] https://docs.openstack.org/operations-guide/ops-upgrades.html

[2] https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html#ha-with-pause-resume