Removing nova-compute unit with scheduled but stopped VM breaks hypervisor-list api call

After removing nova-compute unit on node mycloud-cs-003, nova hypervisor-list stopped working and started to return below error to user and nova-api-os-compute.log on the nova-api server.

While this may be an upstream issue, I believe the charm should probably handle this edge case.

When querying the database, I find that the service and the compute_node entry for the host are both in deleted status, but I see that there is a scheduled vm on the node mycloud-cs-003. I went in and did a nova delete <instanceid> on the instance that was scheduled on that node, and that succeeded, but the "running_vms" total in compute_nodes table did not decrease, so I updated that row to running_vms = 0, and I'm still experiencing the below traceback in nova-api-os-compute.log.

2017-12-19 17:59:35.733 218705 DEBUG nova.api.openstack.wsgi [req-a597c96f-a372-4a0c-9c79-d2859f1612db 2ac8326863b64ea3ba9ba96a7ab70214 51419d6b9c8f475db199c24b0e50a99d - - -] Calling method '<bound method HypervisorsController.index of <nova.api.openstack.compute.hypervisors.HypervisorsController object at 0x7fc679ba5d10>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions [req-a597c96f-a372-4a0c-9c79-d2859f1612db 2ac8326863b64ea3ba9ba96a7ab70214 51419d6b9c8f475db199c24b0e50a99d - - -] Unexpected exception in API method
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py\", line 478, in wrapped
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ return f(*args, **kwargs)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/hypervisors.py\", line 88, in index
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ for hyp in compute_nodes])
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/compute/api.py\", line 3743, in service_get_by_compute_host
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ return objects.Service.get_by_compute_host(context, host_name)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py\", line 181, in wrapper
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ result = fn(cls, context, *args, **kwargs)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/nova/objects/service.py\", line 243, in get_by_compute_host
--
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/objects/service.py\", line 238, in _db_service_get_by_compute_host
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ return db.service_get_by_compute_host(context, host)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/nova/db/api.py\", line 163, in service_get_by_compute_host
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions return IMPL.service_get_by_compute_host(context, host)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions File \"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py\", line 330, in wrapped
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ File \"/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py\", line 585, in service_get_by_compute_host
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions \ raise exception.ComputeHostNotFound(host=host)
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions ComputeHostNotFound: Compute host mycloud-cs-003 could not be found.
2017-12-19 17:59:36.044 218705 ERROR nova.api.openstack.extensions
2017-12-19 17:59:36.046 218705 INFO nova.api.openstack.wsgi [req-a597c96f-a372-4a0c-9c79-d2859f1612db 2ac8326863b64ea3ba9ba96a7ab70214 51419d6b9c8f475db199c24b0e50a99d - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.ComputeHostNotFound'>

Steps to recreate:

1. deploy nova-cloud-controller and nova-compute with proper relations to keystone/mysql/etc
2. deploy a vm to the nova-compute environment
3. stop the instance
4. juju remove-unit <nova-compute/X> for the unit that the VM was scheduled on
5. nova hypervisor-list should exhibit this error.

Please let me know if this does not work.

Notes: this environment was previously upgraded from either icehouse or liberty to mitaka. (guessing liberty since the service deleted and compute_node deleted columns are ordered, incrementing numbers, and not just 0 or 1)

Running openstack 17.02 charms, I believe on trusty/mitaka cloud.