Hi,
I recently ran into this problem and I believe that it is a bug.
Environment Information:
OpenStack version: Yoga
Deployment tool: Kolla-Ansible
Cinder backend: Ceph RBD with multiple pools each as a RBD backend
Steps to reproduce:
1. Boot a VM with volume as root disk
2. Shutdown the VM so that volume can be retyped (will failed if VM is in active state)
3. Retype the VM's root volume from one RBD backend to another
4. Start the VM
Expected result:
VM booted with the retyped root volume
Actual result:
VM goes into error state as the libvirt xml nova creates refers to the old volume backend, which the volume have already been deleted there.
Volume attachment in database is still using the old Ceph pool, manually modified it but still getting the same error.
Nova traceback from dashboard:
volumes-hdd is the old backend Ceph pool
Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 201, in decorated_function return function(self, context, *args, **kwargs) File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 3866, in reboot_instance do_reboot_instance(context, instance, block_device_info, reboot_type) File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 391, in inner return f(*args, **kwargs) File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 3864, in do_reboot_instance self._reboot_instance(context, instance, block_device_info, File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 3958, in _reboot_instance self._set_instance_obj_error_state(instance) File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 3928, in _reboot_instance self.driver.reboot(context, instance, File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 3816, in reboot return self._hard_reboot(context, instance, network_info, File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 3951, in _hard_reboot self._create_guest_with_network( File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7522, in _create_guest_with_network self._cleanup( File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7499, in _create_guest_with_network guest = self._create_guest( File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7438, in _create_guest guest.launch(pause=pause) File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 168, in launch LOG.exception('Error launching a defined domain with XML: %s', File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 165, in launch return self._domain.createWithFlags(flags) File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit result = proxy_call(self._autowrap, f, *args, **kwargs) File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call rv = execute(f, *args, **kwargs) File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute six.reraise(c, e, tb) File "/usr/local/lib/python3.9/dist-packages/six.py", line 719, in reraise raise value File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker rv = meth(*args, **kwargs) File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags raise libvirtError('virDomainCreateWithFlags() failed') libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-12-15T07:56:18.597187Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"volumes-hdd","image":"volume-fc1cc3f7-a418-48a2-ad4a-ebcbb314677d","server":[{"host":"192.168.114.11","port":"6789"},{"host":"192.168.114.12","port":"6789"},{"host":"192.168.114.13","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from volume-fc1cc3f7-a418-48a2-ad4a-ebcbb314677d: No such file or directory
