nova_compute after wallaby (in xena) no longer mounts cinder NFS paths
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla-ansible |
New
|
Undecided
|
Unassigned |
Bug Description
We have multiple cinder services via RBD, iSCSI/LVM, and QCOW2/NFS in one of our stacks. Having completed the migration from wallaby to xena, we can no longer boot any VMs with NFS volumes despite the NFS mounts showing up correctly on the cinder_volume containers and `qemu-img info` working properly as `nova` from those mounts.
The nova_compute containers (even after migrating the VM post-upgrade to another host to reset associations) do not mount the NFS path and when they try to access the volume for any reason they throw:
```
2021-12-26 22:18:40.668 7 ERROR oslo_messaging.
```
Checking the path at `/var/lib/
In summary: nova_compute doesnt start the NFS mount for VMs before trying to access the volume and it appears to remove or bind the underlying FS over the mounts.
I specifically tested this with a nova compute host running containers via overlay atop ext4 and aufs atop ZFS (our normal config) to eliminate that concern - same deal, no NFS mount, sad instances, nothing in the nova compute logs about even trying to mount the NFS export which does get mounted by the cinder-volume containers correctly.
Debug logs show that nova-compute is aware of the mounts: libvirt. volume. mount [req-1fba932d- 3dcd-42aa- 94a5-91ca999025 88 3c4f2250c3004f8 9be7e9057162172 74 e5dba6e61405445 582de5b930f1916 06 - default default] _HostMountState .mount( fstype= nfs, export= <NFS_HOST_ IP>:/srv/ nfs, vol_name= volume- 09c2935f- 1aa3-4bae- 90bc-ae9531302c 39, /var/lib/ nova/mnt/ 893cab8cdeb3fe1 a058e59540fdd53 79, options=[]) generation 0 mount /var/lib/ kolla/venv/ lib/python3. 8/site- packages/ nova/virt/ libvirt/ volume/ mount.py: 287 [root@< HOSTNAME> /]# ls /var/lib/ nova/mnt/ 893cab8cdeb3fe1 a058e59540fdd53 79, nova/mnt/ 893cab8cdeb3fe1 a058e59540fdd53 79,': No such file or directory
```
2021-12-26 22:39:55.691 7 DEBUG nova.virt.
```
Within the container, that parent-path does not exist:
```
(nova-compute)
ls: cannot access '/var/lib/
```
so the NFS mount can't be created. Creating that path inside /var/lib/nova/mnt results in it being removed