Permission denied when trying to resize images

We are hitting it 100% of times also in Yoga-Focal

We are hitting it on a yoga-focal customer cloud also:

```
  : libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/0ad7a673-f7b8-46e6-be52-4cbde42de918/disk' (as uid:64055, gid:108): Permission denied
    2022-08-26 19:06:55.948 430983 ERROR nova.virt.libvirt.guest libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/0ad7a673-f7b8-46e6-be52-4cbde42de918/disk' (as uid:64055, gid:108)
: Permission denied
    2022-08-26 19:06:55.950 430983 ERROR nova.virt.libvirt.driver [req-7b5583ca-63e1-4abd-a53c-f336829c65bb 6f072e73354f46aab5f3bc07d3f7bf0f 6078a05d207a4bdfb271add10119d7ca - 8e2b894c611a4a4e8c787169d353b
44b 8e2b894c611a4a4e8c787169d353b44b] [instance: 0ad7a673-f7b8-46e6-be52-4cbde42de918] Failed to start libvirt guest: libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/0ad7a673-f7b8
-46e6-be52-4cbde42de918/disk' (as uid:64055, gid:108): Permission denied
    2022-08-26 19:06:55.956 430983 ERROR nova.compute.manager [req-7b5583ca-63e1-4abd-a53c-f336829c65bb 6f072e73354f46aab5f3bc07d3f7bf0f 6078a05d207a4bdfb271add10119d7ca - 8e2b894c611a4a4e8c787169d353b44b
8e2b894c611a4a4e8c787169d353b44b] [instance: 0ad7a673-f7b8-46e6-be52-4cbde42de918] Setting instance vm_state to ERROR: libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/0ad7a673-f7b
8-46e6-be52-4cbde42de918/disk' (as uid:64055, gid:108): Permission denied
```

So my initial thought was that this was an apparmor enforcement issue, and that does not appear to be the case. It clearly looks like its a problem with the uid ( I believe 64055 in this case is libvirt-qemu) and gid ( ??? ). However, our data capture does not collect this (grrrr...).

I suspect that this is due to some restrictions of permissions on directories somewhere. Will need to recreate it in order to walk through it and see what this issue is. If there is the possibility of providing the permissions on the directory and fail (as well as the information to confirm the uid/gid - i.e. which uid and gid this is) it would help narrow it down. Until then, a recreate is necessary.

With multiple reports, I will however move this bug to the confirmed state while we continue to triage this.

Subscribing field-crit as it is being encountered on a customer deployment

I was able to recreate this using the exact rules for hardening compute nodes as the bug reporter used.

$ openstack server create c1 --image cirros --flavor m1.tiny # this is successful

On the compute node where that server lands:

ubuntu@juju-da8cbf-zaza-41517351cce1-27:~$ sudo ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35
total 20724
drwxr-xr-x 2 nova nova 4096 Aug 31 16:52 .
drwxr-xr-x 7 nova nova 4096 Aug 31 16:52 ..
-rw------- 1 root root 28411 Aug 31 16:52 console.log
-rw-r--r-- 1 libvirt-qemu kvm 21233664 Aug 31 16:53 disk
-rw-r--r-- 1 nova nova 79 Aug 31 16:52 disk.info

$ openstack server resize --flavor 2 c1

This results in server going into ERROR state and destination nova-compute.log contains:
libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35/disk' (as uid:64055, gid:108): Permission denied

On the source compute node there's a slight change in ownership which is probably fine, but might as well document that here:

ubuntu@juju-da8cbf-zaza-41517351cce1-27:~$ ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35_resize
total 20784
drwxr-xr-x 2 nova nova 4096 Aug 31 16:52 .
drwxr-xr-x 7 nova nova 4096 Aug 31 16:58 ..
-rw------- 1 root root 28571 Aug 31 16:58 console.log
-rw-r--r-- 1 nova nova 21299200 Aug 31 16:58 disk
-rw-r--r-- 1 nova nova 79 Aug 31 16:52 disk.info

On the compute node where the resized instance lands, we see the following:

ubuntu@juju-da8cbf-zaza-41517351cce1-28:~$ sudo ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35
total 20816
drwxrwx--- 2 nova nova 4096 Aug 31 16:58 .
drwxr-xr-x 6 nova nova 4096 Aug 31 16:58 ..
-rw------- 1 root root 0 Aug 31 16:58 console.log
-rw-r----- 1 nova nova 21299520 Aug 31 16:58 disk
-rw-r----- 1 nova nova 79 Aug 31 16:58 disk.info

The file permissions look like they may have been created with a new default umask as they have 640 mode set instead of 644. This is likely cause of the access denial because libvirt-qemu no longer has read access.

Looking at umask on a non-hardened unit:

ubuntu@juju-da8cbf-zaza-41517351cce1-24:~$ umask
0002
ubuntu@juju-da8cbf-zaza-41517351cce1-24:~$ touch /tmp/hello
ubuntu@juju-da8cbf-zaza-41517351cce1-24:~$ ls -al /tmp/hello
-rw-rw-r-- 1 ubuntu ubuntu 0 Aug 31 17:20 /tmp/hello

And on a hardened compute node:

ubuntu@juju-da8cbf-zaza-41517351cce1-26:~$ umask
0027
ubuntu@juju-da8cbf-zaza-41517351cce1-26:~$ touch /tmp/hello
ubuntu@juju-da8cbf-zaza-41517351cce1-26:~$ ls -al /tmp/hello
-rw-r----- 1 ubuntu ubuntu 0 Aug 31 17:20 /tmp/hello

The umask is set to 0027 by rule 5.4.5:

Ensure default user umask is 027 or more restrictive
Execute rule 5.4.5

There's a long history of bugs related to attempts to limit access to files/dirs in /var/lib/nova. Considering that this is the most recent bug opened that is related to this topic, I will leave a history summary here in case we need it in the future.

= (1) April 2020: change permissions of /var/lib/nova to 640 and 750 ==

In the focal development cycle we set file permissions under /var/lib/nova to 640 and directory permissions to 0750. That was done as part of an effort across all the openstack packages via LP: #1859422
For nova, that was handled in the following commit: https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/?id=6acf39faa09ff7cfae799513a05fefbefa568abf

= (2) April 2020: add libvirt-qemu to nova group =

As a result of the previous change, the following bug was opened because the libvirt-qemu user needed access to /var/lib/nova/instances/_base: LP: #1870415
To fix that, we added the libvirt-qemu user to the nova group, since /var/lib/nova is owned by nova:nova.
That was handled in the following commit: https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/?id=a6eb3638389bb0818db6ebd3386eb8fe500487c6

= (3) June 2020: change permissions of /var/lib/nova to 644 and 755 =

The previous changes turned out to cause access issues and the following bug was opened: "unable to start and stop an instance post ugprade": LP: #1885269
As a result we reset file permissions under /var/lib/nova to 644 and directory permissions to 0755.
That was handled in the following commit: https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/?id=327e37428a25df3b96f5dfb9d08d3bc02caaff4f

= (4) Sept 2020: drop libvirt-qemu from nova group =

The following bug was opened due to instance snapshots being broken: LP: #1896617
Removing the libvirt-qemu user from the nova group fixed this.
This was handled in the following commits:
https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/?id=864aa4e744e9f7495caa353ba24efd2c4f7306cc
https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/?id=5e120a021d2d4440fff2f6f25ac7bd6955a5e0d0

= (5) March 2022: don't change permissions of /var/lib/nova/.ssh/id_rsa =

We then had a bug opened as the 0644 permissions for '/var/lib/nova/.ssh/id_rsa' were too open: LP: #1904580
That was handled in the following commit: https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/?id=655b04243c5fd7e6dc32bb722b0242c4efbb65e7

= (6) April 2022: Permission denied when trying to resize instance after CIS hardening =

And finally, the current bug we're looking at was opened due to instance resizing failing on a CIS hardened machine.
As part of hardening the umask is set to 0027. Therefore, new file permissions get 640 mode set instead of 644. This causes an access denial for the libvirt-qemu user as it no longer has read access to the disk. For example:

ubuntu@juju-da8cbf-zaza-41517351cce1-28:~$ sudo ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35
total 20816
drwxrwx--- 2 nova nova 4096 Aug 31 16:58 .
drwxr-xr-x 6 nova nova 4096 Aug 31 16:58 ..
-rw------- 1 root root 0 Aug 31 16:58 console.log
-rw...

I've done some successful testing (and will do some more) with all of the following changes in place:

(1) Set initial file permissions under /var/lib/nova to 640 and directory permissions to 0750. This would be handled in debian/nova-common.postinst.

(2) Add libvirt-qemu user back to the nova group. This would be handled in nova-compute-libvirt.postinst.

(3) Add nova user to kvm group. This would be handled in nova-compute.kvm.postinst. This will allow nova to access files in /var/lib/nova owned by the kvm group. For example:
root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184924
drwxr-x--- 2 nova nova 4096 Jun 26 10:39 ./
drwxr-x--- 5 nova nova 4096 Jun 26 10:38 ../
-rw------- 1 root root 8039 Jun 26 10:39 console.log
-rw-r----- 1 libvirt-qemu kvm 189399040 Jun 26 10:39 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info

(4) Again, this is summarized in more detail at: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/comments/64
To fix this, we will carry a patch in the ubuntu package that changes the tempdir permissions from 0o701 to 0o710. This will allow the libvirt-qemu user to access files in the tempdir. For example:
For example:
$ sudo ls -al /var/lib/nova/instances/snapshots/tmpkajuir8o
total 204
drwx-----x 2 nova nova 4096 Sep 23 19:12 . # <--- change to 0o710 will give libvirt-qemu user access
drwxr-x--- 3 nova nova 4096 Sep 23 19:12 ..
-rw-r--r-- 1 nova nova 197248 Sep 23 19:12 0ece1fb912104f2c849ea4bd6036712c.delta

(5) Leave this as-is and ensure that we don't regress it with any new changes.

(6) This will be solved by (2) where we add the libvirt-qemu user to the nova group. For example, on a hardened system this will allow libvirt-qemu to get read access to the disk file:
ubuntu@juju-da8cbf-zaza-41517351cce1-28:~$ sudo ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35
total 20816
drwxrwx--- 2 nova nova 4096 Aug 31 16:58 .
drwxr-xr-x 6 nova nova 4096 Aug 31 16:58 ..
-rw------- 1 root root 0 Aug 31 16:58 console.log
-rw-r----- 1 nova nova 21299520 Aug 31 16:58 disk
-rw-r----- 1 nova nova 79 Aug 31 16:58 disk.info

Assuming final tests check out, I would like to fix this all in the Zed version of nova via the current bug that we're looking at.

These are the tests that I've run to verify these fixes:

== tempest regression ==

This is our standard deployment of openstack regression testing with tempest to ensure there is no regression in the tests that get executed.

== server resize ==

Ensure the following is successful:

openstack server create c1 --image cirros --flavor m1.tiny --key-name nova-key --security-group 8faf8920-eaad-42ea-a91f-dc78f13a6d36 --nic net-id=036e1cc8-9a40-494b-944d-a0386fe874d7
openstack server resize --flavor 2 c1
openstack server resize confirm c1

Also ensure resize is successful on CIS hardened system.

== instance snapshot ==

Ensure the following is successful:

openstack server image create --name c1-image c1

== CIS hardening test ==

Repeat the image snapshot after CIS hardening on nova-computes, using the rules listed in this bug description.

This is fix released in the kinetic nova package version 3:25.0.1+git2022091213.11cb31258f-0ubuntu1.

also fixed in yoga: https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/nova/commit/4610e698a8a37768f479e423e6e9fcc5d925a158?h=stable/yoga

(although not released yet)

This hasn't been backported to yoga yet.

I'm hesitant to backport this to jammy (yoga). As noted above this has been a delicate area and there are several fixes to consider backporting as a whole to ensure correct access in place. Fixes would also have to be approved by the Ubuntu SRU team (https://wiki.ubuntu.com/StableReleaseUpdates).

Olivier in our team pointed out there is a potential workaround from a previous project:

As a workaround for the bug "Permission denied when trying to resize images", set back to 022 the default umask on the nova-compute units:

juju run --application nova-compute -- \
  "sudo sed -i 's/^UMASK.*027/UMASK 022/' /etc/login.defs"

Status changed to 'Confirmed' because the bug affects multiple users.