Apparmor settings are not updated when config value for instances-path is changed

The aa-profile for the nova-compute daemon is managed by the charm; the virt-aa-helper profile provided by packaging currently supports the default nova location and anything under /srv - there is a different bug to make that more generic in approach.

libvirt bug 1644507

There are several other filesystem locations that the virt-aa-helper profile supports. I just checked Ubuntu 16.04 and the virt-aa-helper profile allows you to set instances-path to a location underneath any of these directories:

/home/*/ (except for hidden directories)
/root/ (except for hidden directories)
/var/lib/nova/instances/_base/
/media/
/mnt/
/opt/
/srv/

There are a few more that I didn't list because they were specific to certain tools.

Is it possible to move your instances-path under one of those directories?

If you can't move your instances-path to one of those locations, there's one other option. The virt-aa-helper profile could be updated to include the following line:

  #include <local/usr.lib.libvirt.virt-aa-helper>

Then the nova-compute charm could drop a rule, granting read access to the custom instances-path, into /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper. The charm would then need to reload the virt-aa-helper profile.

This is how this bug manifests itself.
Env: Focal/Ussuri
     libvirt-image-backend charm config option set to 'qcow2' (default)
     instances-path charm config option set to '/var/lib/nova/instance' (default: '/var/lib/nova/instances')

When trying to launch a VM, nova-compute.log contains following error message:

ERROR nova.compute.manager [instance: 74c9876b-2dce-48a2-96aa-45780e61a434] libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-11-21T14:00:11.340223Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/nova/instance/_base/e837e9a0c9a5769e9df7ae7e446c8efb9b285864","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/nova/instance/_base/e837e9a0c9a5769e9df7ae7e446c8efb9b285864': Permission denied

At first it might be misleading a bit because checking the actual permissions and the ownership of the directories on this path are correct.

Workaround:
1. Edit /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper and add following lines to it (based on the example above) somewhere around the following lines for the default location:
# existing lines
 66 /var/lib/nova/instances/_base/** r,
 67 # nova snapshots (LP: #1244694)
 68 /var/lib/nova/instances/snapshots/** r,
# new lines to be added pointing to the correct path from the instances-path
/var/lib/nova/instance/_base/** r,
/var/lib/nova/instance/snapshots/** r,

2. reload aa profiles into the kernel
sudo apparmor_parser -r /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

Maybe the information about this bug should be also added to the charm description [1] until this problem is fixed, so anyone willing to use non-default instances-path config value is aware of this. In a similar way to the other bug reference in the 'SSH keys and VM migration' section of the charm description.

[1] https://charmhub.io/nova-compute