ussuri: unable to stop and start an instance post upgrade

Linked bug 691590

Additional details from a compute unit:

on nova/compute/1 -- before stopping the unit, but after upgrading from train->ussuri on bionic. Note that the disk, console.log are owned nova.nova.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184692
drwxr-x--- 2 nova nova 4096 Jun 25 17:46 ./
drwxr-x--- 5 nova nova 4096 Jun 25 18:21 ../
-rw-r----- 1 nova nova 31316 Jun 25 17:47 console.log
-rw-r----- 1 nova nova 189136896 Jun 26 10:17 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info
root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90

After stopping the instance, the disk is now owned root.root. As there is no world-readable perm, nova now can't restart the instance.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184948
drwxr-x--- 2 nova nova 4096 Jun 25 17:46 ./
drwxr-x--- 5 nova nova 4096 Jun 25 18:21 ../
-rw-r----- 1 nova nova 31316 Jun 25 17:47 console.log
-rw-r----- 1 root root 189399040 Jun 26 10:38 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info

After fixing the permissions back to nova.nova and restarting the unit, the console.log is now owned by root, and the disk by libvert-qumu.kvm.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184924
drwxr-x--- 2 nova nova 4096 Jun 26 10:39 ./
drwxr-x--- 5 nova nova 4096 Jun 26 10:38 ../
-rw------- 1 root root 8039 Jun 26 10:39 console.log
-rw-r----- 1 libvirt-qemu kvm 189399040 Jun 26 10:39 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info

When this unit is stopped, it again goes back to root.root.

root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 326324
drwxr-x--- 2 nova nova 4096 Jun 26 10:39 ./
drwxr-x--- 5 nova nova 4096 Jun 26 10:38 ../
-rw------- 1 root root 29040 Jun 26 10:41 console.log
-rw-r----- 1 root root 334168064 Jun 26 10:41 disk
-rw-r----- 1 nova nova 79 Jun 25 17:46 disk.info
root@juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7#

The only real solution, for the moment, is to revert the permissions (as indicated in the summary) back to world readable for the disk and ../_disk/* files.

This bug was fixed in the package nova - 2:21.0.0-0ubuntu3

---------------
nova (2:21.0.0-0ubuntu3) groovy; urgency=medium

  * d/rules: Blacklist test_tcp_rst_no_compute_rpcapi.

 -- James Page <email address hidden> Tue, 30 Jun 2020 09:06:12 +0100

nova uploaded to focal-proposed for SRU team review.

Hello James, or anyone else affected,

Accepted nova into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:21.0.0-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verification completed - able to stop and then start an instance using the package from proposed.

verification completed in bionic-ussuri/proposed (although there are not comments or tags to manage here). Able to stop and then start an instance using the proposed package for nova.

This bug was fixed in the package nova - 2:22.0.0~b1~git2020070713.bc784a1c1f-0ubuntu1~cloud0
---------------

 nova (2:22.0.0~b1~git2020070713.bc784a1c1f-0ubuntu1~cloud0) focal-victoria; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (2:22.0.0~b1~git2020070713.bc784a1c1f-0ubuntu1) groovy; urgency=medium
 .
   * New upstream snapshot for OpenStack Victoria.
   * d/control: Align (Build-)Depends with upstream.
   * d/p/add-mysql8-compatibility.patch: Removed. Change landed upstream.
   * d/p/arm-console-patch.patch: Refreshed.
   * d/p/drop-sphinxcontrib-rsvgconverter.patch: Refreshed
 .
 nova (2:21.0.0-0ubuntu3) groovy; urgency=medium
 .
   * d/rules: Blacklist test_tcp_rst_no_compute_rpcapi.
 .
 nova (2:21.0.0-0ubuntu2) groovy; urgency=medium
 .
   * Correct file and directory permissions to ensure that instances
     can be stopped and started correctly (LP: #1885269):
     - d/nova-common.postinst: Reset file permissions under /var/lib/nova
       to 0644 and directory permissions to 0755 to ensure that instances
       can be stopped and started correctly with the workflow of ownership
       changes that occur between nova and libvirt/qemu.
     - d/nova-common.postinst: Don't force ownership to nova:nova.
     - d/nova-common.postinst: Reset ownership and permissions on
       console.log files to root:root and 0600.
 .
 nova (2:21.0.0-0ubuntu1) groovy; urgency=medium
 .
   * d/watch: Scope to 21.x series and get tarballs from opendev.org.
   * d/control: Align (Build-)Depends with upstream.
   * New upstream release for OpenStack Ussuri (LP: #1877642).

This bug was fixed in the package nova - 2:21.0.0-0ubuntu0.20.04.2

---------------
nova (2:21.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * Correct file and directory permissions to ensure that instances
    can be stopped and started correctly (LP: #1885269):
    - d/nova-common.postinst: Reset file permissions under /var/lib/nova
      to 0644 and directory permissions to 0755 to ensure that instances
      can be stopped and started correctly with the workflow of ownership
      changes that occur between nova and libvirt/qemu.
    - d/nova-common.postinst: Don't force ownership to nova:nova.
    - d/nova-common.postinst: Reset ownership and permissions on
      console.log files to root:root and 0600.

 -- James Page <email address hidden> Mon, 29 Jun 2020 13:54:40 +0100

The verification of the Stable Release Update for nova has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.