OpenStack Compute (nova)

[SRU]"TypeError: catching classes that do not inherit from BaseException is not allowed" is raised if volume mount fails in python3

Series zed
Bug #1984736

Bug #1984736 reported by melanie witt on 2022-08-10

This bug affects 3 people

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Medium	melanie witt
Antelope	Fix Released	Undecided	Unassigned
Wallaby	In Progress	Undecided	Unassigned
Xena	In Progress	Undecided	Unassigned
Yoga	In Progress	Undecided	Unassigned
Zed	In Progress	Undecided	Unassigned
Ubuntu Cloud Archive	Fix Released	Undecided	Unassigned
Yoga	In Progress	Medium	Erlon R. Cruz
nova (Ubuntu)	Fix Released	Undecided	Unassigned
Jammy	Fix Committed	Medium	Erlon R. Cruz

Bug Description

*********** SRU TEMPLATE AT THE BOTTOM ***************

Saw this on a downstream CI run where a volume mount failed:

2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [req-67e1cef8-e30a-4a47-8010-9e966fd30fce 8882186b6a324a0e9fb6fd268d337cce 8b290d651e9b42fd89c95b5e2a9a25fb - default default] [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Failed to attach 5a6a5f37-0888-44b2-9456-cf087ae8c356 at /dev/vdb: TypeError: catching classes that do not inherit from BaseException is not allowed
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Traceback (most recent call last):
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/mount.py", line 305, in mount
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] nova.privsep.fs.mount(fstype, export, mountpoint, options)
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] return self.channel.remote_call(name, args, kwargs)
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] raise exc_type(*result[2])
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Command: mount -t nfs 192.168.1.50:/vol_cinder /var/lib/nova/mnt/724dab229d80c6a1a1e49a71c8356eed
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Exit code: 32
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Stdout: ''
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Stderr: 'Failed to connect to bus: No data available\nmount.nfs: Operation not permitted\n'
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177]
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] During handling of the above exception, another exception occurred:
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177]
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] Traceback (most recent call last):
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 7023, in _attach_volume
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] bdm.attach(context, instance, self.volume_api, self.driver,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 46, in wrapped
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] ret_val = method(obj, context, *args, **kwargs)
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 672, in attach
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] self._do_attach(context, instance, volume, volume_api,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 657, in _do_attach
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] self._volume_attach(context, volume, connector, instance,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 606, in _volume_attach
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] volume_api.attachment_delete(context,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] self.force_reraise()
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] raise self.value
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 595, in _volume_attach
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] virt_driver.attach_volume(
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 2052, in attach_volume
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] self._connect_volume(context, connection_info, instance,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1794, in _connect_volume
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] vol_driver.connect_volume(connection_info, instance)
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/fs.py", line 113, in connect_volume
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] mount.mount(self.fstype, export, vol_name, mountpoint, instance,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/mount.py", line 407, in mount
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] mount_state.mount(fstype, export, vol_name, mountpoint, instance,
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/mount.py", line 306, in mount
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] except processutils.ProcessExecutionError():
2022-07-29 11:56:57.606 2 ERROR nova.compute.manager [instance: 6a9a59d1-861d-4536-84ed-e54d817f0177] TypeError: catching classes that do not inherit from BaseException is not allowed

There's a piece of code in nova/virt/libvirt/volume/mount.py that is catching an instance of a class instead of a class [1]:

                try:
                    nova.privsep.fs.mount(fstype, export, mountpoint, options)
                except processutils.ProcessExecutionError():

This results in a TypeError in python3 but not in python2.

Python 2.7.18 (default, Jul 1 2022, 12:27:04)
[GCC 9.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
... raise RuntimeError('oops!')
... except RuntimeError():
... print('hi')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
RuntimeError: oops!

Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
... raise RuntimeError('oops!')
... except RuntimeError():
... print('hi')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
RuntimeError: oops!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 3, in <module>
TypeError: catching classes that do not inherit from BaseException is not allowed

[1] https://github.com/openstack/nova/blob/1220d53c2fd6231bb569a54f004a0e858536e970/nova/virt/libvirt/volume/mount.py#L309

===============
SRU DESCRIPTION
===============

[Impact]

This issue prevents attaching volumes to instances when using the cinder-nfs backend. The workaround is cumbersome and need admin access to the host to mount the nfs share manually.

[Test case]

1) Deploy OpenStack with Cinder NFS backend (not fully configured) and have a test image in glance

2) Create a volume and confirm it was created successfully afterwards

openstack volume create --size 1 v1

openstack volume list

3) Create an instance. Confirm successful creation.

openstack server create --flavor 6 --image cirros-0.4.0 --network private ins1

openstack server list

4) Attempt to attach the volume to the instance. Confirm it hasn't attached.

openstack server add volume ins1 v1

openstack volume list

5) Verify in logs the error message

except processutils.ProcessExecutionError():

TypeError: catching classes that do not inherit from BaseException is not allowed

6) install updated package

7) repeat step 4. It should not show that same error but a different error instead related to improper NFS mounting such as "permission denied".

[Where problems could occur]

The change is extremely trivial as a tiny python error and scope is limited to the mount operation for nfs storage backend, which is much less frequently used than other storage backends. It is also limited to the exception block, so if there is any error in the changed code, it should produce the same symptom as already observed (which is failure to process the exception when a mounting error occurs. If no mounting errors occur, the code should proceed fine).

[Other Info]

See original description

Tags:

Related branches

~halves/ubuntu/+source/nova:yoga-25.2.1-0ubuntu2.9

Merged into ~ubuntu-openstack-dev/ubuntu/+source/nova:stable/yoga at revision a88bbf396b6e25c14527d9f4459e1c003e2d70a7

Guillaume Boutry (community): Approve on 2025-07-18

Ubuntu OpenStack uploaders: Pending requested 2025-07-02

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-10: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/852795

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-25: Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/852795
Committed: https://opendev.org/openstack/nova/commit/f996674c36fe1a109853d88113e27b44a7a5cf45
Submitter: "Zuul (22348)"
Branch: master

commit f996674c36fe1a109853d88113e27b44a7a5cf45
Author: melanie witt <email address hidden>
Date: Wed Aug 10 23:22:47 2022 +0000

Fix exception catch when volume mount fails

    When we call nova.privsep.fs.mount(), we are catching
    ProcessExecutionError() (an instance of a class) and that is not allowed
    in python3.

This changes the except statement to catch ProcessExecutionError
(class) instead.

Closes-Bug: #1984736

Change-Id: I24a269f9809063f864a4f0443d6724dfa8703d9d

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-21: Fix included in openstack/nova 28.0.0.0rc1

This issue was fixed in the openstack/nova 28.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-10-20: Fix proposed to nova (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/898954

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-14: Fix proposed to nova (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/900835

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-14: Fix proposed to nova (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/900836

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-14: Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/900837

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-14: Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/900838

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-02-06: Change abandoned on nova (stable/yoga)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/900836
Reason: stable/yoga branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/yoga if you want to further work on this patch.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-05: Change abandoned on nova (stable/wallaby)

#10

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/900838
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-05: Change abandoned on nova (stable/xena)

#11

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/900837
Reason: stable/xena branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/xena if you want to further work on this patch.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-04-29: Change abandoned on nova (stable/zed)

#12

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/900835
Reason: stable/zed branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/zed if you want to further work on this patch.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-01: Fix merged to nova (stable/2023.1)

#13

Reviewed: https://review.opendev.org/c/openstack/nova/+/898954
Committed: https://opendev.org/openstack/nova/commit/baa0b4ff87a044228438cc32c27d4424fa6bc0c4
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit baa0b4ff87a044228438cc32c27d4424fa6bc0c4
Author: melanie witt <email address hidden>
Date: Wed Aug 10 23:22:47 2022 +0000

Fix exception catch when volume mount fails

    When we call nova.privsep.fs.mount(), we are catching
    ProcessExecutionError() (an instance of a class) and that is not allowed
    in python3.

This changes the except statement to catch ProcessExecutionError
(class) instead.

Closes-Bug: #1984736

Change-Id: I24a269f9809063f864a4f0443d6724dfa8703d9d
(cherry picked from commit f996674c36fe1a109853d88113e27b44a7a5cf45)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-01: Fix proposed to nova (unmaintained/zed)

#14

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/nova/+/925483

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-09: Fix included in openstack/nova 27.5.0

#15

This issue was fixed in the openstack/nova 27.5.0 release.

Rodrigo Barbieri (rodrigo-barbieri2010) on 2025-01-27

description:	updated
summary:	- "TypeError: catching classes that do not inherit from BaseException is - not allowed" is raised if volume mount fails in python3 + [SRU]"TypeError: catching classes that do not inherit from BaseException + is not allowed" is raised if volume mount fails in python3
Changed in nova (Ubuntu):
status:	New → Invalid
Changed in cloud-archive:
status:	New → Invalid

Revision history for this message

Rodrigo Barbieri (rodrigo-barbieri2010) wrote on 2025-01-27:

#16

lp1984736_jammy.debdiff Edit (4.3 KiB, text/plain)

Rodrigo Barbieri (rodrigo-barbieri2010) on 2025-01-29

description:

updated

Heitor Alves de Siqueira (halves) on 2025-02-25

Changed in nova (Ubuntu):
status:	Invalid → Fix Released
Changed in nova (Ubuntu Jammy):
assignee:	nobody → Rodrigo Barbieri (rodrigo-barbieri2010)
importance:	Undecided → Medium
status:	New → In Progress

Heitor Alves de Siqueira (halves) on 2025-02-25

Changed in cloud-archive:
status:	Invalid → Fix Released

Revision history for this message

Heitor Alves de Siqueira (halves) wrote on 2025-04-02:

#17

Hi Rodrigo,

thanks for the Jammy patch! I'd like to get a bit more clarification on a few things:

- on your SRU template, please rework the regression section. It's usually assumed that proposed patches go through testing and that we do whatever possible to minimize regressions, but this should at least detail potential fallouts or changes in behavior caused by your patch. Feel free to refer to the updated template, where this section is called "Where Problems Could Occur" [0]

- for you patch itself, I've noticed that you have the "Origin" field tagged with "vendor". Is this an upstream patch, that went through upstream Openstack/Gerrit? If so, we should have a commit url here, tagged "upstream" or "backport" (depending on whether it's a clean cherry-pick or not)

Once we have those sorted out, I'd be happy to sponsor this for Jammy. Thanks!

[0] https://documentation.ubuntu.com/sru/en/latest/reference/bug-template/

Changed in nova (Ubuntu Jammy):
status:	In Progress → Incomplete

Revision history for this message

Rodrigo Barbieri (rodrigo-barbieri2010) wrote on 2025-04-10:

#18

lp1984736_jammy_v2.debdiff Edit (4.3 KiB, text/plain)

description:

updated

Revision history for this message

Rodrigo Barbieri (rodrigo-barbieri2010) wrote on 2025-04-10:

#19

Hi Heitor, thanks for the feedback! I apologize for the combo of mess. I got a bit carried away with change being so trivial that I didn't put much thought into the regression section. I have updated it as well. Regarding the DEP3 tag, it was another mistake where I had initially coded the fix myself (so here it would make sense to be marked as vendor) since it was so tiny, but then I changed my mind and imported the fix from upstream instead, and forgot to update that field. It should be good now. Thanks a lot for catching those 2 mistakes.

Rodrigo Barbieri (rodrigo-barbieri2010) on 2025-04-24

Changed in nova (Ubuntu Jammy):
status:	Incomplete → In Progress

Revision history for this message

Heitor Alves de Siqueira (halves) wrote on 2025-04-30:

#20

Thank you, Rodrigo! This looks great now, I'll pull it in for the next nova release.

We're currently blocked by the version in proposed due to bug 1972028, once that clears we can proceed with a new release.

Revision history for this message

Edward Hope-Morley (hopem) wrote on 2025-06-16:

#21

@halves 1972028 has now been release so this should be good to go

Revision history for this message

Heitor Alves de Siqueira (halves) wrote on 2025-06-24:

#22

Thanks, all! Sponsored for Jammy

Edward Hope-Morley (hopem) on 2025-07-28

Changed in nova (Ubuntu Jammy):
assignee:	Rodrigo Barbieri (rodrigo-barbieri2010) → Erlon R. Cruz (sombrafam)

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2025-08-21: Please test proposed package

#23

Hello melanie, or anyone else affected,

Accepted nova into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/3:25.2.1-0ubuntu2.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.