OpenStack Nova Compute Charm

nova-compute accepts non-existing path for ephemeral-device

Bug #1870232 reported by Przemyslaw Hausman
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Triaged
Medium
Unassigned

Bug Description

bionic-stein, 20.02 charms release.

nova-compute accepts non-existing path for ephemeral-device, for example '/dev/disk/by-dname/i-dont-exist'. Charm successfully deploys. VMs can be successfully launched.

However, ephemeral disks for VMs end up in /var/lib/nova/instances, instead of on the device '/dev/disk/by-dname/i-dont-exist' (as expected). Nova compute charm is silently ignoring incorrect setting for ephemeral-device. This may sound like a sane fallback.

But in case of e.g. typo in configuration of ephemeral-device, this results is an issue difficult to spot, because VMs run just fine and the charm does not report any errors.

I would suggest to set an error status on the charm with the message indicating that the config value for ephemeral-device does not match current configuration on the host. Or at least a warning in the charm message in juju status output. So that it is easy to spot and correct early.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

The charm should set the status with a warning if the ephemeral device is not available

Changed in charm-nova-compute:
status: New → Triaged
importance: Undecided → Medium
milestone: none → 20.05
David Ames (thedac)
Changed in charm-nova-compute:
milestone: 20.05 → 20.08
James Page (james-page)
Changed in charm-nova-compute:
milestone: 20.08 → none
Revision history for this message
Vladimir Grevtsev (vlgrevtsev) wrote :

> The charm should set the status with a warning if the ephemeral device is not available

Indeed. We just hit this during the BS handover - one node was having incorrectly configured storage (sda/sdb devices were swapped, while charm had ephemeral-device config option set to /dev/sdb).

Could we please revisit this issue again to avoid similar problems in the future?

Pedro Guimarães (pguimaraes)
Changed in charm-nova-compute:
assignee: nobody → Pedro Guimarães (pguimaraes)
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

Just my 0,02: if we are asking for a specific path and it cannot be mounted, it should go into error state and require manual intervention. Even a warning in the logs may be not enough (unless the warning stays on juju status to be easily spotted.

Revision history for this message
Vladimir Grevtsev (vlgrevtsev) wrote :

For the record, we have just hit this again in another deployment.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi Pedro. Are you still working on this? If not, please could you un-assign yourself. Many thanks, Alex.

Pedro Guimarães (pguimaraes)
Changed in charm-nova-compute:
assignee: Pedro Guimarães (pguimaraes) → nobody
Alex Kavanagh (ajkavanagh)
Changed in charm-nova-compute:
status: In Progress → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-nova-compute (master)

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/755780
Reason: This review is stale as it has not had an updates since the review comments.

Revision history for this message
Kamal Bhaskar (kamalb) wrote (last edit ):

Hello

We are facing the same issue in a customer environment (jammy/yoga):

nova-compute-control 25.2.0 active 6 nova-compute yoga/stable 686 no Unit is ready

$ juju config nova-compute-control ephemeral-device
/dev/disk/by-dname/vg0-lvephemeral
$

And the actual/correct path is:

# ls -lrt /dev/disk/by-dname/vg0-lvephemeral
ls: cannot access '/dev/disk/by-dname/vg0-lvephemeral': No such file or directory
# ls -lrt /dev/disk/by-dname/vg1-lvephemeral
lrwxrwxrwx 1 root root 10 Dec 11 06:36 /dev/disk/by-dname/vg1-lvephemeral -> ../../dm-4
#

But, the charm deployed successfully and is in active/idle and ready state; while in the backend it failed to mount the above ephemeral-device path and fallen back to use root(/) partition with default images stored in /var/lib/nova/instances.

Please let me know if any other details are needed in this regard.

Thanks & regards

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.