OpenStack Nova Compute Charm

Missing dependency: lsscsi

Bug #1939390 reported by Gabriel Samfira on 2021-08-10

This bug affects 4 people

	Status	Importance	Assigned to	Milestone
OpenStack Nova Compute Charm	Fix Released	Undecided	Unassigned	OpenStack Nova Compute Charm 22.04
Ubuntu Cloud Archive	Fix Released	High	Unassigned
os-brick	Invalid	Low	Unassigned
python-os-brick (Ubuntu)	Fix Released	High	Unassigned

Bug Description

When resizing a volume, cinder will notify nova that a volume has changed its size. For iSCSI attached volumes (ie: Dell Unity), the package lsscsi is required for that to work. Without it, you get this error:

2021-08-10 09:44:02.400 1816368 INFO os_brick.initiator.connectors.iscsi [req-98b38968-a179-4a3c-bf09-dfe5dcee9b66 aa291ad6c5b4430d9a5e268850c5aed8 ff58079dba31444da859ff6f91a2f983 - 83859026773d453ca315c30285f94efa 83859026773d453ca315c30285f94efa] Found paths for volume ['/dev/disk/by-path/ip-10.123.110.4:3260-iscsi-iqn.1992-04.com.emc:cx.ckm01211605212.a4-lun-86', '/dev/disk/by-path/ip-10.123.120.4:3260-iscsi-iqn.1992-04.com.emc:cx.ckm01211605212.b4-lun-86', '/dev/disk/by-path/ip-10.123.120.5:3260-iscsi-iqn.1992-04.com.emc:cx.ckm01211605212.b5-lun-86', '/dev/disk/by-path/ip-10.123.110.5:3260-iscsi-iqn.1992-04.com.emc:cx.ckm01211605212.a5-lun-86']
2021-08-10 09:44:02.424 1816368 WARNING nova.compute.manager [req-98b38968-a179-4a3c-bf09-dfe5dcee9b66 aa291ad6c5b4430d9a5e268850c5aed8 ff58079dba31444da859ff6f91a2f983 - 83859026773d453ca315c30285f94efa 83859026773d453ca315c30285f94efa] [instance: 6cf7a8c6-255f-4b69-812a-39dc0fa279b6] Extend volume failed, volume_id=ac43884b-916c-44b9-be46-393325ad64cc, reason: [Errno 2] No such file or directory: 'lsscsi'
Command: lsscsi
Exit code: -
Stdout: None
Stderr: None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'lsscsi'
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server [req-98b38968-a179-4a3c-bf09-dfe5dcee9b66 aa291ad6c5b4430d9a5e268850c5aed8 ff58079dba31444da859ff6f91a2f983 - 83859026773d453ca315c30285f94efa 83859026773d453ca315c30285f94efa] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'lsscsi'
Command: lsscsi
Exit code: -
Stdout: None
Stderr: None
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 171, in execute
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return custom_execute(*cmd, **kwargs)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 142, in custom_execute
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return putils.execute(on_execute=on_execute,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py", line 388, in execute
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server obj = subprocess.Popen(cmd,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/green/subprocess.py", line 58, in __init__
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server self._execute_child(args, executable, preexec_fn, close_fds,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server raise child_exception_type(errno_num, err_msg, err_filename)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server FileNotFoundError: [Errno 2] No such file or directory: 'lsscsi'
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server raise value
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 10405, in external_instance_event
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1434, in decorated_function
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server raise value
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 205, in decorated_function
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 10262, in extend_volume
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server self.driver.extend_volume(context, connection_info, instance,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 2199, in extend_volume
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server new_size = self._extend_volume(connection_info, instance,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1743, in _extend_volume
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return vol_driver.extend_volume(connection_info, instance,
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/iscsi.py", line 86, in extend_volume
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server new_size = self.connector.extend_volume(connection_info['data'])
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 144, in trace_logging_wrapper
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/iscsi.py", line 477, in extend_volume
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server return self._linuxscsi.extend_volume(
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py", line 581, in extend_volume
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server device = self.get_device_info(volume_path)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py", line 114, in get_device_info
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server (out, _err) = self._execute('lsscsi')
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/executor.py", line 52, in _execute
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 186, in execute
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server raise putils.ProcessExecutionError(
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory: 'lsscsi'
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server Command: lsscsi
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server Exit code: -
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server Stdout: None
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server Stderr: None
2021-08-10 09:44:02.497 1816368 ERROR oslo_messaging.rpc.server

Running:

sudo apt install lsscsi

on the compute node fixes this issue.

Steps to reproduce:

openstack volume create --type my-iscsi-volume-type --size 10 test-resize
openstack server add volume test-server test-resize
openstack volume set --size 20 test-resize

Check the logs on the compute node where the instance is running. There should be an entry similar to the above snippet.

Check volume size on instance using parted or fdisk. It should be the same size as before.

See original description

Tags:

Revision history for this message

Alex Kavanagh (ajkavanagh) wrote on 2021-08-10:

This seems like is might be an optional feature. I'm not sure whether this should just be a default install, or a config option to bring it in as needed?

Changed in charm-nova-compute:
status:	New → Incomplete
status:	Incomplete → Triaged

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2021-08-10:

The dependency is quite small, and the nova-compute-libvirt package already pulls in open-iscsi. This seems like it could simply be added as a dependency during charm install, or as a dependency for nova-compute-libvirt.

In my case, the issue was fixed by running:

juju run --application nova-compute -- sudo apt-get -y install lsscsi

But this is not ideal.

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-02-21:

Adding Ubuntu packaging task. It seems lsscsi dependency has been added fairly recently (July 2020) so it looks like it's something os-brick binary package should install as a dependency:
https://bugs.launchpad.net/os-brick/+bug/1793259
https://opendev.org/openstack/os-brick/commit/fc6ca22bdb955137d97cb9bcfc84104426e53842

At this point, there is no Depends or Recommends from os-brick or OpenStack related packages.

$ apt rdepends lsscsi
lsscsi
Reverse Depends:
  Depends: libguestfs0
  Depends: libguestfs0
  Depends: zfs-test
  Depends: x2gothinclient-cdmanager
  Suggests: stressant-meta
  Recommends: sitesummary-client

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-02-21:

Similar with this one: https://bugs.launchpad.net/ubuntu/+source/python-os-brick/+bug/1947063

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-02-21: Fix proposed to charm-nova-compute (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/830171

Changed in charm-nova-compute:
status:	Triaged → In Progress

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-02-21:

Subscribing ~field-medium

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-02-21:

I *think* we also had this problem on systems that had NVMe volumes. The nvme-cli package is not pulled in, even though it is used by os-brick:

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/nvmeof.py#L155

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-02-21:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-os-brick (Ubuntu):
status:	New → Confirmed

Gabriel Samfira (gabriel-samfira) on 2022-02-21

description:

updated

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-02-21:

> I *think* we also had this problem on systems that had NVMe volumes. The nvme-cli package is not pulled in, even though it is used by os-brick:

Did it block any operation by missing the nvme command? It looks like it's in a critical path for NVMe-oF usecase, but it generates a warning instead of an error when it's not found.

Gabriel Samfira (gabriel-samfira) on 2022-02-21

description:

updated

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-02-22:

#10

> Did it block any operation by missing the nvme command? It looks like it's in a critical path for NVMe-oF usecase, but it generates a warning instead of an error when it's not found.

No. It was just a warning when using a Dell Unity all flash storage appliance. However, it does seem like it would be a good idea to add that package as well.

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-02-22:

#11

Okay, I've added a comment there:
https://bugs.launchpad.net/ubuntu/+source/python-os-brick/+bug/1947063

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-03-11: Fix merged to charm-nova-compute (master)

#12

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/830171
Committed: https://opendev.org/openstack/charm-nova-compute/commit/7a3982d987b5524c883b33ac5d0432ea1c1b3cdc
Submitter: "Zuul (22348)"
Branch: master

commit 7a3982d987b5524c883b33ac5d0432ea1c1b3cdc
Author: Nobuto Murata <email address hidden>
Date: Mon Feb 21 22:49:01 2022 +0900

Make sure lsscsi is installed for os-brick

    os-brick upstream made some changes and started to use lsscsi for
    discovering iSCSI devices. Make sure lsscsi is installed by the charm
    until the os-brick package in Ubuntu pulls it as a dependency.

Closes-Bug: #1939390
Change-Id: I509d5edb3c6e3a4c35b7252b567a1672cf5e0bbe

Changed in charm-nova-compute:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-05: Fix proposed to charm-nova-compute (stable/xena)

#13

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/836418

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-08: Fix merged to charm-nova-compute (stable/xena)

#14

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/836418
Committed: https://opendev.org/openstack/charm-nova-compute/commit/dd67c2c04e97d9e66ff3589a61fa90470ba8df6d
Submitter: "Zuul (22348)"
Branch: stable/xena

commit dd67c2c04e97d9e66ff3589a61fa90470ba8df6d
Author: Nobuto Murata <email address hidden>
Date: Mon Feb 21 22:49:01 2022 +0900

Make sure lsscsi is installed for os-brick

    Closes-Bug: #1939390
    Change-Id: I509d5edb3c6e3a4c35b7252b567a1672cf5e0bbe
    (cherry picked from commit 7a3982d987b5524c883b33ac5d0432ea1c1b3cdc)

tags:

added: in-stable-xena

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-04-12 (last edit on 2022-04-12):

#15

To accommodate the upstream change, we need backporting down to Victoria.

os-brick (master=)$ git branch -r --contains fc6ca22bdb955137d97cb9bcfc84104426e53842
  origin/HEAD -> origin/master
  origin/master
  origin/stable/victoria
  origin/stable/wallaby
  origin/stable/xena
  origin/stable/yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-12: Fix proposed to charm-nova-compute (stable/wallaby)

#16

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/837325

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-12: Fix proposed to charm-nova-compute (stable/victoria)

#17

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/837326

Alex Kavanagh (ajkavanagh) on 2022-04-14

Changed in charm-nova-compute:
milestone:	none → 22.04

Alex Kavanagh (ajkavanagh) on 2022-05-10

Changed in charm-nova-compute:
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-10: Fix merged to charm-nova-compute (stable/wallaby)

#18

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/837325
Committed: https://opendev.org/openstack/charm-nova-compute/commit/87f9ddc9b8119c88d524d58a4a2e65c3ec7decc2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 87f9ddc9b8119c88d524d58a4a2e65c3ec7decc2
Author: Nobuto Murata <email address hidden>
Date: Mon Feb 21 22:49:01 2022 +0900

Make sure lsscsi is installed for os-brick

    Closes-Bug: #1939390
    Change-Id: I509d5edb3c6e3a4c35b7252b567a1672cf5e0bbe
    (cherry picked from commit 7a3982d987b5524c883b33ac5d0432ea1c1b3cdc)
    (cherry picked from commit dd67c2c04e97d9e66ff3589a61fa90470ba8df6d)

tags:

added: in-stable-wallaby

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-06-25:

#19

For the record, lsscsi required additional apparmor policies, and here is the actual output.

[nova-compute.log]

2022-06-24 14:38:41.910 171033 ERROR oslo_messaging.rpc.server [req-2604a2be-8fb0-438c-9c3d-d74106458755 b2062b97055a415eb23484562f0d2fcf 4441eef28bf5454aa7c9cbdd101b9da3 - 66035521156a4f578f43d215ceaf0e7b 66035521156a4f578f43d215ceaf0e7b] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: tee -a /sys/bus/scsi/drivers/sd/None:None:None:None/rescan
Exit code: 1
Stdout: '1'
Stderr: "tee: '/sys/bus/scsi/drivers/sd/None:None:None:None/rescan': No such file or directory\n"

[running lsscsi by hand]

$ sudo -H -u nova aa-exec -p /usr/bin/nova-compute -- lsscsi -v
list_sdevices: scandir: /sys/bus/scsi/devices: Permission denied
SCSI mid level module may not be loaded

Jun 25 02:46:14 alert-glider kernel: audit: type=1400 audit(1656125174.802:5399): apparmor="DENIED" operation="open" profile="/usr/bin/nova-compute" name="/sys/bus/scsi/devices/" pid=937955 comm="lsscsi" requested_mask="r" denied_mask="r" fsuid=64060 ouid=0

[after adding /sys/bus/scsi/devices/ r]

$ sudo -H -u nova aa-exec -p /usr/bin/nova-compute -- lsscsi
[2:0:0:0] disk LIO-ORG IBLOCK 4.0 -

-> the last column is empty.

Jun 25 02:49:56 alert-glider kernel: audit: type=1400 audit(1656125396.589:5406): apparmor="DENIED" operation="open" profile="/usr/bin/nova-compute" name="/dev/" pid=939829 comm="lsscsi" requested_mask="r" denied_mask="r" fsuid=64060 ouid=0

[after adding /dev/ r]

$ sudo -H -u nova aa-exec -p /usr/bin/nova-compute -- lsscsi
[2:0:0:0] disk LIO-ORG IBLOCK 4.0 /dev/sda

-> the iSCSI device is detected.

$ ll /dev/disk/by-path/ | grep sda$
lrwxrwxrwx 1 root root 9 Jun 24 14:31 ip-192.168.151.128:3260-iscsi-iqn.2010-10.org.openstack:volume-f345d1e7-1326-4f7f-a0b9-ed40fe81432a-lun-0 -> ../../sda

https://review.opendev.org/c/openstack/charm-nova-compute/+/847582

For the record, lsscsi required additional apparmor policies, and here is the actual output.

[nova-compute.log]

[running lsscsi by hand]

$ sudo -H -u nova aa-exec -p /usr/bin/nova-compute -- lsscsi -v
list_sdevices: scandir: /sys/bus/scsi/devices: Permission denied
SCSI mid level module may not be loaded

[after adding /sys/bus/scsi/devices/ r]

$ sudo -H -u nova aa-exec -p /usr/bin/nova-compute -- lsscsi
[2:0:0:0]    disk    LIO-ORG  IBLOCK           4.0   -

-> the last column is empty.

[after adding /dev/ r]

$ sudo -H -u nova aa-exec -p /usr/bin/nova-compute -- lsscsi
[2:0:0:0]    disk    LIO-ORG  IBLOCK           4.0   /dev/sda

-> the iSCSI device is detected.

$ ll /dev/disk/by-path/ | grep sda$
lrwxrwxrwx 1 root root   9 Jun 24 14:31 ip-192.168.151.128:3260-iscsi-iqn.2010-10.org.openstack:volume-f345d1e7-1326-4f7f-a0b9-ed40fe81432a-lun-0 -> ../../sda

https://review.opendev.org/c/openstack/charm-nova-compute/+/847582

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-28: Fix merged to charm-nova-compute (stable/victoria)

#20

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/837326
Committed: https://opendev.org/openstack/charm-nova-compute/commit/28f0b70bce3a53c6f01ff8a974465e4c0b5b6997
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 28f0b70bce3a53c6f01ff8a974465e4c0b5b6997
Author: Nobuto Murata <email address hidden>
Date: Mon Feb 21 22:49:01 2022 +0900

Make sure lsscsi is installed for os-brick

tags:

added: in-stable-victoria

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-29: Related fix merged to charm-nova-compute (master)

#21

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/847582
Committed: https://opendev.org/openstack/charm-nova-compute/commit/cf0f464391df509e752c6010964efe2aca10ef89
Submitter: "Zuul (22348)"
Branch: master

commit cf0f464391df509e752c6010964efe2aca10ef89
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 24 23:22:54 2022 +0900

AppArmor policy update for os-brick and iSCSI

    In iSCSI usecases including cinder-lvm, os-brick requires lock files
    such as:
      - /run/lock/nova/os-brick-connect_volume
      - /run/lock/nova/os-brick-connect_to_iscsi_portal-192.168.0.1

    and lsscsi requires following access to compose a rescan command such as
    "/sys/bus/scsi/drivers/sd/2:0:0:0/rescan":
      - /dev/
      - /sys/bus/scsi/devices/

    Closes-Bug: #1979812
    Related-Bug: #1939390
    Change-Id: Id2db3a70b8d1287bda006f1bbc5442038f7070f1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-29: Related fix proposed to charm-nova-compute (stable/yoga)

#22

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/848054

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-29: Related fix proposed to charm-nova-compute (stable/xena)

#23

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/848055

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-29: Related fix proposed to charm-nova-compute (stable/wallaby)

#24

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/848056

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-29: Related fix proposed to charm-nova-compute (stable/victoria)

#25

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/848057

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-07-06: Related fix merged to charm-nova-compute (stable/yoga)

#26

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/848054
Committed: https://opendev.org/openstack/charm-nova-compute/commit/825b4235f137982cdafb3e3d58601d1cb1081681
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 825b4235f137982cdafb3e3d58601d1cb1081681
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 24 23:22:54 2022 +0900

AppArmor policy update for os-brick and iSCSI

    In iSCSI usecases including cinder-lvm, os-brick requires lock files
    such as:
      - /run/lock/nova/os-brick-connect_volume
      - /run/lock/nova/os-brick-connect_to_iscsi_portal-192.168.0.1

    and lsscsi requires following access to compose a rescan command such as
    "/sys/bus/scsi/drivers/sd/2:0:0:0/rescan":
      - /dev/
      - /sys/bus/scsi/devices/

    Closes-Bug: #1979812
    Related-Bug: #1939390
    Change-Id: Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    (cherry picked from commit cf0f464391df509e752c6010964efe2aca10ef89)

tags:

added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-24: Related fix merged to charm-nova-compute (stable/wallaby)

#27

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/848056
Committed: https://opendev.org/openstack/charm-nova-compute/commit/7fb373d9fa5694f9d53aac261c2650224f8af99e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7fb373d9fa5694f9d53aac261c2650224f8af99e
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 24 23:22:54 2022 +0900

AppArmor policy update for os-brick and iSCSI

    In iSCSI usecases including cinder-lvm, os-brick requires lock files
    such as:
      - /run/lock/nova/os-brick-connect_volume
      - /run/lock/nova/os-brick-connect_to_iscsi_portal-192.168.0.1

    and lsscsi requires following access to compose a rescan command such as
    "/sys/bus/scsi/drivers/sd/2:0:0:0/rescan":
      - /dev/
      - /sys/bus/scsi/devices/

    Closes-Bug: #1979812
    Related-Bug: #1939390
    Change-Id: Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    (cherry picked from commit cf0f464391df509e752c6010964efe2aca10ef89)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-24: Related fix merged to charm-nova-compute (stable/xena)

#28

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/848055
Committed: https://opendev.org/openstack/charm-nova-compute/commit/c16a9aedab7eed0699f3d677eab03374d348e6a8
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c16a9aedab7eed0699f3d677eab03374d348e6a8
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 24 23:22:54 2022 +0900

AppArmor policy update for os-brick and iSCSI

    In iSCSI usecases including cinder-lvm, os-brick requires lock files
    such as:
      - /run/lock/nova/os-brick-connect_volume
      - /run/lock/nova/os-brick-connect_to_iscsi_portal-192.168.0.1

    and lsscsi requires following access to compose a rescan command such as
    "/sys/bus/scsi/drivers/sd/2:0:0:0/rescan":
      - /dev/
      - /sys/bus/scsi/devices/

    Closes-Bug: #1979812
    Related-Bug: #1939390
    Change-Id: Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    (cherry picked from commit cf0f464391df509e752c6010964efe2aca10ef89)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-25: Related fix merged to charm-nova-compute (stable/victoria)

#29

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/848057
Committed: https://opendev.org/openstack/charm-nova-compute/commit/b5e658d04571a7c3019b5c5b711aca632989248c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b5e658d04571a7c3019b5c5b711aca632989248c
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 24 23:22:54 2022 +0900

AppArmor policy update for os-brick and iSCSI

    In iSCSI usecases including cinder-lvm, os-brick requires lock files
    such as:
      - /run/lock/nova/os-brick-connect_volume
      - /run/lock/nova/os-brick-connect_to_iscsi_portal-192.168.0.1

    and lsscsi requires following access to compose a rescan command such as
    "/sys/bus/scsi/drivers/sd/2:0:0:0/rescan":
      - /dev/
      - /sys/bus/scsi/devices/

    Closes-Bug: #1979812
    Related-Bug: #1939390
    Change-Id: Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    (cherry picked from commit cf0f464391df509e752c6010964efe2aca10ef89)

Sofia Enriquez (lsofia-enriquez) on 2022-09-28

Changed in os-brick:
importance:	Undecided → Low

Revision history for this message

Brian Rosmaita (brian-rosmaita) wrote on 2022-09-28:

#30

As far as I can tell, this is a packaging issue. lsscsi has been present in the os-brick bindep.txt file since change I867c972d9f712c0df4260ebc8211b786006ed7a2 (July 2020) in order to fix bug #1793259, so I don't think there are any changes required to os-brick.

Changed in os-brick:
status:	New → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-17: Related fix proposed to charm-nova-compute (stable/ussuri)

#31

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/864907

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-01: Related fix merged to charm-nova-compute (stable/ussuri)

#32

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/864907
Committed: https://opendev.org/openstack/charm-nova-compute/commit/a847442a2c2de85a05752dc2de419d1ecadc1d4f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a847442a2c2de85a05752dc2de419d1ecadc1d4f
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 24 23:22:54 2022 +0900

AppArmor policy update for os-brick and iSCSI

    In iSCSI usecases including cinder-lvm, os-brick requires lock files
    such as:
      - /run/lock/nova/os-brick-connect_volume
      - /run/lock/nova/os-brick-connect_to_iscsi_portal-192.168.0.1

    and lsscsi requires following access to compose a rescan command such as
    "/sys/bus/scsi/drivers/sd/2:0:0:0/rescan":
      - /dev/
      - /sys/bus/scsi/devices/

    Closes-Bug: #1979812
    Related-Bug: #1939390
    Change-Id: Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    (cherry picked from commit b5e658d04571a7c3019b5c5b711aca632989248c)

tags:

added: in-stable-ussuri

Corey Bryant (corey.bryant) on 2023-01-23

Changed in python-os-brick (Ubuntu):
status:	Confirmed → Triaged
importance:	Undecided → High
Changed in cloud-archive:
status:	New → Triaged
importance:	Undecided → High

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2023-01-23:

#33

This is fixed in Lunar 6.1.0-0ubuntu2.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2023-01-24:

#34

This bug was fixed in the package python-os-brick - 6.1.0-0ubuntu2

---------------
python-os-brick (6.1.0-0ubuntu2) lunar; urgency=medium

  * d/control: Add lsscsi to Depends as required for iSCSI attached
    volumes (LP: #1939390).
  * d/control: Add nvme-cli to Depends as required for volumes using
    nvme-cli over RDMA (LP: #1947063).

-- Corey Bryant <email address hidden> Mon, 23 Jan 2023 09:01:34 -0500

Changed in python-os-brick (Ubuntu):
status:	Triaged → Fix Released

Corey Bryant (corey.bryant) on 2023-01-25

Changed in cloud-archive:
status:	Triaged → Fix Committed

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2023-01-26:

#35

This bug was fixed in the package python-os-brick - 6.1.0-0ubuntu2~cloud0
---------------

python-os-brick (6.1.0-0ubuntu2~cloud0) jammy-antelope; urgency=medium
.
   * New update for the Ubuntu Cloud Archive.
.
python-os-brick (6.1.0-0ubuntu2) lunar; urgency=medium
.
   * d/control: Add lsscsi to Depends as required for iSCSI attached
     volumes (LP: #1939390).
   * d/control: Add nvme-cli to Depends as required for volumes using
     nvme-cli over RDMA (LP: #1947063).

Changed in cloud-archive:
status:	Fix Committed → Fix Released

Revision history for this message

Christian Rohmann (christian-rohmann) wrote on 2023-01-29:

#36

As with bug 1947063 (https://bugs.launchpad.net/ubuntu/+source/python-os-brick/+bug/1947063/comments/9) I'd like to ask for a SRU/backport for Xena or Yoga releases via Ubuntu Cloud Archive to have this fixed for those releases.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.