Insufficient mocking in TestGenericHardwareManager

Bug #2037690 reported by Jay Faulkner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ironic-python-agent
Fix Released
Medium
Boushra Sondos Bettir

Bug Description

When running unit tests, the unit tests are relying on local machine state to pass during the GenericHardwareManager._is_read_only_device() check, as well as likely others including _is_virtual_media_device.

IMO, these methods should be mocked to return true, similarly to how _is_linux_raid_member is here:
https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/tests/unit/test_hardware.py#L1710

Output from an example failing test on a machine with a read-only /dev/sda:

ironic_python_agent.tests.unit.test_hardware.TestGenericHardwareManager.test_erase_block_device_ata_success_no_smartctl
-----------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)

      File "/home/jay/ironic-python-agent/ironic_python_agent/tests/unit/test_hardware.py", line 1730, in test_erase_block_device_ata_success_no_smartctl
    self.hardware.erase_block_device(self.node, block_device)

      File "/home/jay/ironic-python-agent/ironic_python_agent/hardware.py", line 1654, in erase_block_device
    raise errors.BlockDeviceEraseError(msg)

    ironic_python_agent.errors.BlockDeviceEraseError: Error erasing block device: Failed to invoke erase of device /dev/sda as the device is flagged read-only, and the conductor has not signaled this is a permitted case.

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
       ERROR [root] Failed to invoke erase of device /dev/sda as the device is flagged read-only, and the conductor has not signaled this is a permitted case.

This issue was found by a new contributor, Boushra, attempting to run unit tests on WSL. It is trivially reproducible in an Ubuntu Jammy WSL terminal.

Changed in ironic-python-agent:
status: New → Triaged
importance: Undecided → Medium
tags: added: low-hanging-fruit
Revision history for this message
Boushra Sondos Bettir (boushrabettir04) wrote :

Noticing some more errors that may definitely be a WSL issue:

```
ironic_python_agent.tests.unit.test_hardware.TestGenericHardwareManager.test_erase_devices_metadata
---------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/ironic_python_agent/tests/unit/test_hardware.py", line 2422, in test_erase_devices_metadata
    self.assertEqual([mock.call('/dev/sda1', self.node['uuid']),

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/.tox/py3/lib/python3.10/site-packages/testtools/testcase.py", line 394, in assertEqual
    self.assertThat(observed, matcher, message)

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/.tox/py3/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: !=:
reference = [call('/dev/sda1', 'dda135fb-732d-4742-8e72-df8f3199d244'),
 call('/dev/sda', 'dda135fb-732d-4742-8e72-df8f3199d244'),
 call('/dev/md0', 'dda135fb-732d-4742-8e72-df8f3199d244')]
actual = [call('/dev/md0', 'dda135fb-732d-4742-8e72-df8f3199d244')]

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
     WARNING [root] Could not determine if /dev/md0 is aread-only device. Error: [Errno 2] No such file or directory: '/sys/block/md/ro'
```

There are about 20 that are similar to this. Since this is all under the hood of the same core issue of WSL, I'll take a look at it.

Revision history for this message
Jay Faulkner (jason-oldos) wrote (last edit ):

I believe many of those will go away if you mock out those methods; in both cases, they're triggering the code paths in:

GenericHardwareManager._is_read_only_device
GenericHardwareManager._is_virtual_media_device

Which fail because they are not mocked. If you add mocking, similar to that used to mock GenericHardwareManager._is_linux_raid_member here: https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/tests/unit/test_hardware.py#L1710 you should solve them.

I expect you to have to add this mock to several tests (it's likely all 20 failures are related to this; but I'm not certain yet). The specific things I look for are: BlockDeviceEraseError exceptions in the traceback (none on the one you posted), or a message about being unable to read a file in /sys/ (this is present in your test failure) -- it's a red flag because at no point should IPA unit tests be reading *anything* on your actual disk.

Revision history for this message
Boushra Sondos Bettir (boushrabettir04) wrote :

I see, thank you. Currently, I am incorporating the necessary mocks into the specific test cases highlighted in the tracebacks.

Revision history for this message
Boushra Sondos Bettir (boushrabettir04) wrote :

I've gone ahead and marked + added mocks to the necessary tests, but it seems that there are unrelated errors as well, specifically two stand out to me the most. I will also go ahead and do my own research + debugging.

```
ironic_python_agent.tests.unit.test_hardware.TestGenericHardwareManager.test_erase_block_device_ata_frozen
----------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/ironic_python_agent/tests/unit/test_hardware.py", line 2166, in test_erase_block_device_ata_frozen
    self.assertRaises(

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/.tox/py3/lib/python3.10/site-packages/testtools/testcase.py", line 468, in assertRaises
    self.assertThat(our_callable, matcher)

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/.tox/py3/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: <bound method GenericHardwareManager.erase_block_device of <ironic_python_agent.hardware.GenericHardwareManager object at 0x7ffa76185fc0>> returned None

```

As well as,
```
ironic_python_agent.tests.unit.test_hardware.TestGenericHardwareManager.test_erase_devices_metadata_error
---------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/lib/python3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/ironic_python_agent/tests/unit/test_hardware.py", line 2533, in test_erase_devices_metadata_error
    self.assertEqual([mock.call('/dev/sdb', self.node['uuid']),

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/.tox/py3/lib/python3.10/site-packages/testtools/testcase.py", line 394, in assertEqual
    self.assertThat(observed, matcher, message)

      File "/home/bbgoddess/mlh-gresearch/ironic-python-agent/.tox/py3/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: !=:
reference = [call('/dev/sdb', 'dda135fb-732d-4742-8e72-df8f3199d244'),
 call('/dev/sda', 'dda135fb-732d-4742-8e72-df8f3199d244')]
actual = [call('/dev/sdb', 'dda135fb-732d-4742-8e72-df8f3199d244')]

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
       ERROR [root] Failed to erase the metadata on device "/dev/sdb". Error: Unexpected error while running command.
Command: None
Exit code: -
Stdout: 'Booo00000ooommmmm'
Stderr: None
```

Revision history for this message
Jay Faulkner (jason-oldos) wrote :

Let's get your attempted fix into gerrit so I can see and comment on the actual code; we'll figure it out :)

Revision history for this message
Jay Faulkner (jason-oldos) wrote :

Unsure why https://review.opendev.org/c/openstack/ironic-python-agent/+/897611 isn't updating here; I believe the syntax in the commit message is correct. Either way, fix proposed there.

Changed in ironic-python-agent:
assignee: nobody → Boushra Sondos Bettir (boushrabettir04)
Revision history for this message
Boushra Sondos Bettir (boushrabettir04) wrote :

Awesome, are we ready to close this bug and mark it as Fix-Committed?

Revision history for this message
Jay Faulkner (jason-oldos) wrote :

It can't be closed and marked as fixed until after the change merges; I have a +2 on the change; you need an additional review before it can land.

Changed in ironic-python-agent:
status: Triaged → Fix Committed
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.