Ironic

partprobe sometime fails with Device or resource busy

Bug #1626540 reported by Martin André on 2016-09-22

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ironic	Incomplete	Undecided	octopuszhang

Bug Description

When deploying a whole disk image, I've been facing issue with partprobe failing due to partition being is use:

2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor [req-6ae28454-4dca-4a21-8c2d-843ac6b28bab - - - - -] Asynchronous exception for node 14510c8a-3bc6-4a07-9f5a-b4fd20461fe2: Node failed to get image for deploy. Exception: Deploy failed for instance c97944f7-b688-45cc-8807-c6708f4a3c37. Error: Failed to retrieve partition labels on disk /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1 for node 14510c8a-3bc6-4a07-9f5a-b4fd20461fe2. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf partprobe /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1
Exit code: 1
Stdout: u''
Stderr: u"Error: Error informing the kernel about modifications to partition /dev/sda1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/sda1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.\nError: Failed to add partition 1 (Device or resource busy)\n"
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 482, in heartbeat
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor self.continue_deploy(task)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 61, in wrapped
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 138, in wrapper
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor return f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 381, in continue_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 61, in wrapped
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 313, in do_agent_iscsi_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 61, in wrapped
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 250, in continue_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor _fail_deploy(task, msg)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 227, in _fail_deploy
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor raise exception.InstanceDeployFailure(msg)
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Deploy failed for instance c97944f7-b688-45cc-8807-c6708f4a3c37. Error: Failed to retrieve partition labels on disk /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1 for node 14510c8a-3bc6-4a07-9f5a-b4fd20461fe2. Error: Unexpected error while running command.
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf partprobe /dev/disk/by-path/ip-192.0.2.12:3260-iscsi-iqn.2008-10.org.openstack:14510c8a-3bc6-4a07-9f5a-b4fd20461fe2-lun-1
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Exit code: 1
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Stdout: u''
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor Stderr: u"Error: Error informing the kernel about modifications to partition /dev/sda1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/sda1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.\nError: Failed to add partition 1 (Device or resource busy)\n"
2016-09-22 09:00:34.977 1302 ERROR ironic.drivers.modules.agent_base_vendor

Retrying the command when if fails by specifying an 'attempts=10' argument in https://github.com/openstack/ironic-lib/blob/c724cbf368398277071f5b2c735eca32cca22646/ironic_lib/disk_utils.py#L588 may help.