boot device API blocks while waiting on the BMC

@Deva, Are you working on this bug? otherwise I could get started on it.

Let me but in my thoughts about. I assume that the problem is not in API taking long to respond. There are two actualy problems:
1. API takes *undefined* amount of time to respond in case of BMC lock up
2. We block our messaging layer while we're waiting for the call to finish by using a sync AMQP call

So I don't think that a solution is just to make API asynchronous. It's a breaking change and one more step in making our errors hard to track. We already to this with power state, but at least power state in an inherently long procedure. Setting boot device should not - unless it breaks or hangs. If you just make API sync, you'll break inspector. Let me show. Inspector currently does roughly this:

 set_boot_device(uuid, 'pxe')
 set_power_state(uuid, 'reboot')

If we allow the former to go async and silently fail, we need to make the latter to fail as well. Otherwise inspector will report success, but actually node won't even try inspection (e.g. if it was set to local boot).

So what about making this call async on the conductor level, but sync on API level? I.e. API waiting for some kind of notification from conductor for some (probably settable timeout)?

Hi Deva! Are you working on this bug? Please make sure to update the bug status accordingly. Thanks!

Hi, for the time-consuming and may failed operations, could we cache its operations and use a configuration wait-time to ensure the API have the maximum response time.?

for the operations like get/set_boot_device,

for get_boot_device:
how about we store the current boot device in node's driver field? every time when we call set_boot_device, we update this value in cache?

For set_boot_device:

whether we can set a maximum wait time, if we cannot get success response from hardware in the given time, we return 202 to notify the user?

We discussed this in our weekly ironic meeting, on 2016-09-17 [1]. We agree that async is the way to go, but Dmitry had suggested async at the conductor level, but sync at the API level. I was going to start a discussion in the ML about this, but decided the best/fastest way was to discuss with Dmitry [2], and he is good with async. The CLI can have a --wait option. And of course, a bump in microversion.

[1] starting at 17:40:17, http://eavesdrop.openstack.org/meetings/ironic/2016/ironic.2016-09-19-17.00.log.html
[2] http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2016-10-12.log.html#t2016-10-12T14:54:04

wrt questions in #4.

get_boot_device: we cannot/should not store the info in the node's driver_info field; that field is available for the user to specify driver-related information. Without thinking too hard about this, I think we should either add a new field, or store it in driver_internal_info (my preference).

for set_boot_device: I'd suggest taking a look at the other async calls we have, and see what config options we've added for handling them. We should do a similar thing here (if there seems to be some consistent way we're handling this).

Change abandoned by Sam Betts (<email address hidden>) on branch: master
Review: https://review.openstack.org/161046
Reason: Review >1.5 years old and is in merge conflict, so we are abandoning this for now. Feel free to reactivate the review by pressing the restore button if you want to rebase and continue this work.