Instance create retry fails

Bug #1749560 reported by Mohammed Naser
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Triaged
Medium
huangsm

Bug Description

This seems to be a combination of bugs that will happen in an up-to-date environment, the following occurs:

1. Heat creates a volume
2. Heat creates an instance and tells Nova to boot from that volume
3. Instance creation fails, however Nova already reserved the volume (so the volume is in attaching state)
4. Heat deletes the failed instance
5. Heat tries to recreate the instance, but the volume is in 'attaching' state (because Nova reserved the volume), so it throws this exception:

BadRequest: Invalid volume: Invalid input received: Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400)

6. Due to the checks here, Heat does not use the first failure as the stack failure, but the second one because it is not an instance of `exception.ResourceInError`.
 This is because of the following: https://github.com/openstack/heat/blob/stable/pike/heat/engine/resource.py#L1178-L1179

I think this has to be mitigated in Nova but also Heat should probably report the first failure if there are subsequent failures in the retry mechanism, even if "BadRequest" was the failure.

Revision history for this message
Mohammed Naser (mnaser) wrote :

The Nova clean-up fixes are here.

https://review.openstack.org/#/c/340614/

Revision history for this message
Rabi Mishra (rabi) wrote :

> I think this has to be mitigated in Nova but also Heat should probably report the first failure if there are subsequent failures in the retry mechanism, even if "BadRequest" was the failure.

Not sure if that would be the right thing to do. Heat would probably expect it to fail with the same error when retrying, though it would not be always true and we can possibly the mask the primary issue.

Changed in heat:
status: New → Triaged
importance: Undecided → Medium
huangsm (huangsm)
Changed in heat:
assignee: nobody → huangsm (huangsm)
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.