OpenStack Heat

Heat appears to fail to retry when being API rate limited

Bug #1547728 reported by Byron McCollum
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Triaged
Medium
Unassigned
OpenStack Heat next

Bug Description

I *think* Heat is failing to retry when being API rate limited. On occasion, when creating several stacks concurrently, I will get a resources that goes into `CREATE_FAILED`, and the reason being `OverLimit: resources.server_block1: OverLimit Retry... (HTTP 413)`.

I do not believe this is related to absolute resources (instance, cores, ram, etc), and I have plenty available for the stacks I am attempting to build.

This is an example resource in question from `heat event-list <stack-name>`:

| server_block1 | 4faa9799-1047-40cf-a3bf-1bf3390bc26a | OverLimit: resources.server_block1: OverLimit Retry... (HTTP 413) | CREATE_FAILED | 2016-02-19T23:21:40 |

Revision history for this message
zhaozhilong (zhaozhilong) wrote :

How many stacks you try to create once time ?

Revision history for this message
Byron McCollum (byron-mccollum) wrote :

@zhaozhilong

It happens when I have 9-10 building...

Revision history for this message
Byron McCollum (byron-mccollum) wrote :

And I'm polling their status every 120 seconds...

Revision history for this message
Zane Bitter (zaneb) wrote :

You're correct, in fact Heat only retries when an asynchronously-created resource (e.g. a Server or Volume) ends up in an error state instead of completing.

I agree that there are other circumstances where we ought to retry (c.f. bug 1745010), and this is one of them.

Changed in heat:
status: New → Triaged
importance: Undecided → Medium
milestone: none → next
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.