RPC receivers may starve periodic tasks
Bug #1308680 reported by
Jim Rollenhagen
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ironic |
Triaged
|
Medium
|
Unassigned |
Bug Description
When issuing many RPC tasks to the conductor, the heartbeat task (and probably other periodic tasks) runs much slower than intended.
In my testing, I am powering off many nodes in a loop:
for node in nodes:
My conductor is set to heartbeat every 10 seconds, and does so until I run this loop. While running this loop, the heartbeat slows down to 3-5 *minutes*.
Theory is that the RPC receiver takes priority and does not yield to the periodic tasks.
Changed in ironic: | |
status: | New → Triaged |
importance: | Undecided → Critical |
Changed in ironic: | |
assignee: | nobody → Jim Rollenhagen (jim-rollenhagen) |
Changed in ironic: | |
status: | In Progress → Triaged |
Changed in ironic: | |
importance: | High → Low |
milestone: | none → kilo-rc1 |
importance: | Low → Medium |
To post a comment you must log in.
Reproduced locally with the "fake" driver. See /review. openstack. org/#/c/ 88076/
https:/
------- ------- ------
This leverages the blocking nature of the mysql connection
to starve the RPC worker pool.
Steps to reproduce in a short amount of time: interval = 5 pool_size = 5 pool_size = 5
- set the following in your config file:
heartbeat_
heartbeat_timeout = 10
rpc_thread_
rpc_conn_
- register 6 nodes with the fake driver
- request 5 nodes to be power cycled
- watch output of 'ironic driver-list'.. after about 6 seconds, it will
be empty
- request the 6th node to be power cycled
- observe error