tempest test TestNodeStatesV1_4:test_set_node_power_state fails

Bug #1669816 reported by Pavlo Shchelokovskyy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Triaged
Medium
Unassigned

Bug Description

Is probably very sporadic (I've seen it once), but better to document it.
It looks like a race in tests.

Test goes and fails like this (node driver is fake):

1. get node -> power is on
2. reboot node
3. get node -> power is on
4. power-off node -> fail: Node ... is locked by host ..., please retry after the current operation is completed

http://logs.openstack.org/15/440715/1/check/gate-tempest-dsvm-ironic-ipa-partition-bios-pxe_ipmitool-tinyipa-ubuntu-xenial/e688a5b/console.html#_2017-03-02_18_10_52_255781

There is an error in ir-cond.log - "Driver returns ERROR power state for node"

http://logs.openstack.org/15/440715/1/check/gate-tempest-dsvm-ironic-ipa-partition-bios-pxe_ipmitool-tinyipa-ubuntu-xenial/e688a5b/logs/screen-ir-cond.txt.gz#_2017-03-02_18_04_24_699

Most probably during step 3 node was still 'rebooting' (not yet reached final power_on state), and so was locked.

To avoid this, the HTTP client in tempest should actually retry on 409 as our python client does.

Revision history for this message
Vladyslav Drok (vdrok) wrote :

Yeah, your solution seems like the way to go, with the only exception is that I think the message about the ERROR state actually means that the node is in NOSTATE power state, it is printed in that case too.

Changed in ironic:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.