Race condition in test_servers_negative.py

Bug #1006586 reported by David Kranz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Critical
David Kranz

Bug Description

I am seeing the following errors. They seem to be caused by the code creating a server, deleting it, and then doing a reboot/rebuild expecting the server to be deleted. But the code does wait for the server to be deleted. I added a call to wait_for_server_termination and the problem went away, but I am not sure what the point of creating and deleting the server is in this case. I am not sure why this has not been reported. I observe this with the current master for tempest/devstack but running devstack on a different server than tempest.

======================================================================
ERROR: Reboot a deleted server
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cygdrive/c/source/tempest/tempest/tests/test_servers_negative.py", line 116, in test_reboot_deleted_server
    resp1, reboot_server = self.client.reboot(self.server_id, 'SOFT')
  File "/cygdrive/c/source/tempest/tempest/services/nova/json/servers_client.py", line 211, in reboot
    post_body, self.headers)
  File "/cygdrive/c/source/tempest/tempest/common/rest_client.py", line 152, in post
    return self.request('POST', url, headers, body)
  File "/cygdrive/c/source/tempest/tempest/common/rest_client.py", line 199, in request
    raise exceptions.Duplicate(resp_body)
Duplicate: An object with that identifier already exists
Details: An object with that identifier already exists
Details: {u'conflictingRequest': {u'message': u"Cannot 'reboot' while instance is in vm_state building", u'code': 409}}
-------------------- >> begin captured logging << --------------------
tempest.common.rest_client: ERROR: Request URL: http://172.18.0.156:8774/v2/fe4b597cb85140ada9ba0994012d4da9/servers/dfb02cb8-fd6c-4a73-b1fb-80f8b79c2b50/action
tempest.common.rest_client: ERROR: Request Body: {"reboot": {"type": "SOFT"}}
tempest.common.rest_client: ERROR: Response Headers: {'date': 'Wed, 30 May 2012 19:43:47 GMT', 'status': '409', 'content-length': '106', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-2a3fafce-93f7-452d-a824-162eaedb315f'}
tempest.common.rest_client: ERROR: Response Body: {u'conflictingRequest': {u'message': u"Cannot 'reboot' while instance is in vm_state building", u'code': 409}}
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Rebuild a deleted server
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cygdrive/c/source/tempest/tempest/tests/test_servers_negative.py", line 133, in test_rebuild_deleted_server
    self.image_ref_alt)
  File "/cygdrive/c/source/tempest/tempest/services/nova/json/servers_client.py", line 238, in rebuild
    self.headers)
  File "/cygdrive/c/source/tempest/tempest/common/rest_client.py", line 152, in post
    return self.request('POST', url, headers, body)
  File "/cygdrive/c/source/tempest/tempest/common/rest_client.py", line 199, in request
    raise exceptions.Duplicate(resp_body)
Duplicate: An object with that identifier already exists
Details: An object with that identifier already exists
Details: {u'conflictingRequest': {u'message': u"Cannot 'rebuild' while instance is in vm_state building", u'code': 409}}
-------------------- >> begin captured logging << --------------------
tempest.common.rest_client: ERROR: Request URL: http://172.18.0.156:8774/v2/fe4b597cb85140ada9ba0994012d4da9/servers/aaf3aa0a-2c66-4ea3-896f-5f2d23b38035/action
tempest.common.rest_client: ERROR: Request Body: {"rebuild": {"imageRef": "ee68ea71-d2c1-4ba8-b97c-3c9efd7023bc"}}
tempest.common.rest_client: ERROR: Response Headers: {'date': 'Wed, 30 May 2012 19:43:48 GMT', 'status': '409', 'content-length': '107', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-f0544e05-f0ad-41a0-93d6-cf8d7958202c'}
tempest.common.rest_client: ERROR: Response Body: {u'conflictingRequest': {u'message': u"Cannot 'rebuild' while instance is in vm_state building", u'code': 409}}
--------------------- >> end captured logging << ---------------------

Revision history for this message
Jay Pipes (jaypipes) wrote :

I'm also seeing this pop up occasionally on the tempest gate jenkins job...

Changed in tempest:
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/7931

Changed in tempest:
assignee: nobody → David Kranz (david-kranz)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.openstack.org/7931
Committed: http://github.com/openstack/tempest/commit/1b39dd4767c057c00ba41ab45298d0b9f5582bcb
Submitter: Jenkins
Branch: master

commit 1b39dd4767c057c00ba41ab45298d0b9f5582bcb
Author: David Kranz <email address hidden>
Date: Wed May 30 17:04:09 2012 -0400

    Wait for server to be deleted before reboot/rebuild. Fixes bug 1006586.

    Change-Id: I000d06fb3f7200b819e62cabbf7a43ef13eb8d7c

Changed in tempest:
status: In Progress → Fix Committed
Jay Pipes (jaypipes)
Changed in tempest:
status: Fix Committed → Fix Released
Sean Dague (sdague)
Changed in tempest:
milestone: none → havana-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.