test_get_service_by_host_name is racy

Bug #1205450 reported by Dan Smith
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Critical
Sean Dague

Bug Description

This racy test breaks if the timestamps on the service list has changed:

2013-07-26 18:00:36.696 |
2013-07-26 18:00:36.697 | ======================================================================
2013-07-26 18:00:36.697 | FAIL: tempest.api.compute.admin.test_services.ServicesAdminTestJSON.test_get_service_by_host_name[gate]
2013-07-26 18:00:36.698 | ----------------------------------------------------------------------
2013-07-26 18:00:36.734 | _StringException: Traceback (most recent call last):
2013-07-26 18:00:36.734 | File "/opt/stack/new/tempest/tempest/api/compute/admin/test_services.py", line 67, in test_get_service_by_host_name
2013-07-26 18:00:36.734 | self.assertEqual(services_on_host, services)
2013-07-26 18:00:36.734 | File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 322, in assertEqual
2013-07-26 18:00:36.734 | self.assertThat(observed, matcher, message)
2013-07-26 18:00:36.734 | File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 417, in assertThat
2013-07-26 18:00:36.734 | raise MismatchError(matchee, matcher, mismatch, verbose)
2013-07-26 18:00:36.734 | MismatchError: !=:
2013-07-26 18:00:36.735 | reference = [{u'binary': u'nova-conductor',
2013-07-26 18:00:36.735 | u'disabled_reason': None,
2013-07-26 18:00:36.735 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.735 | u'state': u'up',
2013-07-26 18:00:36.735 | u'status': u'enabled',
2013-07-26 18:00:36.735 | u'updated_at': u'2013-07-26T17:23:42.854890',
2013-07-26 18:00:36.735 | u'zone': u'internal'},
2013-07-26 18:00:36.735 | {u'binary': u'nova-compute',
2013-07-26 18:00:36.735 | u'disabled_reason': None,
2013-07-26 18:00:36.735 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.735 | u'state': u'up',
2013-07-26 18:00:36.735 | u'status': u'enabled',
2013-07-26 18:00:36.735 | u'updated_at': u'2013-07-26T17:23:43.126240',
2013-07-26 18:00:36.735 | u'zone': u'nova'},
2013-07-26 18:00:36.735 | {u'binary': u'nova-cert',
2013-07-26 18:00:36.735 | u'disabled_reason': None,
2013-07-26 18:00:36.736 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.736 | u'state': u'up',
2013-07-26 18:00:36.736 | u'status': u'enabled',
2013-07-26 18:00:36.736 | u'updated_at': u'2013-07-26T17:23:43.335187',
2013-07-26 18:00:36.736 | u'zone': u'internal'},
2013-07-26 18:00:36.736 | {u'binary': u'nova-scheduler',
2013-07-26 18:00:36.736 | u'disabled_reason': None,
2013-07-26 18:00:36.736 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.736 | u'state': u'up',
2013-07-26 18:00:36.736 | u'status': u'enabled',
2013-07-26 18:00:36.736 | u'updated_at': u'2013-07-26T17:23:43.353705',
2013-07-26 18:00:36.736 | u'zone': u'internal'},
2013-07-26 18:00:36.736 | {u'binary': u'nova-network',
2013-07-26 18:00:36.736 | u'disabled_reason': None,
2013-07-26 18:00:36.736 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.736 | u'state': u'up',
2013-07-26 18:00:36.737 | u'status': u'enabled',
2013-07-26 18:00:36.737 | u'updated_at': u'2013-07-26T17:23:43.805627',
2013-07-26 18:00:36.737 | u'zone': u'internal'}]
2013-07-26 18:00:36.737 | actual = [{u'binary': u'nova-compute',
2013-07-26 18:00:36.737 | u'disabled_reason': None,
2013-07-26 18:00:36.737 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.737 | u'state': u'up',
2013-07-26 18:00:36.737 | u'status': u'enabled',
2013-07-26 18:00:36.737 | u'updated_at': u'2013-07-26T17:23:43.126240',
2013-07-26 18:00:36.737 | u'zone': u'nova'},
2013-07-26 18:00:36.737 | {u'binary': u'nova-cert',
2013-07-26 18:00:36.737 | u'disabled_reason': None,
2013-07-26 18:00:36.737 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.737 | u'state': u'up',
2013-07-26 18:00:36.737 | u'status': u'enabled',
2013-07-26 18:00:36.738 | u'updated_at': u'2013-07-26T17:23:43.335187',
2013-07-26 18:00:36.738 | u'zone': u'internal'},
2013-07-26 18:00:36.738 | {u'binary': u'nova-scheduler',
2013-07-26 18:00:36.738 | u'disabled_reason': None,
2013-07-26 18:00:36.738 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.738 | u'state': u'up',
2013-07-26 18:00:36.738 | u'status': u'enabled',
2013-07-26 18:00:36.738 | u'updated_at': u'2013-07-26T17:23:43.353705',
2013-07-26 18:00:36.738 | u'zone': u'internal'},
2013-07-26 18:00:36.738 | {u'binary': u'nova-network',
2013-07-26 18:00:36.738 | u'disabled_reason': None,
2013-07-26 18:00:36.738 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.738 | u'state': u'up',
2013-07-26 18:00:36.738 | u'status': u'enabled',
2013-07-26 18:00:36.738 | u'updated_at': u'2013-07-26T17:23:43.805627',
2013-07-26 18:00:36.739 | u'zone': u'internal'},
2013-07-26 18:00:36.739 | {u'binary': u'nova-conductor',
2013-07-26 18:00:36.739 | u'disabled_reason': None,
2013-07-26 18:00:36.739 | u'host': u'devstack-1374858507',
2013-07-26 18:00:36.739 | u'state': u'up',
2013-07-26 18:00:36.739 | u'status': u'enabled',
2013-07-26 18:00:36.739 | u'updated_at': u'2013-07-26T17:23:52.855895',
2013-07-26 18:00:36.739 | u'zone': u'internal'}]
2013-07-26 18:00:36.739 |
2013-07-26 18:00:36.739 | -------------------- >> begin captured logging << --------------------

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/38859

Changed in tempest:
assignee: nobody → Sean Dague (sdague)
status: New → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

causing a lot of gate resets, so we should try to land this soon

Changed in tempest:
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.openstack.org/38859
Committed: http://github.com/openstack/tempest/commit/3b61649d4d610fddfbf852245bd941f721c069ed
Submitter: Jenkins
Branch: master

commit 3b61649d4d610fddfbf852245bd941f721c069ed
Author: Sean Dague <email address hidden>
Date: Fri Jul 26 15:04:12 2013 -0400

    fix race condition in service list compares

    there is a small chance that between the 2 service queries that
    the service list is updated by the periodic job. We'll assume
    stability of the service host list in the regular case, but
    the other metadata is changeable, so only compary the binary
    lists.

    Fixes bug #1205450

    Change-Id: I29f687d4279b79f192de3bae3d15abebd3d858a1

Changed in tempest:
status: In Progress → Fix Released
Sean Dague (sdague)
Changed in tempest:
milestone: none → havana-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.