Fuel for OpenStack

[system_tests]We need to provide a liitle bit more time to galera after failover before ostf run

Bug #1352862 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Artem Panchenko
Fuel for OpenStack 6.0
5.1.x
Fix Released
Medium
Artem Panchenko
Fuel for OpenStack 5.1.1

Bug Description

http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.thread_5/126/testReport/junit/%28root%29/ha_destroy_controllers/ha_destroy_controllers/

Error Message

Failed tests, fails: 5 should fail: 1 failed tests name: [{u'Check data replication over mysql': u'Time limit exceeded while waiting for detect mysql node to finish. Please refer to OpenStack logs for more details.'}, {u'Check amount of tables in databases is the same on each node': u'Time limit exceeded while waiting for get amount of tables for each database to finish. Please refer to OpenStack logs for more details.'}, {u'Check RabbitMQ is available': u'Number of controllers is not equal to number of cluster nodes.'}, {u'Request image list': u'Time limit exceeded while waiting for images listing to finish. Please refer to OpenStack logs for more details.'}, {u'Check that required services are running': u'Some nova services have not been started.. Please refer to OpenStack logs for more details.'}, {u'Create volume and boot instance from it': u'There are no cinder nodes or ceph storage for volume'}, {u'Create volume and attach it to instance': u'There are no cinder nodes or ceph storage for volume'}]

Test failed with timeout to connect to db , As about image list tests - the same problem, when we run this tests mysql was note ready , but then all smoke tests pass. I beleive that we should wait a litlle bit after failover before ostf run

Tags: system-tests
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Okay, but my suggestion to wait until oslo patch will be in master, if situation persists then increase timeout.

Changed in fuel:
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-main (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/118563

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/118563
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=9d0064b342881bd59668c018ae1876341da059a2
Submitter: Jenkins
Branch: master

commit 9d0064b342881bd59668c018ae1876341da059a2
Author: NastyaUrlapova <email address hidden>
Date: Wed Sep 3 11:35:26 2014 +0400

    Added galera check after revert snapshot

    Change-Id: I8548a560658154929d43a29e36536214cb6d8f5c
    Related-Bug: #1352862

Nastya Urlapova (aurlapova)
Changed in fuel:
status: Incomplete → Confirmed
Bogdan Dobrelya (bogdando)
Changed in fuel:
status: Confirmed → Triaged
milestone: 5.1 → 6.0
Nastya Urlapova (aurlapova)
Changed in fuel:
status: Triaged → Fix Committed
Nastya Urlapova (aurlapova)
summary: - [system_tests] 5.1 We need to provide a liitle bit more time to galera
- after failover before ostf run
+ [system_tests]We need to provide a liitle bit more time to galera after
+ failover before ostf run
Changed in fuel:
status: Fix Committed → Fix Released
status: Fix Released → Fix Committed
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

We have a reproduction here: http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.centos.ha_neutron_destructive/35/testReport/%28root%29/ha_destroy_controllers/ha_destroy_controllers/

Changed in fuel:
status: Fix Committed → Confirmed
Nastya Urlapova (aurlapova)
Changed in fuel:
milestone: 6.0 → 6.1
Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

Here is the part of OSTF logs:

AuthorizationFailure: Authorization Failed: Cannot connect to proxy. Socket error: [Errno 113] No route to host.
fuel_health.nmanager: DEBUG: Unable to initialize Keystone client: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 73, in __init__
    self.identity_client = self._get_identity_client()
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 171, in _get_identity_client
    insecure=dscv)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 144, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 392, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 179, in get_raw_token_from_identity_service
    "%s" % e)
AuthorizationFailure: Authorization Failed: Cannot connect to proxy. Socket error: [Errno 113] No route to host.
--------------------- >> end captured logging << --------------------

Health checks fail because OSTF tries to use proxy which is running on suspended controller.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/137676

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Artem Panchenko (apanchenko-8)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/137677

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/137676
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=30d9cc4fdf973849e2a57d073b5d3313cdda1bec
Submitter: Jenkins
Branch: master

commit 30d9cc4fdf973849e2a57d073b5d3313cdda1bec
Author: Artem Panchenko <email address hidden>
Date: Thu Nov 27 18:29:27 2014 +0200

    Wait until Nailgun set suspended node as offline

    When we run OSTF after suspending one of controllers
    it tries to use proxy server on some of online nodes.
    We should wait until Nailgun updates suspended node
    status to 'offline' to avoid proxy connection errors.
    Also run OSTF after each node shutdown.

    Change-Id: I0d8287706fbc199cb49f3af2552b14c73fdc94cb
    Closes-bug: #1352862

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (stable/5.1)

Reviewed: https://review.openstack.org/137677
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=27631f8e7758ee67cfbf3bf320aeb00c1ae26a5e
Submitter: Jenkins
Branch: stable/5.1

commit 27631f8e7758ee67cfbf3bf320aeb00c1ae26a5e
Author: Artem Panchenko <email address hidden>
Date: Thu Nov 27 18:29:27 2014 +0200

    Wait until Nailgun set suspended node as offline

    When we run OSTF after suspending one of controllers
    it tries to use proxy server on some of online nodes.
    We should wait until Nailgun updates suspended node
    status to 'offline' to avoid proxy connection errors.
    Also run OSTF after each node shutdown.

    Change-Id: I0d8287706fbc199cb49f3af2552b14c73fdc94cb
    Closes-bug: #1352862

Artem Panchenko (apanchenko-8)
Changed in fuel:
milestone: 6.1 → 6.0
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Released on 5.1.1:
{u'build_id': u'2014-12-03_01-07-36', u'ostf_sha': u'64cb59c681658a7a55cc2c09d079072a41beb346', u'build_number': u'48', u'auth_required': True, u'nailgun_sha': u'500e36d08a45dbb389bf2bd97673d9bff48ee84d', u'production': u'docker', u'api': u'1.0', u'fuelmain_sha': u'7626c5aeedcde77ad22fc081c25768944697d404', u'astute_sha': u'ef8aa0fd0e3ce20709612906f1f0551b5682a6ce', u'feature_groups': [u'mirantis'], u'release': u'5.1.1', u'release_versions': {u'2014.1.3-5.1.1': {u'VERSION': {u'build_id': u'2014-12-03_01-07-36', u'ostf_sha': u'64cb59c681658a7a55cc2c09d079072a41beb346', u'build_number': u'48', u'api': u'1.0', u'nailgun_sha': u'500e36d08a45dbb389bf2bd97673d9bff48ee84d', u'production': u'docker', u'fuelmain_sha': u'7626c5aeedcde77ad22fc081c25768944697d404', u'astute_sha': u'ef8aa0fd0e3ce20709612906f1f0551b5682a6ce', u'feature_groups': [u'mirantis'], u'release': u'5.1.1', u'fuellib_sha': u'a3043477337b4a0a8fd166dc83d6cd5d504f5da8'}}}, u'fuellib_sha': u'a3043477337b4a0a8fd166dc83d6cd5d504f5da8'}

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Released on 6.0:
api: '1.0'
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-07_22-41-06
build_number: '43'
feature_groups:
- mirantis
fuellib_sha: 3e1f23e042ea8f4f25b6bf956aca17e29e57e708
fuelmain_sha: 134f70e8332d9a52f17565e23e8b14f092efb735
nailgun_sha: 73a2fa3aa669ca8d95788e80dc052dfa16eca24a
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
      build_id: 2014-12-07_22-41-06
      build_number: '43'
      feature_groups:
      - mirantis
      fuellib_sha: 3e1f23e042ea8f4f25b6bf956aca17e29e57e708
      fuelmain_sha: 134f70e8332d9a52f17565e23e8b14f092efb735
      nailgun_sha: 73a2fa3aa669ca8d95788e80dc052dfa16eca24a
      ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
      production: docker
      release: '6.0'

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.