CI is broken for fuel-ostf repository (stable/mitaka branch)

Bug #1621057 reported by Timur Nurlygayanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Fuel QA Team

Bug Description

Short Summary:
we use different test groupd for the jobs which select stable version of fuel-qa code and for gate job for fuel-ostf repository. This is the reason why sometimes we have broken fuel-ostf gates and we don't have any logic which prevents us using the broken fuel-qa code for velidation of the commits to fuel-ostf.

For validation of fuel-qa code we use "bvt2" test group
For fuel-ostf gate job we use "gate_ostf_update" test group.

We need to add "gate_ostf_update" test group to the job https://ci.fuel-infra.org/view/devops/job/devops.mitaka.env/ to make sure that new version of fuel-qa code won't break the gates.

example of commit on review:

https://review.openstack.org/#/c/352416

The reason:

in job https://ci.fuel-infra.org/job/mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update we use the following group of tests: "gate_ostf_update"
https://github.com/openstack/fuel-qa/blob/347cd353f5a82d1ff44e4ef8302f36d5984b0a0f/gates_tests/tests/test_review_in_ostf.py#L107-L143
see logs for more details:
https://ci.fuel-infra.org/job/mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update/415/consoleFull

In job which validate the fuel-qa code (https://ci.fuel-infra.org/view/devops/job/devops.mitaka.env/) we use group "bvt_2" for the verification of fuel-qa code which we will use in gate_ostf_update job, in the result we use different tests in these jobs and sometimes we select the version of the code which breaks the gates for fuel-ostf.

Tags: area-qa
Changed in fuel:
assignee: nobody → Fuel QA Team (fuel-qa)
milestone: none → 9.1
importance: Undecided → High
status: New → Confirmed
tags: added: area-qa
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Timur, please look at the fail logs from review, and assign to the proper team

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Timur Nurlygayanov (tnurlygayanov)
Changed in fuel:
assignee: Timur Nurlygayanov (tnurlygayanov) → Fuel Sustaining (fuel-sustaining-team)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

In OSTF logs we can see:

paramiko.transport: DEBUG: Dropping user packet because connection is dead.
fuel_health.tests.ha.test_haproxy: DEBUG: Dead backends ['object-storage node-2 Status: DOWN/L7TOUT Sessions: 0 Rate: 0 ']

2016-09-07 12:08:25 FAILURE Check state of haproxy backends on controllers (fuel_health.tests.ha.test_haproxy.HAProxyCheck.test_001_check_state_of_backends) Some haproxy backend has down state.. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/ha/test_haproxy.py", line 92, in test_001_check_state_of_backends
    "Step 2 failed: Some haproxy backend has down state.")
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 164, in verify_response_true
    self.fail(message.format(failed_step_msg, msg))
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 2 failed: Some haproxy backend has down state.. Please refer to OpenStack logs for more details.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

It looks like the issue was fixed by updating of the version of fuel-qa code which we use in this job.

From the other side the issue is not completely fixed because now we use new fuel-qa version which is also unstable (see runs 410 and 411 for example - two runs with the same version of code, one run failed and one passed...):
https://ci.fuel-infra.org/job/mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update/

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The reason:

in job https://ci.fuel-infra.org/job/mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update we use the following group of tests: "gate_ostf_update"
https://github.com/openstack/fuel-qa/blob/347cd353f5a82d1ff44e4ef8302f36d5984b0a0f/gates_tests/tests/test_review_in_ostf.py#L107-L143
see logs for more details:
https://ci.fuel-infra.org/job/mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update/415/consoleFull

In job which validate the fuel-qa code we use group bvt_2 for the verification of fuel-qa code which we will use in gate_ostf_update job, in the result we use different tests in these jobs and sometimes we select the version of the code which breaks the gates for fuel-ostf.

description: updated
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Fuel CI (fuel-ci)
Revision history for this message
Aleksandra Fedorova (bookwar) wrote :

Infra team has nothing to do with this issue. PLease discuss between Dev and QA teams what exactly you want to test on a patchset in fuel-ostf repo and then send a patch to update CI configuration

Changed in fuel:
assignee: Fuel CI (fuel-ci) → Fuel QA Team (fuel-qa)
description: updated
description: updated
description: updated
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :
Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

We have committed the fix to fuel-qa repository, need to update the CI job, here is the bug for CI team:

https://bugs.launchpad.net/fuel/+bug/1621523

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.