[master] Telemetry Tempest integration tests failed giving Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet.

Bug #1792862 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Won't Fix
Wishlist
chandan kumar

Bug Description

On master promotion pipeline, telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration is failing at FS017
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/90fd211/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-09-17_02_39_38

2018-09-17 02:39:38 |
2018-09-17 02:39:38 | tearDownClass (telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration)
2018-09-17 02:39:38 | -----------------------------------------------------------------------------------------------------
2018-09-17 02:39:38 |
2018-09-17 02:39:38 | Captured traceback:
2018-09-17 02:39:38 | ~~~~~~~~~~~~~~~~~~~
2018-09-17 02:39:38 | Traceback (most recent call last):
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 224, in tearDownClass
2018-09-17 02:39:38 | six.reraise(etype, value, trace)
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 196, in tearDownClass
2018-09-17 02:39:38 | teardown()
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/test.py", line 688, in clear_credentials
2018-09-17 02:39:38 | cls._creds_provider.clear_creds()
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 445, in clear_creds
2018-09-17 02:39:38 | self._clear_isolated_net_resources()
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 436, in _clear_isolated_net_resources
2018-09-17 02:39:38 | creds.subnet['name'])
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/dynamic_creds.py", line 386, in _clear_isolated_subnet
2018-09-17 02:39:38 | client.delete_subnet(subnet_id)
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
2018-09-17 02:39:38 | return self.delete_resource(uri)
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 41, in delete_resource
2018-09-17 02:39:38 | resp, body = self.delete(req_uri)
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
2018-09-17 02:39:38 | return self.request('DELETE', url, extra_headers, headers, body)
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 670, in request
2018-09-17 02:39:38 | self._error_checker(resp, resp_body)
2018-09-17 02:39:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 791, in _error_checker
2018-09-17 02:39:38 | raise exceptions.Conflict(resp_body, resp=resp)
2018-09-17 02:39:38 | tempest.lib.exceptions.Conflict: Conflict with state of target resource
2018-09-17 02:39:38 | Details: {u'message': u'Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}

https://review.openstack.org/583875 -> Allow to use environment variable to override some metric values patch on telemetry-tempest-plugin causes the failure

Some config are not added on ceilometer side which causes the failure.

Tags: ci tempest
Revision history for this message
chandan kumar (chkumar246) wrote :

Fix proposed on ceilometer side:
https://review.openstack.org/#/c/602625/1 -> Fix logic when selecting configuration file
https://review.openstack.org/#/c/602624/1 -> agent: Remove unused parameter in load_config

Changed in tripleo:
assignee: nobody → Mehdi Abaakouk (sileht)
Revision history for this message
chandan kumar (chkumar246) wrote :
Mehdi Abaakouk (sileht)
Changed in tripleo:
status: Triaged → Fix Committed
Revision history for this message
chandan kumar (chkumar246) wrote :
Revision history for this message
Mehdi Abaakouk (sileht) wrote :
Revision history for this message
wes hayutin (weshayutin) wrote :

I only see periodic jobs affected here.. removing alert

tags: removed: alert
Revision history for this message
Arx Cruz (arxcruz) wrote :

I am seeing some of the patches not merged yet, and the test still failing on our side.
@Wes it's not failing anymore, because it is in our skip list.
This is still a problem, also, I'm not seeing the patches here related to the failure.

Changed in tripleo:
status: Fix Committed → Triaged
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
assignee: Mehdi Abaakouk (sileht) → chandan kumar (chkumar246)
status: Triaged → In Progress
Revision history for this message
chandan kumar (chkumar246) wrote :
Revision history for this message
chandan kumar (chkumar246) wrote :

review to unskip telemetry tests https://review.openstack.org/604311

Revision history for this message
chandan kumar (chkumar246) wrote :
Revision history for this message
chandan kumar (chkumar246) wrote :

Once this change is merged https://github.com/gnocchixyz/gnocchi/pull/1017 we are good to go.

Revision history for this message
chandan kumar (chkumar246) wrote :

After applying all the changes we are now seeing a new issue:
https://logs.rdoproject.org/49/18749/11/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/a103a28/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-19_14_31_34

2019-02-19 14:31:34 | Traceback (most recent call last):
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/telemetry_tempest_plugin/scenario/utils.py", line 61, in test
2019-02-19 14:31:34 | run_test(self, test_dir, filename)
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/telemetry_tempest_plugin/scenario/utils.py", line 53, in run_test
2019-02-19 14:31:34 | test_class_instance.fail(msg)
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
2019-02-19 14:31:34 | raise self.failureException(msg)
2019-02-19 14:31:34 | AssertionError: From test "control stack status" :
2019-02-19 14:31:34 | _StringException: Traceback (most recent call last):
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/gabbi/suitemaker.py", line 95, in do_test
2019-02-19 14:31:34 | return test_method(*args, **kwargs)
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 94, in wrapper
2019-02-19 14:31:34 | func(self)
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 148, in test_request
2019-02-19 14:31:34 | self._run_test()
2019-02-19 14:31:34 | File "/usr/lib/python2.7/site-packages/gabbi/case.py", line 551, in _run_test
2019-02-19 14:31:34 | raise failure
2019-02-19 14:31:34 | testtools.matchers._impl.MismatchError: 'CREATE_COMPLETE' != u'CREATE_FAILED': Unable to match $.stack.stack_status as CREATE_COMPLETE, got CREATE_FAILED

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart-extras (master)

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.openstack.org/604311
Reason: already done

Revision history for this message
Gabriele Cerami (gcerami) wrote :
Changed in tripleo:
status: Fix Released → Triaged
tags: added: ci tempest
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/661599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.opendev.org/661599
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=dcfb135795f64134ff8bb21a92a81905d839a340
Submitter: Zuul
Branch: master

commit dcfb135795f64134ff8bb21a92a81905d839a340
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 27 19:38:43 2019 +0530

    Put telemetry tempest tests under master skip list

    Currently telemetry tempest tests are broken on scenario 01
    and 02 and is also blocking upstream check jobs, we are putting
    it to skip list till we find what causes the issue.

    Related-Bug: #1792862

    Change-Id: Iae962eb585e179a980915cb4b402c415319aa75f
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

Revision history for this message
Marios Andreou (marios-b) wrote :

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-scenario002-standalone-master

last fail for this was on the 28th but otherwise green - the skip merged

removing promotion blocker please re-add if you disagree

tags: removed: promotion-blocker
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Incomplete
Changed in tripleo:
importance: Critical → Wishlist
Amol Kahat (amolkahat)
Changed in tripleo:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.