NetworkNotFound failures on network test teardown because of retries due to the initial request taking >60 seconds

Bug #1794870 reported by Matt Riedemann
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Confirmed
Medium
Unassigned
tempest
Invalid
Undecided
Unassigned

Bug Description

I've seen this in a few different tests and branches, network tests are tearing down and hitting NetworkNotFound presumably because the test already deleted the network and we're racing on teardown:

http://logs.openstack.org/70/605270/1/gate/tempest-full-py3/f18bf28/testr_results.html.gz

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/services/network/networks_client.py", line 52, in delete_network
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 310, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 675, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 781, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {'detail': '', 'type': 'NetworkNotFound', 'message': 'Network 0574d093-73f1-4a7c-b0d8-49c9f43d44fa could not be found.'}

We should just handle the 404 and ignore it since we're trying to delete the network anyway.

Revision history for this message
Matt Riedemann (mriedem) wrote :

(3:18:02 PM) slaweq: mriedem: it's not a race
(3:18:30 PM) slaweq: mriedem: I saw it already before, it's some issue with neutron that it process request very slow, see: http://logs.openstack.org/70/605270/1/gate/tempest-full-py3/f18bf28/controller/logs/screen-q-svc.txt.gz?level=INFO#_Sep_27_01_11_47_552844
(3:19:12 PM) slaweq: so first request is processed very long, and tempest client has http timeout, so client retries same request but then network is already removed
(3:19:14 PM) mriedem: time: 66.6156189
(3:19:19 PM) mriedem: wow
(3:19:19 PM) slaweq: exacly :/
(3:19:26 PM) slaweq: *exactly

Changed in tempest:
status: New → Invalid
Matt Riedemann (mriedem)
summary: - NetworkNotFound failures on network test teardown
+ NetworkNotFound failures on network test teardown because of retries due
+ to the initial request taking >60 seconds
Changed in neutron:
importance: Undecided → Medium
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.