Neutron jobs won't pass tempest running in parallel

Bug #1216076 reported by Matthew Treinish
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fix Released
Matthew Treinish
Fix Released
Miguel Lavalle

Bug Description

When running tempest in parallel and neutron is enabled the tests do not work. You hit this failure:

ft5.1: setUpClass (tempest.api.volume.test_volumes_actions.VolumesActionsTest)_StringException: Traceback (most recent call last):
  File "tempest/api/volume/", line 37, in setUpClass
  File "tempest/services/compute/json/", line 81, in create_server
    resp, body ='servers', post_body, self.headers)
  File "tempest/common/", line 259, in post
    return self.request('POST', url, headers, body)
  File "tempest/common/", line 387, in request
    resp, resp_body)
  File "tempest/common/", line 437, in _error_checker
    raise exceptions.BadRequest(resp_body)
BadRequest: Bad request
Details: {u'badRequest': {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400}}

(It occurs on a few tests but the error message is the same, see the sample logs below.)

Logs can be found here:

Additionally, by default in devstack tenant isolation is disabled when neutron is enabled. This is a requirement for running tempest in parallel. If parallel tempest is going to be used for neutron gate jobs this needs to be changed

Revision history for this message
Attila Fazekas (afazekas) wrote :

This looks like a duplicate of

Revision history for this message
Miguel Lavalle (minsel) wrote :


Did you add a network to the tennant you are using to create this server? That error happens when you boot a server, you don't specify networks in the boot operation and the tenant has acces to 2 or more networks. In that situation, nova doesn't know what network to connect the server to.

In any case, I have a proposal to solve the tenant isolation / parallel execution situation. It boilds down to implement in devstack the use case "Single Flat Network", as described here

and here

 A devstack instance, when created with neutron as the networking service, creates a network called 'private' for tenant 'demo'. That network is private to that tenant. We need to make that network shared. For this to happen, we have to change in devstack/lib/neutron line 332 from:

NET_ID=$(neutron net-create --tenant_id $TENANT_ID "$PRIVATE_NETWORK_NAME" | grep ' id ' | get_field 2)


NET_ID=$(neutron net-create --tenant_id $TENANT_ID --shared "$PRIVATE_NETWORK_NAME" | grep ' id ' | get_field 2)

Once we do this, I think we can enable tenant isolation in Tempest and all the tenants that we create will have access to this shared network. And the end result is that we will be able to run Tempest in parallel.

I just created a devstack instance in my home lab. It works fine. I tested the shared network booting servers with tenants demo and alt_demo and it works fine.

I didn't try parallel execution because I am still running nosetests. But you can test this idea tomorrow. I will ping you as soon as I get to the office

If this works, we will also have to change line 328 in devstack/lib/neutron to create a shared network. Also, over time, we may want to change the name of the network from private to something else.

Revision history for this message
Attila Fazekas (afazekas) wrote :

Thank you.

I do not know what is the reason behind the 'private' network does not created as shared at the first place.

The other option is adding new option for tempest for handling the environment without shared network and create a router+net+subnet for every 'randomly' created tenant.

It can be made more smarter by adding '_new_net_for' set for every test class with the , 'primary', 'admin', 'alt' , and just create networks for the explicitly mentioned tenants.

Changed in tempest:
assignee: nobody → Miguel Lavalle (minsel)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master

Changed in tempest:
status: Confirmed → In Progress
Changed in devstack:
assignee: nobody → Matthew Treinish (treinish)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Submitter: Jenkins
Branch: master

commit b8fabc5614f2648a297cb3933723cf37e0418861
Author: Miguel Lavalle <email address hidden>
Date: Fri Aug 23 11:19:57 2013 -0500

    Handling network resources in tenant isolation

    Functionality is added to create and delete isolated network, subnet and router
    when isolated credentials are created and deleted for a primary, admin or alt

    When a set of isolated credentials are created, the tenant gets also a private
    network, subnet and router. The router connects the tenat's network to the
    public network designated by public_network_id in etc/tempest.conf

    Change-Id: I7587c85017cca09f7a67eae0670f67b2bceacb60
    Fixes: bug #1216076

Changed in tempest:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Submitter: Jenkins
Branch: master

commit 3d84cf2d7c323750971cf2d27f3a4eaa26cb7a9f
Author: Matthew Treinish <email address hidden>
Date: Thu Sep 12 13:25:54 2013 -0400

    Enable tenant isolation to tempest for neutron

    This commit re-enables tenant isolation in tempest for neutron. This
    is a requirement for running tempest in parallel. This commit depends
    on tempest change I7587c85017cca09f7a67eae0670f67b2bceacb60

    Fixes bug 1216076

    Change-Id: I63a30bacd48cecd110fb90e1fc718249c2b1904b

Changed in devstack:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.