An existing network cannot be found while creating port

Bug #1550358 reported by Mikhail Chernik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Mikhail Chernik
8.0.x
Invalid
High
Mikhail Chernik
9.x
Invalid
High
Mikhail Chernik

Bug Description

Rally scenario NeutronNetworks.create_and_delete_ports fails with error "Network 1793acce-3d4a-452b-85da-d1ded83e03f0 could not be found", while the network definitely exists.

Rally trace: http://paste.openstack.org/show/488355/
Neutron server log: http://paste.openstack.org/show/488357/

Cluster configuration:
MOS 8.0 RC2 (ISO 570)
Network segmentation: vlan
Total nodes: 203
+ controllers: 3
+ computes: 200
Optional components: heat
Storage: ephemeral_ceph, images_ceph, objects_ceph, volumes_ceph

Rally scenario:
{
  "NeutronNetworks.create_and_delete_ports": [
    {
      "runner": {
        "type": "constant",
        "concurrency": 10,
        "times": 200
      },
      "args": {
        "network_create_args": {},
        "ports_per_network": 50,
        "port_create_args": {}
      },
      "sla": {
        "failure_rate": {
          "max": 0
        }
      },
      "context": {
        "users": {
          "project_domain": "default",
          "users_per_tenant": 2,
          "tenants": 2,
          "resource_management_workers": 30,
          "user_domain": "default"
        },
        "quotas": {
          "neutron": {
            "network": -1,
            "port": -1
          }
        }
      }
    }
  ]
}

Revision history for this message
Oleg Bondarev (obondarev) wrote :
Revision history for this message
Oleg Bondarev (obondarev) wrote :

Neutron logs show the only difference of failed create port request is that it was called from a different tenant (not as other requests for creating ports on this network have). Looks like this tenant does not own the network so NetworkNotFound was raised.

I guess this is some bug in rally. I attached rally logs (comment #1): if grep by X-Auth-Token [1] we can see that one network was created with that token and then a bunch of ports on it. However the last request with that token is for creating port on other network - and it failed.

[1] gAAAAABW0GAOPyxHLh4JMgq_EZ2mNLQ1Mt7zHcKKuZffvtSafWWNsHZ1R6mfQz127xQ_157Vvs2s6--CSfOhWVzP_JBxLZImMNt15SIK5dJcgOn9a0c0fe_0lJsqWQ6D4oqOhiIU4vptYplQ_xQeATTaGIVDiTARaUVo1zvwbK6Im_r0uTO1EsI

Assigning to Rally team

Changed in mos:
assignee: Oleg Bondarev (obondarev) → Andrey Kurilin (andreykurilin)
Revision history for this message
Andrey Kurilin (andreykurilin) wrote :

The code of this scenario is too simple:
1) create or get existing network created by network context(it is turned off in task from bug-description)
    https://github.com/openstack/rally/blob/0.3.1/rally/plugins/openstack/scenarios/neutron/network.py#L305
    https://github.com/openstack/rally/blob/0.3.1/rally/plugins/openstack/scenarios/neutron/utils.py#L285-L291
2) create and delete ports for this networks
   https://github.com/openstack/rally/blob/0.3.1/rally/plugins/openstack/scenarios/neutron/network.py#L307-L308
   https://github.com/openstack/rally/blob/0.3.1/rally/plugins/openstack/scenarios/neutron/utils.py#L230-L239
   https://github.com/openstack/rally/blob/0.3.1/rally/plugins/openstack/scenarios/neutron/utils.py#L261-L266

Since initialized clients are cached for each iterations, both of steps use the same instance of neutronclient, so one tenant and one user should be used(in neutronclient did not do some magic).

Changed in mos:
assignee: Andrey Kurilin (andreykurilin) → nobody
Changed in mos:
status: New → Confirmed
Changed in mos:
assignee: nobody → MOS Neutron (mos-neutron)
importance: Undecided → High
tags: added: area-neutron neutron
Revision history for this message
Oleg Bondarev (obondarev) wrote :

So according to rally code it creates one tenant and instance of neutron client with that tenant per worker.
Each client has its own auth token.
According to rally logs we have (at least) two workers in parallel, each creating a network and then creating and deleting ports on it.
Then one worker creates a port on a network which belongs to other tenant (worker) and such request fails.
I guess this might happen due to worker somehow uses either neutron client of other worker or network of other worker.

After a conversation with Andrey Kurilin we agreed to add more rally logs and reproduce at scale to help investigation.

Revision history for this message
Alexander Ignatov (aignatov) wrote :

@mchernik according to conversation between Oleg and Andrey we request sacle team to reproduce this issue with additional logging added to Rally ^^

Revision history for this message
Alexander Ignatov (aignatov) wrote :

*scale

Revision history for this message
Andrey Kurilin (andreykurilin) wrote :

It is a rally issue. This neutron scenario is a bit unique, so only NeutronNetworks.create_and_delete_ports scenario is affected.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.