ERROR neutron.api.v2.resource index was received during rally run on 200 nodes

Bug #1509444 reported by Iryna Vovk
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
Medium
Fuel Plugins Bugs
8.0.x
Won't Fix
Medium
Fuel Plugins Bugs
Mitaka
Won't Fix
Medium
Fuel Plugins Bugs
Newton
Confirmed
Medium
Fuel Plugins Bugs

Bug Description

During rally run on 200 nodes for Contrail Plugin with 6.1 release iso version we received such error in /var/log/neutron/server.log.2:

2015-10-21 14:23:54.735 24088 ERROR neutron.api.v2.resource [req-0411ec2f-455d-4034-b06e-9a7b3598a68e None] index failed

Steps to reproduce:
1. Install Contrail plugin on the master node;
1. Create cluster with gre segmentation and select ceph OSD;
2. Add 3 nodes with controller + mongo DB roles,
add 170 computes, 20 compute + ceph OSD roles and 3 base-os roles;
3. Configure interfaces;
4. Deploy cluster.
5. Run rally scenarios against cluster.

Actual result from neutron logs:
 from /var/log/neutron/server.log.2.gz on node-65(Controller)
2015-10-21 14:23:54.735 24088 ERROR neutron.api.v2.resource [req-0411ec2f-455d-4034-b06e-9a7b3598a68e None] index failed
2015-10-21 14:23:54.735 24033 ERROR neutron.api.v2.resource [req-cfafa26d-1a92-484d-8370-bbfe91c6a55d None] create failed
2015-10-21 14:23:54.746 24021 ERROR neutron.api.v2.resource [req-e428b013-6468-4cfa-822d-4102c9b14cc3 None] create failed
2015-10-21 14:23:54.745 24080 ERROR neutron.api.v2.resource [req-e0745347-d019-4b8d-bf98-55ad7870432c None] create failed
2015-10-21 14:23:58.326 24067 ERROR neutron.api.v2.resource [req-14bee609-916e-43bf-ad3c-8a125c91e370 None] create failed
2015-10-21 14:23:59.137 24122 ERROR neutron.api.v2.resource [req-e67d4d9a-0a08-4e02-9d25-65d569cbb081 None] create failed
2015-10-21 14:24:42.526 24009 ERROR neutron.api.v2.resource [req-c672e420-6f3c-4038-be0b-2b18df5381c2 None] index failed
2015-10-21 14:24:45.798 24021 ERROR neutron.api.v2.resource [req-8965d9a9-edbd-42fd-9e82-b4e14795f584 None] index failed
2015-10-21 14:24:49.019 24075 ERROR neutron.api.v2.resource [req-40a0992d-aeaf-4bd5-98d0-59a44708e0c9 None] index failed
2015-10-21 14:29:34.195 24030 ERROR neutron.api.v2.resource [req-2b270a3e-87ee-440b-96fc-d4b6eb9d0d37 None] show failed

Additional info:
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

Revision history for this message
Iryna Vovk (ivovk) wrote :
Revision history for this message
Iryna Vovk (ivovk) wrote :
Changed in fuel:
status: New → Confirmed
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → MOS Neutron (mos-neutron)
milestone: none → 8.0
tags: added: scale
Changed in fuel:
importance: Undecided → Medium
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Full error message from logs looks like this:

2015-10-21 12:50:15.237 24085 INFO neutron.api.v2.resource [req-8e179b7d-7b05-4ece-bce1-11150d2a0df3 None] create failed (client error): No more IP addresses available on network a360a153-eebc-410c-b608-2ce1e3551e99

This is an attempt of creation of more ports than ip addresses available, and indicate an error in tests.

Closing is as 'Invalid' for mos-neutron.
Please ping me directly if you don't think it's an issue with tests.

Changed in fuel:
status: Confirmed → Invalid
Dmitry Pyzhov (dpyzhov)
tags: added: area-mos
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Missed that during initial analisys. THere is 'NeutronException: An unknown exception occurred' being thrown. Contrail core plugin is the source.

Changed in fuel:
status: Invalid → Confirmed
assignee: MOS Neutron (mos-neutron) → nobody
Maciej Relewicz (rlu)
Changed in fuel:
assignee: nobody → Fuel Plugins Bugs (fuel-plugins-bugs)
Dmitry Pyzhov (dpyzhov)
tags: added: area-plugins
removed: area-mos
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

It looks like the issue was not fixed in MOS 8.0, moved to MOS 8.0 updates because it is medium bug and we have HCF for 8.0 release.

Changed in fuel:
milestone: 8.0 → 8.0-updates
milestone: 8.0-updates → 9.0
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Won't Fix for 8.0-updates because of Medium importance

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.