DHCP's are not created instantaneously for each network when 3600 networks are created

Bug #1762412 reported by Margarita Mazepa
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
MOS Maintenance

Bug Description

a)Create 3600 neutron networks and one sub-network per network.
example : neutron net-create Net-01 (good to have a for loop in bash)
neutron subnet-create --name SubNet-01 Net-01 10.0.0.0/24
and so on.

b)Create a VM using one of the above networks.
example : nova boot --image <image-id> --flavor <flavor-id> --nic net-id=xxx VM1
c)Login to created VM
root@cic-1:~# sudo ip netns exec qdhcp-80d1e7e2-e1b9-49fd-94c6-b7f0bce85639 ssh ubuntu@10.10.0.2
Cannot open network namespace "qdhcp-80d1e7e2-e1b9-49fd-94c6-b7f0bce85639": No such file or directory

It is not possible to login to VM as DHCP is not created for the network used while booting the VM.
DHCP is created only for a few networks, not for all networks by the time all networks are created.

While 3600 networks creation is taking nearly 2 and a half hours, namespaces for these networks are created after more than 10 hours.

c)Sample script used in creating networks

root@cic-1:~# cat sample.sh
k=1
for i in $(seq 1 36);
do
for j in $(seq 1 100);
do
neutron net-create stR6-436Net-$k;
neutron subnet-create --name stR6-436SubNet-$k stR6-436Net-$k 10.0.$i.$j/24;
k=$(($k + 1))
done
done

we can reproduce this bug http://paste.openstack.org/show/718318/

tags: added: customer-found
Changed in mos:
assignee: nobody → MOS Maintenance (mos-maintenance)
description: updated
Changed in mos:
milestone: none → 9.2-mu-6
importance: Undecided → High
status: New → Confirmed
Anton Matveev (amatveev)
tags: added: sla1
Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

The root cause of this effect is that the query for creating a new network/subnet
takes less time than the actual creation of all subnet attributes:
namespaces, rules, interfaces, et cetera. Due to the asynchronous nature of neutron
you are allowed to schedule the creation of a big bunch of subnets in a short period of
time, but the real readiness of a network/subnet will occur later than the CLI returns
from the execution cycle of your query. The script you have attached does not consider
the state of the creation process and schedules a new network/subnet right after
the CLI returns from the previous query. This makes a huge queue of requests in neutron
which is being processed slower than it grows. Moreover, to avoid unordered execution,
dhcp agent executes external tools (ip, iptables, etc.) in a single thread and that
leads even to more slowness of the overall process. You end up having a snowball
effect. This is not a bug in neutron, this is a normal behavior. Please make your script
consider the states of network/subnet creation.

Given the all above I'm setting the bug report status as Invalid.

Changed in mos:
status: Confirmed → Invalid
milestone: 9.2-mu-6 → 9.x-updates
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.