Rally

Rally does not clean the security_group when the nova_boot testing is finished

Bug #1355629 reported by Sun Jun
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Rally
Fix Released
High
Unassigned
Rally 0.0.4

Bug Description

When I am going to run the nova_boot_server benchmark which is configured as below:
{
    "NovaServers.boot_server": [
        {
            "args": {
                "flavor": {
                    "name": "m1.tiny"
                },
                "image": {
                    "name": "cirros"
                },
                "nics":[{
                    "net-id":"ccd6ce3e-eeba-47b6-a104-63c2ca853aa9"
                }]
            },
            "runner": {
                "type": "constant",
                "times": 10,
                "concurrency": 5
            },
            "context": {
                "users": {
                    "tenants": 10,
                    "users_per_tenant": 1
                },
                "quotas": {
                    "nova": {
                        "instances": 5000,
                        "cores": 50
                    }
                }
            }
        }
    ]
}

and before the testing , I type the command "neutron security-group-list" , the results are:
# neutron security-group-list
+--------------------------------------+---------+-------------+
| id | name | description |
+--------------------------------------+---------+-------------+
| b4fdfac6-4d95-4ec0-85a6-a0c657844920 | default | default |
| ba3bb113-9d47-4d7b-9831-2e724927fb0c | default | default |
+--------------------------------------+---------+-------------+

but when I finish the rally test, the command "neutron security-group-list" shows :
+--------------------------------------+---------+-------------+
| id | name | description |
+--------------------------------------+---------+-------------+
| 0204725e-da34-44ee-b9ea-d93d7ca22630 | default | default |
| 062df76e-3c81-4fd7-9b1a-572f19e39ff9 | default | default |
| 07c0cb44-e444-49ad-a16f-d6ecdee93fd6 | default | default |
| 21f96045-aab9-4ed9-8234-e5ffa7a32c9a | default | default |
| 414c2ab1-3f54-4d97-8015-dca3d646d338 | default | default |
| 6b01f2bc-838e-49a8-86a6-08562b169075 | default | default |
| 8a543f76-77ac-4d13-a540-93f90c60d029 | default | default |
| b4fdfac6-4d95-4ec0-85a6-a0c657844920 | default | default |
| ba3bb113-9d47-4d7b-9831-2e724927fb0c | default | default |
| cf731999-973f-40e8-a597-46ba3b1c6271 | default | default |
| f9c769b6-c488-43eb-823a-91e96799d6ea | default | default |
| fd8fb38a-b984-4e9b-86f1-431273885a2c | default | default |
+--------------------------------------+---------+-------------+

It seems that Rally create neutron a security group for each tenant, but it forget to clean them up.

Revision history for this message
Sergey Skripnick (eyerediskin) wrote :

Thanks for reporting. I'll try to reproduce and fix.

Changed in rally:
assignee: nobody → Sergey Skripnick (eyerediskin)
Revision history for this message
Sergey Skripnick (eyerediskin) wrote :

Just tried rally from master and fresh devstack with neutron, and dont see any extra neutron security groups. Could you please provide more info? Which rally and cloud was used?

Changed in rally:
status: New → Incomplete
Revision history for this message
Sun Jun (sanjeo) wrote :

I also ran rally from admin user, but i did not use an all-in-one setup.

rally version:0.0.1 (clone from github on Jul 18)
openstack: 2013.2.3-1.el6 @openstack-havana

Revision history for this message
Sergey Skripnick (eyerediskin) wrote :

Reproduced with scenario VMTasks.boot_runcommand_delete

Changed in rally:
status: Incomplete → Confirmed
importance: Undecided → High
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

There is a pretty old bug report regarding this, with no real progress made in almost three years:

https://bugs.launchpad.net/keystone/+bug/967832

Would be nice to have some workaround for it in rally until this is solved globally.

Revision history for this message
Boris Pavlovic (boris-42) wrote :

Jens,

Actually Rally is cleaning up all resources. We have special mechanism for this. And as far as I know Rally is deleting all security groups now

Revision history for this message
Boris Pavlovic (boris-42) wrote :

Jens,

By the way could you reproduce this locally?

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Hi Boris,

sorry for the late reply, I didn't subscribe myself to the bug, so I saw your question only now.

Yes, I can reproduce it. I also looked into the code a bit and it seems that there are three issues here:

1. rally.benchmark.context.cleanup.manager tries to delete the secgroups for all users. However, if there are multiple users per tenant, they share the same secgroup, so after the first deletion the other ones produce errors. These however are only seen when running in debug mode, so this is a minor issue.

2. After rally.benchmark.context.cleanup.manager has finished, rally.benchmark.context.secgroup tries to clean up the secgroups once more, resulting in errors that are also shown in normal output. So either the cleanup in rally.benchmark.context.secgroup is obsolete now, or it should run before rally.benchmark.context.cleanup.manager and also remove the corresponding attributes from the context.

These two issues are more or less cosmetical, now comes the major one:

3. While rally does very thoroughly clean up the secgroups that are explicitly created, neutron is playing tricks and also creates a security group called 'default' for every tenant. These are not removed on tenant deletion automatically, so we should add some code into rally in order to do this.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

@Sergey: Are you still working on this? Otherwise please unassign yourself and let someone else take over.

Boris Pavlovic (boris-42)
Changed in rally:
assignee: Sergey Skripnick (eyerediskin) → nobody
Dr. Jens Harbott (j-harbott)
Changed in rally:
assignee: nobody → Dr. Jens Rosenboom (j-rosenboom-j)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to rally (master)

Fix proposed to branch: master
Review: https://review.openstack.org/179428

Changed in rally:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to rally (master)

Reviewed: https://review.openstack.org/179428
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=977afacf33a71b6e520a214bba81975367663ab9
Submitter: Jenkins
Branch: master

commit 977afacf33a71b6e520a214bba81975367663ab9
Author: Jens Rosenboom <email address hidden>
Date: Fri May 1 19:08:38 2015 +0200

    Fix security groups cleanup

    In addition to the security groups that are explicitly created, Neutron
    automatically creates a security group called "default" for each tenant.
    We have to clean these up after destroying networks using the admin
    context.

    Co-Authored-By: Chris St. Pierre <email address hidden>
    Change-Id: Ib8da1c608a3355cd2e08a7bc1a5aaa86444d1c39
    Closes-Bug: 1355629

Changed in rally:
status: In Progress → Fix Committed
Boris Pavlovic (boris-42)
Changed in rally:
status: Fix Committed → Fix Released
milestone: none → 0.0.4
Dr. Jens Harbott (j-harbott)
Changed in rally:
assignee: Dr. Jens Rosenboom (j-rosenboom-j) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.