OpenStack Compute (nova)

race conditions in security-group additions

Bug #1417975 reported by Ran Ziv
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
Low
Unassigned

Bug Description

When attempting to connect multiple security groups to a single server concurrently, some of the security groups may not actually get connected despite the call returning successfully and with no error.

I've been able to reproduce this by issuing two different "server.add_security_group" calls (using the Nova python client) concurrently - while most of the time both security groups will get connected, sometimes only one does (and it's not necessarily the same one every time)

CVE References

Sean Dague (sdague)
Changed in nova:
status: New → Confirmed
importance: Undecided → High
tags: added: network security-groups
summary: - problem when connecting multiple security-groups to server concurrently
+ race conditions in security-group additions
Revision history for this message
Chang-Yi Lee (cy-lee) wrote :

Hello, could you please provide reproduce steps?
I use shell script like https://gist.github.com/anonymous/3fe6400a311723b10deb
Add/Remove multiple times but don't see any error on Horizon.

Thank you.

Revision history for this message
Ran Ziv (ran-k) wrote :

As I said, in order to reproduce this I just used the python client and issued two different calls at the same time (without even using a script, I actually just switched shells and ran the commands manually) - and again, it doesn't happen most of the time, but it does happen every now and again (obviously, a timing issue).

(also it might be better to verify this not via Horizon? I haven't used Horizon for verification afterwards, but rather used the clients to query the server for its security groups)

Chung Chih, Hung (lyanchih)
Changed in nova:
assignee: nobody → lyanchih (lyanchih)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/202436

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/202436
Reason: This patch has been stalled for quite a while, so I am going to abandon it to keep the code review queue sane. Please restore the change when it is ready for review.

Sean Dague (sdague)
Changed in nova:
status: In Progress → Confirmed
assignee: Chung Chih, Hung (lyanchih) → nobody
importance: High → Medium
stgleb (gstepanov)
Changed in nova:
assignee: nobody → stgleb (gstepanov)
stgleb (gstepanov)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/344787
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the nova review queue sane. Please feel free to restore the change if you're still working on it.

Revision history for this message
Sujitha (sujitha-neti) wrote :

The patch submitted for this bug is abandoned and there is no progress since a long time. To signal that to other contributors which might provide patches for this bug, I'm removing the assignee. Feel free to add yourself as assignee and push a patch for it.

Changed in nova:
assignee: stgleb (gstepanov) → nobody
status: In Progress → Confirmed
Sean Dague (sdague)
Changed in nova:
importance: Medium → High
tags: added: race-condition
hongbin (hongbin034)
Changed in nova:
assignee: nobody → hongbin (hongbin034)
Revision history for this message
hongbin (hongbin034) wrote :

I wanted to leverage the 'If-Match' constraints [1] to resolve the race condition. However, neutronclient doesn't seem to support this HTTP header yet.

[1] https://developer.openstack.org/api-ref/network/v2/#revisions

Revision history for this message
hongbin (hongbin034) wrote :

Created a ticket at python-neutronclient: https://bugs.launchpad.net/python-neutronclient/+bug/1737473 .

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/535510

Changed in nova:
assignee: hongbin (hongbin034) → Hongbin Lu (hongbin.lu)
status: Confirmed → In Progress
Tao Li (eric-litao)
information type: Public → Public Security
information type: Public Security → Public
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Hongbin Lu (<email address hidden>) on branch: master
Review: https://review.openstack.org/535510
Reason: According to @Andrey Volkov's comment, the API for adding/removing security is deprecated so my default position is not to proceed with this effort which might add additional complexity of the deprecated API, unless nova team still wants the fix..

Revision history for this message
Matt Riedemann (mriedem) wrote :

> the API for adding/removing security is deprecated

That's not true. add/removeSecurityGroup APIs are not deprecated. The non-server resource related security group APIs were deprecated in newton:

https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/deprecate-api-proxies.html

And confusingly some other network related proxy APIs were deprecated in Pike like add/removeFloatingIP:

https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/deprecate-multinic-proxy-api.html

But add/removeSecurityGroup APIs are not officially deprecated.

Changed in nova:
status: In Progress → Triaged
importance: High → Low
assignee: Hongbin Lu (hongbin.lu) → nobody
Revision history for this message
Sven Kieske (s-kieske) wrote :

is this actually still a bug?
Because if it is, I don't understand why it's importance is deemed "Low"?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.