multi-l3 adding foreign node to cluster causes traceback

Bug #1271043 reported by Andrew Woodward
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Ryan Moe

Bug Description

adding a node to a cluster with a admin IP that is foreign admin network internals returns 500 and ejects node from from the nodes list

2014-01-21 06:39:04.671 DEBUG [7ff73ebfd700] (logger) Request PUT /api/v1/nodes/
 from 127.0.0.1:45760 [{"pending_addition": true, "pending_roles": ["controller"
], "cluster_id": "1", "id": 4, "pending_deletion": false}]
2014-01-21 06:39:04.731 ERROR [7ff73ebfd700] (logger) Traceback (most recent cal
l last):
  File "/opt/nailgun/lib/python2.6/site-packages/web/application.py", line 239,
in process
    return self.handle()
  File "/opt/nailgun/lib/python2.6/site-packages/web/application.py", line 230,
in handle
    return self._delegate(fn, self.fvars, args)
  File "/opt/nailgun/lib/python2.6/site-packages/web/application.py", line 420,
in _delegate
    return handle_class(cls)
  File "/opt/nailgun/lib/python2.6/site-packages/web/application.py", line 396,
in handle_class
    return tocall(*args)
  File "<string>", line 2, in PUT
  File "/opt/nailgun/lib/python2.6/site-packages/nailgun/api/handlers/base.py",
line 55, in content_json
    data = func(*args, **kwargs)
  File "/opt/nailgun/lib/python2.6/site-packages/nailgun/api/handlers/node.py", line 438, in PUT
    network_manager.assign_networks_by_default(node)
  File "/opt/nailgun/lib/python2.6/site-packages/nailgun/network/neutron.py", line 127, in assign_networks_by_default
    ifaces = [node.admin_interface]
  File "/opt/nailgun/lib/python2.6/site-packages/nailgun/db/sqlalchemy/models/node.py", line 228, in admin_interface
    return self.interfaces[0]
IndexError: list index out of range

2014-01-21 06:39:04.732 ERROR [7ff73ebfd700] (logger) Response code '500 Internal Server Error' for PUT /api/v1/nodes/ from 127.0.0.1:45760

Andrew Woodward (xarses)
tags: added: nailgun
Andrew Woodward (xarses)
Changed in fuel:
milestone: none → 4.1
Revision history for this message
Andrew Woodward (xarses) wrote :

We found that the actual issue is that during the agent update, it would remove all of the interfaces from the host because no interface in the list was tagged as the admin interface.

To fix this, we will add the same fall-back method that the agent uses. If a suitable interface isn't found, we will match the mac that matched the system interface (the one that contacted fuel in the first place) and use that one.

Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/68262

Changed in fuel:
status: Triaged → In Progress
Changed in fuel:
assignee: Andrew Woodward (xarses) → Ryan Moe (rmoe)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/68262
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=752aa4f87b968ab3a4d208fc9db683fc12f8a466
Submitter: Jenkins
Branch: master

commit 752aa4f87b968ab3a4d208fc9db683fc12f8a466
Author: Andrew Woodward <email address hidden>
Date: Tue Jan 21 12:42:01 2014 -0800

    Add fallback test in __check_interfaces_correctness

    In some cases, it may be OK for there to be an admin interface
     that is outside the admin network. To support this case we we will
     test if the interface mac matches the mac that contacted us, if this
     is the case, we will use this interface.

    This prevents the NetworkManager from erroneously removing interfaces
     from an otherwise valid node.

    Resolves-bug: #1271043

    Change-Id: I04120d538e2efba6cb7a04cfbd7cb4852f2576c9

Changed in fuel:
status: In Progress → Fix Committed
Andrew Woodward (xarses)
Changed in fuel:
status: Fix Committed → Fix Released
summary: - adding foreign node to cluster causes traceback
+ multi-l3 adding foreign node to cluster causes traceback
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.