Incorrectly defined MAC of network card which is located in admin network

Bug #1271505 reported by Anastasia Palkina
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Aleksey Kasatkin

Bug Description

"build_id": "2014-01-22_01-17-41",
"ostf_sha": "10bfcd6373c03cf4e781b8da88246a277daa635c",
"build_number": "54",
"nailgun_sha": "6e15c822dd95b0c4730910f1af38bb67f4710129",
"fuelmain_sha": "192a6fdb9ac58e81786e6a0c315e0df91f839e97",
"astute_sha": "f967bf7d643c35e9c3bf8d8ad6cd6a13caa6e4ac",
"release": "4.1",
"fuellib_sha": "61f91c14dc8504c479b5e9459d93c0ec88d7ab24"

1. Create new environment (CentOS, simple mode)
2. Choose VLAN segmentation, Ceph for images
3. Add controller, compute, cinder, 2 ceph nodes
4. Untag Storage network and move it to other interface
5. Start deployment. It was successful but controller is gone away on UI

Revision history for this message
Anastasia Palkina (apalkina) wrote :
Revision history for this message
Evgeniy L (rustyrobot) wrote :

Additional info, nodes 1 and 7 went to offline.

I think it could be caused by this patch
https://github.com/stackforge/fuel-web/commit/807ecd7b3ced2ba5d0a5b8cc4c1eaceca9990f75

In db we have node with mac a2:61:15:7d:b4:43
But we don't have interface with this mac in interface list from node
http://paste.openstack.org/show/61669/

And then the same node sends data with mac 08:00:27:27:8A:55
http://paste.openstack.org/show/61670/
Nailgun's response is '403 Forbidden'

So, we have non-persistent alghorithm of admin interface definition.

Changed in fuel:
status: New → Triaged
Revision history for this message
Andrey Danin (gcon-monolake) wrote :

During the deployment while the L23network Puppet module is reconfiguring node's interfaces this instant configuration can occur:
  * Public interface (br-ex) has been configured,
  * admin interface (e.g. eth0) goes down,
  * Nailgun agent has collected info with broken IP and MAC fields and is trying to reach Master node
  * the packets go to the default route (br-ex) and come to the host node of our virtual test environment
  * the packets go through the NAT on the host node and return to the Master node via PXE net with the source IP of the host node
  * all information of the node has updated successfully.

After that has happened the admin interface brings up and next agent's message is fine, but Nailgun can't determine which node should be updated. The problem lines are https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/api/validators/node.py#L130-L145 It can be fixed with something like http://paste.openstack.org/show/61935/ with according fixes in validate_update method.

Also we must discuss the future of using MACs as unique node ids.

Revision history for this message
Evgeniy L (rustyrobot) wrote :

We have merged patch by Bogdan recently.
https://github.com/stackforge/fuel-web/commit/a641e5dcadd474d1dc6404b17adc8bd3ed66067c
Which provide us uuid of node, we should use this uuid to determine uniqueness of node instead of mac.

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Aleksey Kasatkin (alekseyk-ru)
status: Triaged → In Progress
Changed in fuel:
status: In Progress → Triaged
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/72700
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=de6c0a6e1e14919430323e1cc93cb13e3f1ec954
Submitter: Jenkins
Branch: master

commit de6c0a6e1e14919430323e1cc93cb13e3f1ec954
Author: Aleksey Kasatkin <email address hidden>
Date: Tue Feb 11 18:43:30 2014 +0200

    Check all node's MACs against MACs of all NICs of nodes

    All MACs of node being added/updated are checked against MACs of all NICs of each existing node.
    Tests are added.
    It's a quick fix. Usage of uuid to determine uniqueness of node will be considered.

    Closes-Bug: #1271505

    Change-Id: If9e8ac82ff9aa1c8359591821e70d4394a5a24ae

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #116
"build_id": "2014-02-13_13-09-40",
"mirantis": "no",
"build_number": "116",
"nailgun_sha": "0653f2e052cad099686de0e8259d4b861577599e",
"ostf_sha": "f86abe5544b5ffcf621e0c450bca15737c92361f",
"fuelmain_sha": "a4f314699aa97a274513a5ee182b13d25bf137ed",
"astute_sha": "7eed50fc30cec675fff7787c37fcf6da6dd518ee",
"release": "4.1",
"fuellib_sha": "dc45bb0b8520a91a66fd0493a87b4cf4bd33c32c"

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.