Comment 0 for bug 1584044

Revision history for this message
Andrey Volochay (wallavv) wrote :

Detailed bug description:
 During upgrade MOS6.0 to MOS8.0 via Octane(stable/8.0), I have faced with following bug.

 By doing second controller upgrade via Octane (octane upgrade-node <target_node_id> <seed_env_id>), Nailgun returns an error with code 500

2016-05-18 13:01:00.188 ERROR [7f39799c4880] (handlers) MOVE NODE FAILED
Traceback (most recent call last):
 File "/usr/lib/python2.7/site-packages/nailgun/extensions/cluster_upgrade/handlers.py", line 93, in POST
   upgrade.UpgradeHelper.assign_node_to_cluster(node, cluster)
 File "/usr/lib/python2.7/site-packages/nailgun/extensions/cluster_upgrade/upgrade.py", line 191, in assign_node_to_cluster
   node, netgroups_id_mapping)
 File "/usr/lib/python2.7/site-packages/nailgun/extensions/cluster_upgrade/objects/adapters.py", line 140, in set_bond_assignment_netgroups_ids
   node.node, mapping)
 File "/usr/lib/python2.7/site-packages/nailgun/network/manager.py", line 203, in set_bond_assignment_netgroups_ids
   netgroups_id_mapping[bond_assignment.network_id]
KeyError: 7

Steps to reproduce:
 1) Have MOS6.0 environment
 2) Upgrade master node (fuel 6.0) to 6.1 and then to 7.0
 3) Backup environment settings, following documented process
 4) Prepare new master node (fuel 8.0)
 5) Restore the backup from third step
 5) Upgrade restored environment via Octane
 6) Upgrade primary controller via Octane
 7) Upgrade DB via Octane
 8) Upgrade Control Plane via Octane
 9) Upgrade second and third controllers

Expected results:
 Provisioning of second and third controllers is started

Actual result:
 Nailgun returns an error 500 which in turn breaks node upgrade.

Reproducibility:
 Use network templates ans network bonding.

Workaround:
 Comment out following code in nailgin/cluster_upgrade/upgrade.py, when first controller is moved and then restart Nailgun:

   orig_manager.set_bond_assignment_netgroups_ids(
      node, netgroups_id_mapping)

Description of the environment:
 Versions of components: MOS8.0, Octane(stable/8.0)
 Network model: Environment uses network bonding
 Related projects installed: Octane

Additional information:
 It seems like Nailgun falls at "set_bond_assignment_netgroups_ids" method because the method performs an action, which has been done before.

 Presumably it happens at assign_node_to_cluster method.
1) assign_node_to_cluster
2) node.update_cluster_assignment(seed_cluster)
3) update_cluster_assignment
4) set_network_template
5) apply_network_template #there is a condition IF, in case when network_template is not absent
6) assign_networks_by_template #there are mentioned actions