Primary controller role can be assigned to another controller during controllers scale up

imho the only viable option as to mark first controller added to the cluster as primary

There is corresponding bug in Fuel-Library that controllers scalability fails:
  https://bugs.launchpad.net/fuel/+bug/1390480
So probably this case was supported, we need retest this issue after bug 1390480 will be fixed

@Dima, you mean we need to introduce new role "primary-controller" as permanent, not as dynamic ? If so - I'm +1 on it.

Storing primary-controller potentially will introduce incosistent state in nailgun database, because "primary" resources can move between several nodes.

As for the bug, i am marking it Invalid, because i was able to successfully scale up controllers from 1 to 3

Discussed the issue with Deployment team, so, how it should be solved.

1. on new env, Nailgun makes elections who is primary, saves this information in the db (where to store this info)
2. change serializer logic to assign primary role on the node which has this info in db
3. when primary-controller is to be deleted (or there are no primary-controllers), Nailgun runs new elections and assign primary on new node, which is ready and is not offline.

https://bugs.launchpad.net/fuel/+bug/1396016 may be a duplicate of this one. If so, this bug should be nominated to 5.1.1 as well

patch for some reason isnt attached - https://review.openstack.org/#/c/136848/

Reviewed: https://review.openstack.org/136848
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=15106db883857ed982823421af03f6983ac82869
Submitter: Jenkins
Branch: master

commit 15106db883857ed982823421af03f6983ac82869
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 25 16:22:30 2014 +0200

    Persist primary roles in nailgun database

    Currently we have a problem, when new controller added
    with lower uid than already deployed controller.
    In this case primary-controller will be assigned to new one,
    but we already have existing controller which will conflict with
    new one.

    To overcome this issue primary role will be persisted in
    database.
    - Added primary attribute for pending_roles and roles tables
      by default primary will be False, it will be replaced only
      by special Cluster.set_primary_roles procedure
    - primary flag wont migrate from roles to pending roles,
      because such migration means that node was reseted to bootstrap
    - Primary roles used in logic of primary prefix assignment
    - For roles that should have primary - has_primary: true
      should be added in openstack.yaml

    TBD:
    - migration from pending roles to roles, currently it is covered
      by deployment logic, but maybe there is some cases where we want to do
      it explicitly

    Closes-Bug: 1390397
    Change-Id: I94b0014e9b4e16b2f8f0f6b9a092effb116a33ea