Juniper Openstack

add/delete controller provisioning stuck @ uninstall_config_started

Bug #1542463 reported by Sarath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
Thilak Raj
Juniper Openstack r3.0-fcs

Bug Description

when add/delete controller it gets into issue. if i only delete the controller it went fine.

Provisioning configs
###

{
    "server": [
        {
            "id": "a5d11e36",
            "ip_address": "10.87.143.84",
            "roles": [
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e14",
            "ip_address": "10.87.143.90",
            "roles": [
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e13",
            "ip_address": "10.87.143.92",
            "roles": [
                "config",
                "control",
                "collector",
                "webui",
                "database",
                "openstack"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e12",
            "ip_address": "10.87.143.94",
            "roles": [
                "config",
                "control",
                "collector",
                "webui",
                "database",
                "openstack"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e10",
            "ip_address": "10.87.143.98",
            "roles": [
                "compute"
            ],
            "status": "reimage_completed"
        },
        {
            "id": "a5d11e09",
            "ip_address": "10.87.143.100",
            "roles": [
                "compute"
            ],
            "status": "reimage_completed"
        }
    ]
}
root@Blaster-nsarath-004:~#

Provisioning status
###

root@Blaster-nsarath-004:~# server-manager status server
{
    "server": [
        {
            "id": "a5d11e36",
            "ip_address": "10.87.143.84",
            "mac_address": "00:E0:81:CA:59:11",
            "status": "uninstall_collector_completed"
        },
        {
            "id": "a5d11e14",
            "ip_address": "10.87.143.90",
            "mac_address": "00:E0:81:CA:5A:87",
            "status": "uninstall_collector_started"
        },
        {
            "id": "a5d11e13",
            "ip_address": "10.87.143.92",
            "mac_address": "00:E0:81:CA:55:13",
            "status": "config_started"
        },
        {
            "id": "a5d11e12",
            "ip_address": "10.87.143.94",
            "mac_address": "00:E0:81:C5:81:95",
            "status": "config_started"
        },
        {
            "id": "a5d11e10",
            "ip_address": "10.87.143.98",
            "mac_address": "00:E0:81:CA:59:59",
            "status": "provision_completed"
        },
        {
            "id": "a5d11e09",
            "ip_address": "10.87.143.100",
            "mac_address": "00:E0:81:CA:59:D3",
            "status": "provision_completed"
        }
    ]
}

target-node syslog
###

Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[heat-api]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[heat-api]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[heat-api-cfn]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[heat-api-cfn]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[openstack-dashboard]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[openstack-dashboard]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-api]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-api]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-novncproxy]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-novncproxy]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-scheduler]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-scheduler]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-objectstore]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-objectstore]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-consoleauth]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-consoleauth]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-conductor]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[nova-conductor]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[contrail-openstack-dashboard]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[contrail-openstack-dashboard]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[mysql]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[mysql]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[keystone]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[keystone]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[mysql-server-wsrep]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[mysql-server-wsrep]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[mysql-common]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Package[mysql-common]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Remove_mysql_log_files/Exec[remove_mysql_log_files]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Remove_mysql_log_files/Exec[remove_mysql_log_files]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Remove_mysql_log_files/Notify[executed remove_mysql_log_files]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Remove_mysql_log_files/Notify[executed remove_mysql_log_files]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Profile::Openstack_controller/Contrail::Uninstall_openstack/Exec[apt_auto_remove_openstack]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Profile::Openstack_controller/Contrail::Uninstall_openstack/Exec[apt_auto_remove_openstack]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Profile::Openstack_controller/Contrail::Uninstall_openstack/Contrail::Lib::Report_status[uninstall_openstack_completed]/Exec[contrail-status-uninstall_openstack_completed]) Dependency Exec[remove_mysql_root_user] has failures: true
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Profile::Openstack_controller/Contrail::Uninstall_openstack/Contrail::Lib::Report_status[uninstall_openstack_completed]/Exec[contrail-status-uninstall_openstack_completed]) Skipping because of failed dependencies
Feb 5 11:37:51 a5d11e14 puppet-agent[20805]: (/Stage[main]/Contrail::Profile::Config/Contrail::Uninstall_config/Contrail::Lib::Report_status[uninstall_config_started]/Exec[contrail-status-uninstall_config_started]/returns) executed successfully
Feb 5 11:38:03 a5d11e14 kernel: [169167.137976] init: haproxy main process (24460) terminated with status 1
Feb 5 11:38:03 a5d11e14 kernel: [169167.137992] init: haproxy main process ended, respawning
Feb 5 11:39:01 a5d11e14 CRON[26122]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Feb 5 11:39:10 a5d11e14 kernel: [169233.867023] init: haproxy main process (26434) terminated with status 1

root@a5d11e14:~#
root@a5d11e14:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0-2709 2709
contrail-config 3.0-2709 2709
contrail-config-openstack 3.0-2709 2709
contrail-control 3.0-2709 2709
contrail-dns 3.0-2709 2709
contrail-docs 3.0-2709 2709
contrail-f5 3.0-2709 2709
contrail-fabric-utils 3.0-2709 2709
contrail-heat 3.0-2709 2709
contrail-lib 3.0-2709 2709
contrail-nodemgr 3.0-2709 2709
contrail-nova-networkapi 3.0-2709 2709
contrail-openstack 3.0-2709 2709
contrail-openstack-analytics 3.0-2709 2709
contrail-openstack-config 3.0-2709 2709
contrail-openstack-control 3.0-2709 2709
contrail-openstack-dashboard 3.0-2709 2709
contrail-openstack-ha 3.0-2709 2709
contrail-setup 3.0-2709 2709
contrail-utils 3.0-2709 2709
ifmap-python-client 0.1-2 2709
ifmap-server 0.3.2-1contrail2 2709
neutron-plugin-contrail 3.0-2709 2709
nova-api 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-conductor 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-console 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-consoleauth 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-novncproxy 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-objectstore 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
nova-scheduler 1:2014.2.3-0ubuntu1~cloud0.3contrail2709
python-contrail 3.0-2709 2709
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail2709
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail2709

Revision history for this message
Sarath (nsarath) wrote :

Please find below the logs of Target node and SM

-bash-4.1$
-bash-4.1$ hostname
centos-build02
-bash-4.1$
-bash-4.1$ pwd
/users/nsarath/PR/PR-1542463
-bash-4.1$
-bash-4.1$ ls -l
total 1912800
-rwxrwxrwx 1 nsarath test 600483840 Feb 5 11:59 4-log.tar*
-rwxrwxrwx 1 nsarath test 1349007360 Feb 5 11:59 SMlog.tar*
-rwxrwxrwx 1 nsarath test 1515520 Feb 5 11:59 SMRunlog.tar*
-bash-4.1$

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/16964
Submitter: Thilak Raj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/16964
Committed: http://github.org/Juniper/contrail-puppet/commit/7ccfeaf524a5fb523c3717f9eff3baff0eea0583
Submitter: Zuul
Branch: master

commit 7ccfeaf524a5fb523c3717f9eff3baff0eea0583
Author: tsurendra <email address hidden>
Date: Sun Feb 7 11:39:11 2016 -0800

Closes-Bug: #1542463
When cmon_user is not present,removal of cmon_user fails.

Fix:
made unless clause to check for existence of cmon/root user.
Also changed galera_setup to use cluster_statr to detect existence of
existing galera_cluster.

Testing:
Verfied by deleting and adding a cluster at the first.

Change-Id: I906ef4c2e70ce6a90c26c03275885158b63033b5

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.