Juniper Openstack

SM:centos:R3.1build36: Provision does not complete with compute_started state

Bug #1636916 reported by sundarkh on 2016-10-26

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.1	Fix Committed	Critical	kamlesh parmar	Juniper Openstack r3.1.1.0
R3.2	Fix Committed	Critical	kamlesh parmar	Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"
Trunk	Fix Committed	Critical	kamlesh parmar	Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

1) Install Ubuntu ServerManager on the SM Machine
2) Reimage the target with centos71; reimage completes successfully
3) Initiate Provision using contrail package ; Provision stops at compute_started

Logs

Oct 26 05:12:39 nodeh8 puppet-agent[761]: Adding /usr/sbin/shutdown -r +0 "Puppet is rebooting the computer" </dev/null >/dev/null 2>&1 & to ruby's at_exit handler
Oct 26 05:12:39 nodeh8 puppet-agent[761]: (/Stage[compute]/Contrail::Compute::Config/Reboot[compute]) Triggered 'refresh' from 1 events
Oct 26 05:12:40 nodeh8 puppet-agent[761]: Finished catalog run in 43.63 seconds
Oct 26 05:12:41 nodeh8 systemd: Started Delayed Shutdown Service.
Oct 26 05:12:41 nodeh8 systemd: Starting Delayed Shutdown Service...
Oct 26 05:12:41 nodeh8 systemd-shutdownd: Shutting down at Wed 2016-10-26 05:12:41 PDT (reboot)...
Oct 26 05:12:41 nodeh8 systemd-shutdownd: Creating /run/nologin, blocking further logins...
Oct 26 05:12:41 nodeh8 systemd: Stopping Session c1 of user zookeeper.
Oct 26 05:12:41 nodeh8 systemd: Stopping SYSV: Starts and stops Cassandra...
Oct 26 05:12:41 nodeh8 systemd: Stopping Session c3 of user rabbitmq.
Oct 26 05:12:41 nodeh8 systemd: Deactivating swap /dev/mapper/centos-swap...
Oct 26 05:12:41 nodeh8 systemd: Stopping Session c2 of user cassandra.
Oct 26 05:12:41 nodeh8 systemd: Stopping Session 8 of user root.
Oct 26 05:12:41 nodeh8 systemd: Stopped Dump dmesg to /var/log/dmesg.
Oct 26 05:12:41 nodeh8 systemd: Stopping Dump dmesg to /var/log/dmesg...
Oct 26 05:12:41 nodeh8 systemd: Stopped target Timers.
Oct 26 05:12:41 nodeh8 systemd: Stopping Timers.
Oct 26 05:12:41 nodeh8 systemd: Stopped daily update of the root trust anchor for DNSSEC.
Oct 26 05:12:41 nodeh8 systemd: Stopping daily update of the root trust anchor for DNSSEC.
Oct 26 05:12:41 nodeh8 systemd: Stopped Stop Read-Ahead Data Collection 10s After Completed Startup.
Oct 26 05:12:41 nodeh8 systemd: Stopping Stop Read-Ahead Data Collection 10s After Completed Startup.
Oct 26 05:12:41 nodeh8 systemd: Stopped Daily Cleanup of Temporary Directories.
Oct 26 05:12:41 nodeh8 systemd: Stopping Daily Cleanup of Temporary Directories.
Oct 26 05:12:41 nodeh8 systemd: Stopped target Multi-User System.
Oct 26 05:12:41 nodeh8 systemd: Stopping Multi-User System.
Oct 26 05:12:41 nodeh8 systemd: Stopping Openstack Heat CFN-compatible API Service...
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.304 23836 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.305 23836 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.305 23836 DEBUG oslo_concurrency.lockutils [req-b818e6d0-e9bc-4538-b32d-b47eba375004 - - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.306 23836 DEBUG oslo_concurrency.lockutils [req-b818e6d0-e9bc-4538-b32d-b47eba375004 - - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.306 23836 INFO oslo_service.service [req-b818e6d0-e9bc-4538-b32d-b47eba375004 - - - - -] Caught SIGTERM, exiting
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.307 23836 WARNING oslo_messaging.server [req-b818e6d0-e9bc-4538-b32d-b47eba375004 - - - - -] start/stop/wait must be called in the same thread
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.307 23836 DEBUG oslo_concurrency.lockutils [req-b818e6d0-e9bc-4538-b32d-b47eba375004 - - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:198
Oct 26 05:12:41 nodeh8 cinder-scheduler: 2016-10-26 05:12:41.307 23836 DEBUG oslo_concurrency.lockutils [req-b818e6d0-e9bc-4538-b32d-b47eba375004 - - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:211
Oct 26 05:12:41 nodeh8 rabbitmq-server: Event crashed log handler:
Oct 26 05:12:41 nodeh8 rabbitmq-server: {info_msg,<0.69.0>,
Oct 26 05:12:41 nodeh8 rabbitmq-server: {<0.219.0>,"closing AMQP connection ~p (~s)~n",
Oct 26 05:12:41 nodeh8 rabbitmq-server: [<0.308.0>,"10.204.217.112:57487 -> 10.204.217.112:5672"]}}
Oct 26 05:12:41 nodeh8 rabbitmq-server: function_clause
Oct 26 05:12:41 nodeh8 systemd: Stopping OpenStack Cinder Scheduler Server...
Oct 26 05:12:43 nodeh8 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="17018" x-info="http://www.rsyslog.com"] exiting on signal 15.

ct 26 05:12:43 nodeh8 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="17018" x-info="http://www.rsyslog.com"] exiting on signal 15.
Oct 26 05:17:06 localhost rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="702" x-info="http://www.rsyslog.com"] start
Oct 26 05:16:50 localhost journal: Runtime journal is using 8.0M (max allowed 3.1G, trying to leave 4.0G free of 31.3G available → current limit 3.1G).
Oct 26 05:16:50 localhost systemd-journald[113]: Received SIGTERM from PID 1 (systemd).
Oct 26 05:16:50 localhost kernel: SELinux: Disabled at runtime.
Oct 26 05:16:50 localhost kernel: type=1404 audit(1477459008.044:2): selinux=0 auid=4294967295 ses=4294967295
Oct 26 05:16:50 localhost systemd[1]: RTC configured in localtime, applying delta of -420 minutes to system time.
Oct 26 05:16:50 localhost kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Oct 26 05:16:50 localhost systemd[1]: Inserted module 'ip_tables'
Oct 26 05:16:50 localhost systemd-sysv-generator[502]: Overwriting existing symlink /run/systemd/generator.late/cassandra.service with real service
Oct 26 05:16:50 localhost systemd[1]: Configuration file /usr/lib/systemd/system/openstack-nova-api.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Oct 26 05:16:50 localhost systemd[1]: Configuration file /etc/systemd/system/rabbitmq-server.service.d/rabbitmq-restart-contrail.conf is marked executable. Please remove executable permission bits. Proceeding anyway.
Oct 26 05:16:50 localhost systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Oct 26 05:16:50 localhost systemd[1]: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Oct 26 05:16:50 localhost systemd[1]: Stopped Switch Root.
Oct 26 05:16:50 localhost systemd[1]: Cannot find unit for notify message of PID 113.
Oct 26 05:16:50 localhost systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Oct 26 05:16:50 localhost systemd[1]: Created slice system-getty.slice.
Oct 26 05:16:50 localhost systemd[1]: Starting system-getty.slice.
Oct 26 05:16:50 localhost systemd[1]: Starting Create list of required static device nodes for the current kernel...
Oct 26 05:16:50 localhost systemd[1]: Stopped target Switch Root.
Oct 26 05:16:50 localhost systemd[1]: Stopping Switch Root.
Oct 26 05:16:50 localhost systemd[1]: Listening on udev Kernel Socket.
Oct 26 05:16:50 localhost systemd[1]: Starting udev Kernel Socket.
Oct 26 05:16:50 localhost systemd[1]: Listening on udev Control Socket.
Oct 26 05:16:50 localhost systemd[1]: Starting udev Control Socket.
Oct 26 05:16:50 localhost systemd[1]: Starting Journal Service...
Oct 26 05:16:50 localhost systemd[1]: Mounting Huge Pages File System...
Oct 26 05:16:50 localhost systemd[1]:

After reimage

[root@nodeh8 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.204.217.254 0.0.0.0 UG 0 0 0 enp3s0f0
10.204.217.0 0.0.0.0 255.255.255.0 U 0 0 0 enp3s0f0
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 enp3s0f0
[root@nodeh8 ~]# ifconfig -a
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

[root@nodeh8 ~]# cat /etc/hostname
localhost.localdomain
[root@nodeh8 ~]#

Tags:

sundarkh (sundar-kh) on 2016-10-26

Changed in juniperopenstack:
milestone:	none → r3.1.1.0
importance:	Undecided → Critical
assignee:	nobody → kamlesh parmar (kparmar)

Revision history for this message

kamlesh parmar (kparmar) wrote on 2016-10-27:

Not sure what is exposing this. But same kickstart worked before. There is no change in code SM or puppet to explain this change in behavior. Anyways, setting hostname explictly in kickstart will be good thing to do.
I have patched your SM with change in kickstart to set hostname. Provisioning is going fine now. Please test centos 7.1 and 7.2 and let me know, i'll check it in.

the change is made in the /var/www/html/kickstarts/contrail-centos.ks

rm -rf /etc/hostname
echo "$system_name" >> /etc/hostname

sundarkh (sundar-kh) on 2016-11-02

tags:

added: blocker

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-11-02:

adding the fix mentioned #1 , works

after adding
rm -rf /etc/hostname
echo "$system_name" >> /etc/hostname

Provison got completed

Revision history for this message

Abhay Joshi (abhayj) wrote on 2016-11-02:

Since there is a workaround, removing blocker tag.

tags:

removed: blocker

Jeba Paulaiyan (jebap) on 2016-11-02

Changed in juniperopenstack:
milestone:	r3.1.1.0 → r4.0

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-03: [Review update] R3.2

Review in progress for https://review.opencontrail.org/25650
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-03: [Review update] master

Review in progress for https://review.opencontrail.org/25651
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-03: [Review update] R3.1

Review in progress for https://review.opencontrail.org/25662
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-03: A change has been merged

#10

Reviewed: https://review.opencontrail.org/25650
Committed: http://github.org/Juniper/contrail-server-manager/commit/70522b7a677b3d02ff7c62acff1e9949fbb8df30
Submitter: Zuul
Branch: R3.2

commit 70522b7a677b3d02ff7c62acff1e9949fbb8df30
Author: Kamlesh Parmar <email address hidden>
Date: Thu Nov 3 10:30:00 2016 -0700

Closes-Bug: #1636916
Add hostname to /etc/hostname for hostname persistence across reboot.

Change-Id: If4382ce7505c03afc72cb1df3a7fbace4f8f6fe9

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-03:

#11

Reviewed: https://review.opencontrail.org/25651
Committed: http://github.org/Juniper/contrail-server-manager/commit/1257b4e93c2b88980b5bee669f73d7442b732997
Submitter: Zuul
Branch: master

commit 1257b4e93c2b88980b5bee669f73d7442b732997
Author: Kamlesh Parmar <email address hidden>
Date: Thu Nov 3 10:30:00 2016 -0700

Closes-Bug: #1636916
Add hostname to /etc/hostname for hostname persistence across reboot.

Change-Id: If4382ce7505c03afc72cb1df3a7fbace4f8f6fe9

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-04:

#12

Reviewed: https://review.opencontrail.org/25662
Committed: http://github.org/Juniper/contrail-server-manager/commit/e07e313d899a20d459fd5c5e57132a643d891bb2
Submitter: Zuul
Branch: R3.1

commit e07e313d899a20d459fd5c5e57132a643d891bb2
Author: Kamlesh Parmar <email address hidden>
Date: Thu Nov 3 10:30:00 2016 -0700

Closes-Bug: #1636916
Add hostname to /etc/hostname for hostname persistence across reboot.

Change-Id: If4382ce7505c03afc72cb1df3a7fbace4f8f6fe9

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.