Fuel for OpenStack

Ubuntu Provisioning Failure

Series 4.1.x
Bug #1319995

Bug #1319995 reported by Dmitry Ilyin on 2014-05-15

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Vladimir Kozhukalov	Fuel for OpenStack 5.0
	4.1.x	Fix Committed	High	Fuel Library (Deprecated)	Fuel for OpenStack 4.1.1

Bug Description

Ubuntu provisioning fails most of the time.

ISO: 206 (started from 19?)

Ubuntu HA Neutron-GRE
Ceph for both CInder and Glance

2 x Controller + Ceph OSD
1 x Compute

2014-05-15 19:43:09 NOTICE
Unexpected error; command not executed: 'hwclock --systohc'
2014-05-15 19:43:09 NOTICE
Unexpected error; command not executed: 'sed -i s/msg_age = Time.now.utc.to_i - msgtime/msg_age = 0 #Time.now.utc.to_i - msgtime/g /usr/lib/ruby/1.8/mcollective/message.rb'
2014-05-15 19:43:09 NOTICE
Unexpected error; command not executed: 'sh -c echo ${0} | base64 --decode | gunzip -c | sh - H4sIAMwXdVMC/5WTX2vCMBTF3/spAn0b2CZV6VZQEHVsD9uD+C4xvbXBmkia6tdf0jj/lUpLoYHcc84vvTf1fR8xKTK+qxQXO1SlcEKqKqBEvu97wHKJpigEzUJbCutSkIYxHhxBlbzUIPRAgA7qivczW8+/ZovFajKOkvEowThJcbIlyTBD37/r5epzNl9OQOcRCgu+daFnxTVsTMimNYTGyZAkH6PHENIvJH63b0weQ/CLEM9/6g8XGlRG2X1/SlkpBq5JxnqWah/edKZZb+i/h82yd9inXLWbHYJWWqLCjGL6AsMzlu0GRuU83O5bEzdqs8rjlrJ9t4jL4LtLHayea41Lc3bs4reOzrCb+IojvXGkD4484XBvHO6Dw80LV7
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'update-grub'
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'grub-install /dev/vda'
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'grub-mkdevicemap'
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'grub-mkconfig'
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'sed -i -e s/.*GRUB_TERMINAL.*/GRUB_TERMINAL=console/g -e s/.*GRUB_GFXMODE.*/#GRUB_GFXMODE=640x480/g -e s/.*GRUB_CMDLINE_LINUX.*/GRUB_CMDLINE_LINUX=" console=ttyS0,9600 console=tty0 rootdelay=90 nomodeset "/g /etc/default/grub'
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'apt-get install -y grub-pc'
2014-05-15 19:43:07 NOTICE
Unexpected error; command not executed: 'sgdisk --typecode=6:4fbd7e29-9d25-41b8-afd0-062c0ceff05d /dev/vda'
2014-05-15 19:42:38 NOTICE

2014-05-15 19:42:38 NOTICE
Unable to locate package language-support-en
2014-05-15 19:42:38 NOTICE
:
2014-05-15 19:42:38 NOTICE
E
2014-05-15 19:42:38 NOTICE

2014-05-15 19:42:38 NOTICE
Reading state information...

On the node's log:

May 15 19:38:51 main-menu[413]: DEBUG: resolver (libc6-udeb): package doesn't exist (ignored)
May 15 19:38:51 main-menu[413]: DEBUG: resolver (libnewt0.52): package doesn't exist (ignored)
May 15 19:38:51 main-menu[413]: INFO: Menu item 'network-preseed' selected
May 15 19:38:55 preseed: successfully loaded preseed file from http://10.20.0.2/cblr/svc/op/ks/system/node-1
May 15 19:38:55 preseed: running preseed command preseed/early_command: /bin/mkdir /.ssh ; /bin/echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx1gd5aphXBfaRaz8bwQLY1EfDkLldMPBJRxvwf1BapDNnE5olKjZhJS8k5mjBHlu8sEf6laof2ixoW95480cnPwGaJfTrEGgEIA2/GRTheJ8vFsD4JfipXuWUISIWVMKu2nYKXcCpWLdGZQUV9uTcJn9HDeXSLDG0/hCsub41ruOW9hbOx0/lxY5NNb1ybi/MfVTZUZjgSdPM7M9+o7MXKj29ZNyuDo+ZqhpkILTnU3ru3vmid0C2UgLiJvIs7F88eeO71ssjuYJOpqAh2U+rPUmt0K1StUlvLyYOR/+ZzwMcgQNCgCgLLnue8axj7it7kCcJnExg8yJhWKCG7EKjQ== <email address hidden>" >
May 15 19:38:55 log-output: sed: /etc/ssh/sshd_config: No such file or directory
May 15 19:38:55 log-output: killall: sshd: no process killed

SSH to node is working. Key is present. sshd_config is present too.

# cat /etc/ssh/sshd_config
# Installer generated configuration file

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
LoginGraceTime 600
PermitRootLogin yes
StrictModes yes
PermitEmptyPasswords no
PasswordAuthentication yes
UsePrivilegeSeparation no

X11Forwarding no
PrintMotd yes
TCPKeepAlive yes

Did this command fail?
./iso/ks.template:sed -i -e "/^\s*GSSAPICleanupCredentials yes/d" -e "/^\s*GSSAPIAuthentication yes/d" /etc/ssh/sshd_config
No. Looks like it was successful... What happened then?

Revision history for this message

Dmitry Ilyin (idv1985) wrote on 2014-05-15:

bug.zip Edit (774.1 KiB, application/zip)

Mike Scherbakov (mihgen) on 2014-05-15

Changed in fuel:
assignee:	nobody → Fuel Library Team (fuel-library)

Revision history for this message

Dmitry Ilyin (idv1985) wrote on 2014-05-16:

Ubuntu HA without Ceph - provisioning and deployment went normally.

Revision history for this message

Dmitry Ilyin (idv1985) wrote on 2014-05-16:

Deployment with external ceph osd nodes went successful too. This pretty much narrows the problem down to situations when ceph osd is placed on controller.

Revision history for this message

Dmitry Borodaenko (angdraug) wrote on 2014-05-16:

Data point: Ubuntu with ceph-osd on controller works for me with ISO #192.

Revision history for this message

Dmitry Ilyin (idv1985) wrote on 2014-05-20:

--- pmanager-new.py 2014-05-19 19:13:44.630723562 +0000
+++ pmanager-old.py 2014-05-18 21:24:28.000000000 +0000
@@ -104,7 +104,7 @@

     def _gettabfsoptions(self, vol):
         if self._gettabfstype(vol) == "xfs":
- return "-f"
+ return "-f -s size=512"
         return ""

     def _getfstype(self, vol):
@@ -277,13 +277,14 @@
                               "--typecode={0}:{1} {2}".format(
                                   pcount, part["partition_guid"],
                                   self._disk_dev(disk)))
- if size > 0 and size <= 16777216 and part["mount"] != "none":
+ if size > 0 and size <= 16777216 and tabfstype != "xfs":
                     self.kick("partition {0} "
                               "--onpart={2}"
                               "{3}{4}".format(part["mount"], size,
                                            self._disk_dev(disk),
                                            self._pseparator(disk["id"]),
                                            pcount))
+
                 else:
                     if part["mount"] != "swap" and tabfstype != "none":
                         disk_label = self._getlabel(part.get('disk_label'))
@@ -410,7 +411,7 @@
                 tabfstype = self._gettabfstype(lv)
                 tabfsoptions = self._gettabfsoptions(lv)

- if size > 0 and size <= 16777216:
+ if size > 0 and size <= 16777216 and tabfstype != "xfs":
                     self.kick("logvol {0} --vgname={1} --size={2} "
                               "--name={3} {4}".format(
                                   lv["mount"], vg["id"], size,
@@ -587,7 +588,7 @@

     def _fsoptions(self, fstype):
         if fstype == "xfs":
- return "-f"
+ return "-f -s size=512"
         return ""

     def _umount_target(self):
@@ -616,18 +617,14 @@
         """This method puts script which splits
         long line and sends them to logger
         #!/bin/sh
-
- read string
         chunk=80
-
- iterations=`expr ${#string} / $chunk + 1`
- i=0
- while [ $i -le $(( iterations - 1)) ]; do
- start_sym=$(( $i * ${chunk} + 1 ))
- end_sym=$(( $(( $i + 1 )) * ${chunk}))
- echo $string | cut -c ${start_sym}-${end_sym} | logger
- i=$(( i + 1 ))
- done
+ while read string; do
+ iterations=`expr ${#string} / $chunk + 1`; i=0;
+ while [ $i -le $(( iterations - 1)) ]; do
+ start_sym=$(( $i * ${chunk} + 1 ))
+ end_sym=$(( $(( $i + 1 )) * ${chunk}))
+ echo $string | cut -c ${start_sym}-${end_sym} | logger
+ i=$(( i + 1 )); done; done;
         """
         return (
             """echo '#!/bin/sh' > /tmp/long_logger.sh;""",

This is diff between the working version of pmanager.py and the current one.

--- pmanager-new.py	2014-05-19 19:13:44.630723562 +0000
+++ pmanager-old.py	2014-05-18 21:24:28.000000000 +0000
@@ -104,7 +104,7 @@
 
     def _gettabfsoptions(self, vol):
         if self._gettabfstype(vol) == "xfs":
-            return "-f"
+            return "-f -s size=512"
         return ""
 
     def _getfstype(self, vol):
@@ -277,13 +277,14 @@
                               "--typecode={0}:{1} {2}".format(
                                   pcount, part["partition_guid"],
                                   self._disk_dev(disk)))
-                if size > 0 and size <= 16777216 and part["mount"] != "none":
+                if size > 0 and size <= 16777216 and tabfstype != "xfs":
                     self.kick("partition {0} "
                               "--onpart={2}"
                               "{3}{4}".format(part["mount"], size,
                                            self._disk_dev(disk),
                                            self._pseparator(disk["id"]),
                                            pcount))
+
                 else:
                     if part["mount"] != "swap" and tabfstype != "none":
                         disk_label = self._getlabel(part.get('disk_label'))
@@ -410,7 +411,7 @@
                 tabfstype = self._gettabfstype(lv)
                 tabfsoptions = self._gettabfsoptions(lv)
 
-                if size > 0 and size <= 16777216:
+                if size > 0 and size <= 16777216 and tabfstype != "xfs":
                     self.kick("logvol {0} --vgname={1} --size={2} "
                               "--name={3} {4}".format(
                                   lv["mount"], vg["id"], size,
@@ -587,7 +588,7 @@
 
     def _fsoptions(self, fstype):
         if fstype == "xfs":
-            return "-f"
+            return "-f -s size=512"
         return ""
 
     def _umount_target(self):
@@ -616,18 +617,14 @@
         """This method puts script which splits
         long line and sends them to logger
         #!/bin/sh
-
-        read string
         chunk=80
-
-        iterations=`expr ${#string} / $chunk + 1`
-        i=0
-        while [ $i -le $(( iterations - 1)) ]; do
-            start_sym=$(( $i * ${chunk} + 1 ))
-            end_sym=$(( $(( $i + 1 )) * ${chunk}))
-            echo $string | cut -c ${start_sym}-${end_sym} | logger
-            i=$(( i + 1 ))
-        done
+        while read string; do
+            iterations=`expr ${#string} / $chunk + 1`; i=0;
+            while [ $i -le $(( iterations - 1)) ]; do
+                start_sym=$(( $i * ${chunk} + 1 ))
+                end_sym=$(( $(( $i + 1 )) * ${chunk}))
+                echo $string | cut -c ${start_sym}-${end_sym} | logger
+                i=$(( i + 1 )); done; done;
         """
         return (
             """echo '#!/bin/sh' > /tmp/long_logger.sh;""",

This is diff between the working version of pmanager.py and the current one.

Changed in fuel:
status:	New → Triaged

Mike Scherbakov (mihgen) on 2014-05-20

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Dmitry Ilyin (idv1985)

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-05-20:

Moved this to High, as it fails only in very certain cases.

Changed in fuel:
importance:	Critical → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-21: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/94619

Changed in fuel:
status:	Triaged → In Progress

OpenStack Infra (hudson-openstack) on 2014-05-21

Changed in fuel:
assignee:	Dmitry Ilyin (idv1985) → Vladimir Kozhukalov (kozhukalov)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-05-21: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/94619
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=347646bd46052e1db938b0c4f8464eab44e247a1
Submitter: Jenkins
Branch: master

commit 347646bd46052e1db938b0c4f8464eab44e247a1
Author: Dmitry Ilyin <email address hidden>
Date: Wed May 21 19:51:27 2014 +0400

Fix ubuntu provisioning

    This patch fixes partitioning counting. We use kind of
    '$(readlink -f $(ls /dev/disk/by-path/foo) )' as a
    key for a partitioning counter, but in that particular place
    it was like /dev/disk/by-path/foo.

Change-Id: Iafbfcda96d0904ff87781df4d3b04d9293656d04
Closes-Bug: 1319995