Ubuntu
lvm2 package

initramfs boot hang (waiting for termination ofwatershed vgscan vgchange proccesses) if logical volume contains volume group

Bug #1069579 reported by Andreas Greve
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
lvm2 (Ubuntu)
Confirmed
Undecided
Dimitri John Ledkov

Bug Description

Hello,

I have a problem like in

   https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/802626
   vgchange may deadlock in initramfs when VG present that's not used for rootfs

but I'm not sure.

If there exist a logical volume (/dev/mapper/nova--volumes-nova--disk /dev/dm-1) which self contains a volume group (XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686) which contains a logical volume (XSLocalEXT--5c88197f--6ec8--9a68--82f3--a475be0cd686-5c88197f--6ec8--9a68--82f3--a475be0cd686 /dev/dm-23) the system will drop to initramfs shell on booting.

The reason is the that the wateshed processes (lvm lvscan; lvm vgchange -a -y ) hangs until they are killed by timeout but at this point the boot process is already droped to initramfs shell. After the kill of the watershed processes all looks well , all logical volumes found and symlinks are constructed to /dev/dm-*. Leaving the initramfs shell is the only thing to do to continue the boot process.

To analyse the problem and as work around I have modified the initramfs-tools files scripts/function and scripts/local

in scripts/functions I add a function ls8panic

ls8panic() {
 if command -v chvt >/dev/null 2>&1; then
  chvt 1
 fi
        echo "ls8panic: entered reason: $@"
        ls -l /dev/mapper
        lvm pvs
        lvm lvs
        ps | grep -v grep | grep watershed
        if [ $? -eq 0 ]; then
          #watershed is running
          echo "lspanic: wateshed is running sleep 15 secs than do as handled and continue."
          sleep 15
         else
          #watershed is not running
          echo "ls8panic: watershed not running sleep 1 secs do a 'vgscan && vgschange -a y' than do as handled and continue."
          sleep 1
          lvm vgscan && lvm vgchange -a y
        fi
        ls -l /dev/mapper
        lvm pvs
        lvm lvs
        echo "ls8panic: leaved."
}

and in scripts/local I modified the function pre_mountroot() so that the routine ls8panic() was called 6 times before the function panic() will be called:

pre_mountroot()
{
        [...]
 done
        _ls8count="654321"
 # We've given up, but we'll let the user fix matters if they can
 while [ -z "${FSTYPE}" -a ! -e "${ROOT}" ]; do
        [...]
  esac
                if [ -n "${_ls8count}" ]; then
                  _ls8count="${_ls8count%?}"
                  ls8panic "ALERT!(${_ls8count}) ${ROOT} does not exist."
                else
  echo "Gave up waiting for root device. Common problems:"
                [...]
}

What you can see in the second example is that the watershed proccesses seams to hang after creating the symlinks for the volume which contains a volume group.

lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1

but after the termination of the watershed proccesses all expected symlinks in /dev/mapper created.

A second dubios thin are the messages

    udevd[467]: inotify_add_watch(6, /dev/dm-22, 10) failed: Invalid argument
    udevd[476]: inotify_add_watch(6, /dev/dm-23, 10) failed: Invalid argument

in the working boot and

    udevd[138]: inotify_add_watch(6, /dev/dm-24, 10) failed: No such file or directory

in the "hanging boot.

I dont know nothing about wath volumes/devices dm-22, dm-23 or dm-24 should be or where they come from.

I hope the information will help.

Thanks

Andreas

Version numbers of used packages:

initramfs-tools 0.99ubuntu13
udev 175-0ubuntu9.1
lvm2 2.02.66-4ubuntu7.1
watershed 6

Partion and pv configuration of used disks:
root@ls8vmw04:~# fdisk -l /dev/sda /dev/sdb

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00078fab

   Device Boot Start End Blocks Id System
/dev/sda1 * 63 835379 417658+ 83 Linux
/dev/sda4 835380 488392064 243778342+ 5 Extended
/dev/sda5 835443 488392064 243778311 8e Linux LVM

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x936e4c69

   Device Boot Start End Blocks Id System
/dev/sdb1 63 1012094 506016 83 Linux
/dev/sdb2 1012095 1953520064 976253985 5 Extended
/dev/sdb5 1012158 525306557 262147200 8e Linux LVM
/dev/sdb6 525308606 1049596605 262144000 8e Linux LVM
/dev/sdb7 1049598654 1469029053 209715200 8e Linux LVM

root@ls8vmw04:~# pvs
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0

interesting part of initramfs messages copied from console for a working boot:

[...]
[ 2.134160] usbhid: USB HID core driver
[ 2.153411] ata3.00: configured for UDMA/133
[ 2.155647] scsi 2:0:0:0: Direct-Access ATA SEAGATE ST32500N n/a PQ: 0 ANSI: 5
[ 2.159954] sd 2:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 2.164171] sd 2:0:0:0: [sda] Write Protect is off
[ 2.166332] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 2.166423] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.197409] sda: sda1 sda4 < sda5 >
[ 2.200043] sd 2:0:0:0: [sda] Attached SCSI disk
[ 2.368070] usb 2-5: new low-speed USB device number 3 using ohci_hcd
[ 2.599394] input: HID 046a:0011 as /devices/pci0000:00/0000:00:02.0/usb2/2-5/2-5:1.0/input/input4
[ 2.603987] generic-usb 0003:046A:0011.0003: input,hidraw2: USB HID v1.11 Keyboard [HID 046a:0011] on usb-0000:00:02.0-5/input0
[ 2.636905] forcedeth 0000:00:09.0: ifname eth3, PHY OUI 0x5043 @ 3, addr 00:1e:68:04:a6:fe
[ 2.640091] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2.643184] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[ 2.649015] ata4.00: ATA-8: WDC WD1003FBYX-01Y7B1, 01.01V02, max UDMA/133
[ 2.651139] ata4.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[ 2.661025] ata4.00: configured for UDMA/133
[ 2.663266] scsi 3:0:0:0: Direct-Access ATA WDC WD1003FBYX-0 01.0 PQ: 0 ANSI: 5
[ 2.667792] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 2.667812] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 2.674331] sd 3:0:0:0: [sdb] Write Protect is off
[ 2.676517] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.729049] sdb: sdb1 sdb2 < sdb5 sdb6 sdb7 >
[ 2.731724] sd 3:0:0:0: [sdb] Attached SCSI disk
Begin: Running /scripts/local-premount ... done.
[ 5.376959] EXT3-fs (dm-20): recovery required on readonly filesystem
[ 5.379085] EXT3-fs (dm-20): write access will be enabled during recovery
[ 5.886755] kjournald starting. Commit interval 5 seconds
[ 5.886926] EXT3-fs (dm-20): recovery complete
[ 5.906886] EXT3-fs (dm-20): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... udevd[467]: inotify_add_watch(6, /dev/dm-22, 10) failed: Invalid argument

udevd[476]: inotify_add_watch(6, /dev/dm-23, 10) failed: Invalid argument

done.
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
/dev/mapper/vg00-ls8vmw04_root: clean, 77091/1048576 files, 680568/4194304 blocks
/dev/sda1: clean, 298/102408 files, 183459/417656 blocks
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
[...]

interesting part of initramfs messages copied from console for a failed boot:

[ 2.100264] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 23 (level, low) -> IRQ 23
[ 2.105384] ata1.00: configured for UDMA/133
[ 2.107606] scsi 0:0:0:0: Direct-Access ATA SEAGATE ST32500N n/a PQ: 0 ANSI: 5
[ 2.111950] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 2.112009] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 2.112070] sd 0:0:0:0: [sda] Write Protect is off
[ 2.112107] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.149403] sda: sda1 sda4 < sda5 >
[ 2.151894] sd 0:0:0:0: [sda] Attached SCSI disk
[ 2.328074] usb 2-5: new low-speed USB device number 3 using ohci_hcd
[ 2.560399] input: HID 046a:0011 as /devices/pci0000:00/0000:00:02.0/usb2/2-5/2-5:1.0/input/input4
[ 2.564809] generic-usb 0003:046A:0011.0003: input,hidraw2: USB HID v1.11 Keyboard [HID 046a:0011] on usb-0000:00:02.0-5/input0
[ 2.592090] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2.601265] ata2.00: ATA-8: WDC WD1003FBYX-01Y7B1, 01.01V02, max UDMA/133
[ 2.603396] ata2.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[ 2.613287] ata2.00: configured for UDMA/133
[ 2.615531] scsi 1:0:0:0: Direct-Access ATA WDC WD1003FBYX-0 01.0 PQ: 0 ANSI: 5
[ 2.619858] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 2.619880] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 2.626404] sd 1:0:0:0: [sdb] Write Protect is off
[ 2.626785] scsi 2:0:0:0: CD-ROM TEAC DV-28E-V 1.AC PQ: 0 ANSI: 5
[ 2.628861] forcedeth 0000:00:09.0: ifname eth3, PHY OUI 0x5043 @ 3, addr 00:1e:68:04:a6:fe
[ 2.628864] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[ 2.635406] sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[ 2.635408] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 2.635713] sr 2:0:0:0: Attached scsi generic sg2 type 5
[ 2.648098] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.709774] sdb: sdb1 sdb2 < sdb5 sdb6 sdb7 >
[ 2.712393] sd 1:0:0:0: [sdb] Attached SCSI disk
ls8panic: entered reason: ALERT!(65432) /dev/mapper/vg00-ls8vmw04_root does not exist.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
  286 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  289 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  291 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  292 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  304 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
lspanic: wateshed is running sleep 15 secs than do as handled and continue.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
ls8panic: leaved.
ls8panic: entered reason: ALERT!(6543) /dev/mapper/vg00-ls8vmw04_root does not exist.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
  286 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  289 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  291 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  292 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  304 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
lspanic: wateshed is running sleep 15 secs than do as handled and continue.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
ls8panic: leaved.
ls8panic: entered reason: ALERT!(654) /dev/mapper/vg00-ls8vmw04_root does not exist.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
  286 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  289 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  291 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  292 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  304 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
lspanic: wateshed is running sleep 15 secs than do as handled and continue.
udevd[140]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [292]

udevd[142]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [289]

udevd[138]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [286]

udevd[140]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [292] terminated by signal 9 (Killed)

udevd[138]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [286] terminated by signal 9 (Killed)

udevd[142]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [289] terminated by signal 9 (Killed)

udevd[146]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [291]

udevd[146]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [291] terminated by signal 9 (Killed)

udevd[137]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [304]

udevd[137]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [304] terminated by signal 9 (Killed)

udevd[138]: inotify_add_watch(6, /dev/dm-24, 10) failed: No such file or directory

crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
lrwxrwxrwx 1 7 vg01-kilab065--hadoop -> ../dm-2
lrwxrwxrwx 1 7 vg01-ls8olc04--swap -> ../dm-3
lrwxrwxrwx 1 7 vg01-dom0_swap -> ../dm-4
lrwxrwxrwx 1 7 vg01-dom0_root_LTS8 -> ../dm-6
lrwxrwxrwx 1 7 vg01-kilab065--swap -> ../dm-5
lrwxrwxrwx 1 7 vg01-dom0_root_LTS12 -> ../dm-7
lrwxrwxrwx 1 7 vg01-ls8olc04--disk -> ../dm-9
lrwxrwxrwx 1 7 vg01-kilab065--disk -> ../dm-8
lrwxrwxrwx 1 8 vg00-dom0_root -> ../dm-10
lrwxrwxrwx 1 8 vg00-ls8pc09_root -> ../dm-12
lrwxrwxrwx 1 8 vg00-dom0_swap -> ../dm-11
lrwxrwxrwx 1 8 vg00-dom0_root_ULTS12 -> ../dm-14
lrwxrwxrwx 1 8 vg00-dom0_mysql -> ../dm-15
lrwxrwxrwx 1 8 vg00-dom0_postgresql -> ../dm-16
lrwxrwxrwx 1 8 vg00-ls8pc09_swap -> ../dm-13
lrwxrwxrwx 1 8 vg00-ls8vmw04_swap -> ../dm-17
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_back_base_ohne_openstack -> ../dm-19
lrwxrwxrwx 1 8 vg00-ls8vmw04_root -> ../dm-20
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack -> ../dm-21
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_old -> ../dm-18
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_with_xcp_partiell_openstack -> ../dm-22
lrwxrwxrwx 1 8 XSLocalEXT--5c88197f--6ec8--9a68--82f3--a475be0cd686-5c88197f--6ec8--9a68--82f3--a475be0cd686 -> ../dm-23
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi-a- 29.99g
  nova-disk nova-volumes -wi-ao 30.00g
  dom0_mysql vg00 -wi-a- 10.00g
  dom0_postgresql vg00 -wi-a- 10.00g
  dom0_root vg00 -wi-a- 16.00g
  dom0_root_ULTS12 vg00 -wi-a- 16.00g
  dom0_swap vg00 -wi-a- 32.00g
  ls8pc09_root vg00 -wi-a- 16.00g
  ls8pc09_swap vg00 -wi-a- 16.00g
  ls8vmw04_root vg00 -wi-a- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi-a- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi-a- 4.00g
  ls8vmw04_root_old vg00 -wi-a- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi-a- 4.00g
  ls8vmw04_swap vg00 -wi-a- 2.00g
  dom0_root_LTS12 vg01 -wi-a- 16.00g
  dom0_root_LTS8 vg01 -wi-a- 16.00g
  dom0_swap vg01 -wi-a- 4.00g
  kilab065-disk vg01 -wi-a- 16.00g
  kilab065-hadoop vg01 -wi-a- 100.00g
  kilab065-swap vg01 -wi-a- 8.00g
  ls8olc04-disk vg01 -wi-a- 16.00g
  ls8olc04-swap vg01 -wi-a- 16.00g
  lvswift vgswift -wi-a- 200.00g
ls8panic: leaved.
Begin: Running /scripts/local-premount ... done.
[ 78.273712] EXT3-fs (dm-20): recovery required on readonly filesystem
[ 78.275821] EXT3-fs (dm-20): write access will be enabled during recovery
[ 78.600757] kjournald starting. Commit interval 5 seconds
[ 78.600805] EXT3-fs (dm-20): recovery complete
[ 78.601250] EXT3-fs (dm-20): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
fsck from util-linux 2.20.1

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

While I do have nested LVMs myself, I never actually boot off nested LVM. E.g. the inner vg is presented as top-level for the VM.
Sounds like a very interesting bug.

Changed in lvm2 (Ubuntu):
assignee: nobody → Dmitrijs Ledkovs (xnox)
Revision history for this message
Andreas Greve (andreas-greve) wrote :

I don't now much about udev works. But it starts with the highest partition (sdb7 vg: vgswift lv: lvswift as dm-0) of the second disk which is configured for lvm and takes than the next lower partition (sdb6 vg: nova-volumes lv: nova-disk dm-1) and so on. Ok one reason for the working order could be that the first disk sda is an elder SATA-I ST32500N and the disk sdb is an newer SATA-III WD1003FBYX-0.

The "critical" inner vg/lv was created by the following commands.

lvcreate -L 30G -n nova-disk nova-volumes

SR="$( xe sr-create type=ext device-config:device=/dev/nova-volumes/nova-disk name-label=ext )"
POOL="$( xe pool-list --minimal )"
xe pool-param-set uuid=${POOL} default-SR=${SR}

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in lvm2 (Ubuntu):
status: New → Confirmed
Revision history for this message
Martijn Lievaart (j-launchpad-net-rtij-nl) wrote :

My problem seems exactly the same, except as far as I can see all symlinks are created. Interestingly, a vgscan -y -a at the command prompt returns immediately, without error.

I do have a rather elaborate setup, with lot's of LVs containing either filesystems or PVs (yes, PVs in LVs) and some snapshots of LVs.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.