mkfs.xfs: cannot open /dev/sdb: Device or resource busy

Bug #1513204 reported by Ryan Beisner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Swift Storage Charm
Invalid
Undecided
Unassigned
swift-storage (Juju Charms Collection)
Invalid
Undecided
Unassigned

Bug Description

The mkfs.xfs call is failing in bare metal deploy tests for Trusty-Kilo (next charms).

http://paste.ubuntu.com/13105318/

2015-11-04 20:06:24 INFO install mkfs.xfs: cannot open /dev/sdb: Device or resource busy
2015-11-04 20:06:24 INFO install Traceback (most recent call last):
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/install.real", line 175, in <module>
2015-11-04 20:06:24 INFO install main()
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/install.real", line 167, in main
2015-11-04 20:06:24 INFO install hooks.execute(sys.argv)
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
2015-11-04 20:06:24 INFO install self._hooks[hook_name]()
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/install.real", line 70, in install
2015-11-04 20:06:24 INFO install setup_storage()
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/lib/swift_storage_utils.py", line 252, in setup_storage
2015-11-04 20:06:24 INFO install mkfs_xfs(dev)
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/lib/swift_storage_utils.py", line 244, in mkfs_xfs
2015-11-04 20:06:24 INFO install check_call(cmd)
2015-11-04 20:06:24 INFO install File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
2015-11-04 20:06:24 INFO install raise CalledProcessError(retcode, cmd)
2015-11-04 20:06:24 INFO install subprocess.CalledProcessError: Command '['mkfs.xfs', '-f', '-i', 'size=1024', u'/dev/sdb']' returned non-zero exit status 1
2015-11-04 20:06:24 INFO juju.worker.uniter.context context.go:579 handling reboot
2015-11-04 20:06:24 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

Related branches

Revision history for this message
Ryan Beisner (1chb1n) wrote :

After waiting a while, I was able to manually run the same mkfs command on the unit:

http://paste.ubuntu.com/13105459/

ubuntu@international-steam:~$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
└─sda1 8:1 0 465.8G 0 part /
sdb 8:16 0 465.8G 0 disk
sr0 11:0 1 1024M 0 rom

...

ubuntu@international-steam:~$ sudo parted -l
Model: ATA ST9500620NS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 1049kB 500GB 500GB primary ext4 boot

Error: /dev/sdb: unrecognised disk label

...

ubuntu@international-steam:~$ sudo mkfs.xfs -f -i size=1024 /dev/sdb
meta-data=/dev/sdb isize=1024 agcount=4, agsize=30524162 blks
         = sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=122096646, imaxpct=25
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=59617, version=2
         = sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
ubuntu@international-steam:~$ echo $?
0

...

ubuntu@international-steam:~$ sudo parted -l
Model: ATA ST9500620NS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 1049kB 500GB 500GB primary ext4 boot

Model: ATA ST9500620NS (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number Start End Size File system Flags
 1 0.00B 500GB 500GB xfs

Revision history for this message
James Page (james-page) wrote :

Possibly the addition of a udevadm settle call may avoid this race.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

@jamespage

This is odd. I added udevadm settle like so, so that it is called before the storage operations:
lp:~openstack-charmers/charms/trusty/swift-storage/next

The udevadm command itself errored.

When I went back into the unit, I was able to do the udevadm commands and such without error.

I've not dug deeper than that. I'll hold on this, pending your input.

FWIW, when I arrived at the unit, a previous deployment's ceph gpt was still present on /dev/sdb:

root@decisive-punishment:/var/log# sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
└─sda1 8:1 0 465.8G 0 part /
sdb 8:16 0 465.8G 0 disk
├─sdb1 8:17 0 464.8G 0 part
└─sdb2 8:18 0 1023M 0 part
sr0 11:0 1 1024M 0 rom

root@decisive-punishment:/var/log# sudo parted -l
Model: ATA ST9500620NS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 1049kB 500GB 500GB primary ext4 boot

Model: ATA ST9500620NS (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 2 1049kB 1074MB 1073MB ceph journal
 1 1075MB 500GB 499GB xfs ceph data

James Page (james-page)
Changed in swift-storage (Juju Charms Collection):
status: New → Invalid
Revision history for this message
James Page (james-page) wrote :

This bug appears to relate to a transient kernel bug that we saw around this time; We don't seen this now with 16.04 and later testing so marking this bug as invalid.

Changed in charm-swift-storage:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.