[disco] [eoan] After unmount, cannot open /dev/vdb: Device or resource busy

Disco and Eoan device is busy after unmounting an ephemeral disk, cannot format the device until rebooting.

This is blocking all of OpenStack Charms which interact with block devices (Nova Compute, Ceph, Swift, Cinder), on the Disco and Eoan series. As we are nearing LTS-1, this will become urgent pretty quickly.

Reproducer, on an OpenStack cloud:
juju deploy cs:ubuntu ubuntu-bionic --series bionic
juju deploy cs:ubuntu ubuntu-disco --series disco
juju deploy cs:ubuntu ubuntu-eoan --series eoan

#### Succeeds on Bionic:

ubuntu@juju-8d01b7-foo-14:~$ uname -a
Linux juju-8d01b7-foo-14 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@juju-8d01b7-foo-14:~$ lsb_release -c
Codename: bionic

ubuntu@juju-8d01b7-foo-14:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk /mnt

ubuntu@juju-8d01b7-foo-14:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 985M 0 985M 0% /dev
tmpfs 200M 712K 199M 1% /run
/dev/vda1 20G 1.7G 18G 9% /
tmpfs 997M 0 997M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 997M 0 997M 0% /sys/fs/cgroup
/dev/vda15 105M 3.6M 101M 4% /boot/efi
/dev/vdb 40G 49M 38G 1% /mnt
tmpfs 100K 0 100K 0% /var/lib/lxd/shmounts
tmpfs 100K 0 100K 0% /var/lib/lxd/devlxd
tmpfs 200M 0 200M 0% /run/user/1000

ubuntu@juju-8d01b7-foo-14:~$ sudo umount /dev/vdb

ubuntu@juju-8d01b7-foo-14:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
meta-data=/dev/vdb isize=1024 agcount=4, agsize=2621440 blks
         = sectsz=512 attr=2, projid32bit=1
         = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=10485760, imaxpct=25
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=5120, version=2
         = sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

#### Fails on Disco:

ubuntu@juju-8d01b7-foo-12:~$ uname -a
Linux juju-8d01b7-foo-12 5.0.0-27-generic #28-Ubuntu SMP Tue Aug 20 19:53:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@juju-8d01b7-foo-12:~$ lsb_release -c
Codename: disco

ubuntu@juju-8d01b7-foo-12:~$ sudo umount /mnt

ubuntu@juju-8d01b7-foo-12:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk

ubuntu@juju-8d01b7-foo-12:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
mkfs.xfs: cannot open /dev/vdb: Device or resource busy

#### Fails on Eoan:

ubuntu@juju-8d01b7-foo-13:~$ sudo umount /mnt

ubuntu@juju-8d01b7-foo-13:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk

ubuntu@juju-8d01b7-foo-13:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
mkfs.xfs: cannot open /dev/vdb: Device or resource busy

..

Original bug description:

On disco-stein, hook failed: "config-changed" with mkfs.xfs: cannot open /dev/vdb: Device or resource busy

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-cinder/678676/3/3803/index.html

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-cinder/678676/3/3803/test_charm_func_full_7062/juju-status-zaza-5b39f0208674.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-cinder/678676/3/3803/test_charm_func_full_7062/juju-crashdump-7d3902a2-4fdf-435a-82c4-5d2ad9af4cb5.tar.xz

2019-09-03 21:33:27 DEBUG config-changed Unpacking apparmor-utils (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-libapparmor (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-apparmor (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up apparmor-utils (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Processing triggers for man-db (2.8.5-2) ...
2019-09-03 21:33:30 INFO juju-log Setting up the apparmor profile for usr.bin.nova-compute in enforce mode.
2019-09-03 21:33:30 DEBUG config-changed Setting /etc/apparmor.d/usr.bin.nova-compute to enforce mode.
2019-09-03 21:33:30 DEBUG config-changed mkfs.xfs: cannot open /dev/vdb: Device or resource busy
2019-09-03 21:33:30 DEBUG config-changed Traceback (most recent call last):
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line 751, in <module>
2019-09-03 21:33:30 DEBUG config-changed main()
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line 744, in main
2019-09-03 21:33:30 DEBUG config-changed hooks.execute(sys.argv)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
2019-09-03 21:33:30 DEBUG config-changed self._hooks[hook_name]()
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1560, in wrapped_f
2019-09-03 21:33:30 DEBUG config-changed stopstart, restart_functions)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2019-09-03 21:33:30 DEBUG config-changed r = lambda_f()
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1559, in <lambda>
2019-09-03 21:33:30 DEBUG config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
2019-09-03 21:33:30 DEBUG config-changed return f(*args, **kwargs)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line 264, in config_changed
2019-09-03 21:33:30 DEBUG config-changed configure_local_ephemeral_storage()
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/nova_compute_utils.py", line 1015, in configure_local_ephemeral_storage
2019-09-03 21:33:30 DEBUG config-changed mkfs_xfs(dev, force=True)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/storage/linux/utils.py", line 128, in mkfs_xfs
2019-09-03 21:33:30 DEBUG config-changed check_call(cmd)
2019-09-03 21:33:30 DEBUG config-changed File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
2019-09-03 21:33:30 DEBUG config-changed raise CalledProcessError(retcode, cmd)
2019-09-03 21:33:30 DEBUG config-changed subprocess.CalledProcessError: Command '['mkfs.xfs', '-f', '-i', 'size=1024', '/dev/vdb']' returned non-zero exit status 1.
2019-09-03 21:33:30 ERROR juju.worker.uniter.operation runhook.go:132 hook "config-changed" failed: exit status 1
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 5 20:47 seq
 crw-rw---- 1 root audio 116, 33 Sep 5 20:47 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 19.04
Ec2AMI: ami-0000078c
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-27-generic root=PARTUUID=c6e26b6d-1cc5-47e0-8025-e7a9c79727cc ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.0.0-27.28-generic 5.0.21
RelatedPackageVersions:
 linux-restricted-modules-5.0.0-27-generic N/A
 linux-backports-modules-5.0.0-27-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: disco ec2-images
Uname: Linux 5.0.0-27-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.10:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.10
dmi.sys.vendor: OpenStack Foundation
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 5 20:48 seq
 crw-rw---- 1 root audio 116, 33 Sep 5 20:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.11-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 19.10
Ec2AMI: ami-0000078d
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.2.0-15-generic root=PARTUUID=c92b681b-892a-4294-a588-50850030af8a ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.2.0-15.16-generic 5.2.9
RelatedPackageVersions:
 linux-restricted-modules-5.2.0-15-generic N/A
 linux-backports-modules-5.2.0-15-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: eoan ec2-images
Uname: Linux 5.2.0-15-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.10:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.10
dmi.sys.vendor: OpenStack Foundation
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 5 20:48 seq
 crw-rw---- 1 root audio 116, 33 Sep 5 20:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 18.04
Ec2AMI: ami-00000789
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-60-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 4.15.0-60.67-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-60-generic N/A
 linux-backports-modules-4.15.0-60-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic ec2-images
Uname: Linux 4.15.0-60-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.10:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.10
dmi.sys.vendor: OpenStack Foundation