Can't remove VGs when partial LVs exist

Not sure if the following scenarios should be mentioned in this bug or new bugs should be created (both similar but unique tracebacks included below in respective order):

1. A volume with that name already exists (perhaps on a different disk)
2. A volume group is assumed to exist when it does not

config-changed Physical volume "/dev/loop0" successfully created
config-changed A volume group called cinder-volumes already exists.
config-changed Traceback (most recent call last):
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 519, in <module>
config-changed hooks.execute(sys.argv)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 557, in execute
config-changed self._hooks[hook_name]()
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/host.py", line 312, in wrapped_f
config-changed f(*args, **kwargs)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 131, in config_changed
config-changed conf['remove-missing'])
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 372, in configure_lvm_storage
config-changed create_lvm_volume_group(volume_group, new_devices[0])
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/storage/linux/lvm.py", line 105, in create_lvm_volume_group
config-changed check_call(['vgcreate', volume_group, block_device])
config-changed File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
config-changed raise CalledProcessError(retcode, cmd)
config-changed subprocess.CalledProcessError: Command '['vgcreate', u'cinder-volumes', '/dev/loop0']' returned non-zero exit status 5

config-changed Volume group "cinder-volumes" not found
config-changed Traceback (most recent call last):
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 519, in <module>
config-changed hooks.execute(sys.argv)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 557, in execute
config-changed self._hooks[hook_name]()
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/host.py", line 312, in wrapped_f
config-changed f(*args, **kwargs)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/config-changed", line 131, in config_changed
config-changed conf['remove-missing'])
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 377, in configure_lvm_storage
config-changed reduce_lvm_volume_group_missing(volume_group)
config-changed File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 313, in reduce_lvm_volume_group_missing
config-changed subprocess.check_call(['vgreduce', '--removemissing', volume_group])
config-changed File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
config-changed raise CalledProcessError(retcode, cmd)
config-changed subprocess.CalledProcessError: Command '['vgreduce', '--removemissing', u'cin...

I'm testing lp:~jason-hobbs/charms/trusty/cinder/force-vgreduce to see if it resolves this issue.

Jason, If your patch works I think we should put that under a different config option, perhaps 'remove-missing-force' to keep behavior of 'remove-missing' they same.

As for the failure of 'vgcreate cinder-volumes /dev/loop0' when a volume with that name already exists. I'm thinking we might be able to use --force there too, if the 'overwrite' option is true.

With this fix we stopped seeing the issue I reported, but are still seeing at least one of the issues Greg mentioned, so I dug into this a bit more to see what's happening in these different cases.

The prompt to use "--force" comes up when attempting to use 'vgreduce --removemissing cinder-volumes'.

To recreate this you error you need at least 2 devices. Create a volume group named 'cinder-volumes' on the devices, then create a big enough logical volume (or more than one) to use physical extents from both devices. Then use 'dd' to overwrite one of the devices. Then run 'vgreduce --removemissing cinder-volumes'. This will produce an error saying there are still partial LVs in VG cinder-volumes, since cinder-volumes extended across both devices and one is now missing. It doesn't want to remove the missing PV from cinder-volumes because there are partial LVs, so it's asking for confirmation.

We could hit this scenario if system with multiple storage devices has an existing PV on one of the devices configured for use by cinder, and that PV is part of a VG with other PVs now missing (overwritten and used for other things since the node was last used for cinder, or maybe was just a loopback device on filesystem that's gone now).

Using --force will cause vgreduce to remove the missing PV, and to remove any LVs that used it. The charm will function then. The ability to use '--force' could be added as an option to the charm.

I'll submit a patch with a new 'remove-missing-force' option as you suggest Corey.

Greg mentioned two other failures - I think those are both unique bugs and separate from this one. I filed them as bug 1476418 and bug 1476419.