It turns out the waiting for waiting for conductor is when the conductor is being upgraded.


The real bug was  file injection was enabled by default in havana. It would silently error out when libguestfs isn't installed (as its not in in havana). but when we upgrade libguestfs get installed and nova dynamically loads it.


2014-03-03 18:41:39.708 ERROR nova.virt.libvirt.driver [req-fc92fbbf-acd4-40a5-8d74-a8bdd263c297 ServersTestManualDisk-834103425 ServersTestManualDisk-1514739295] [instance: a9a923e6-5748-465e-8ad3-84077b15447e] Error injecting data into image cf82ca56-7fda-4e29-ae75-96be49f2a059 (Error mounting /opt/stack/data/nova/instances/a9a923e6-5748-465e-8ad3-84077b15447e/disk with libguestfs (external command failed, see earlier error messages))
2014-03-03 18:41:39.709 ERROR nova.compute.manager [req-fc92fbbf-acd4-40a5-8d74-a8bdd263c297 ServersTestManualDisk-834103425 ServersTestManualDisk-1514739295] [instance: a9a923e6-5748-465e-8ad3-84077b15447e] Instance failed to spawn
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e] Traceback (most recent call last):
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]   File "/opt/stack/old/nova/nova/compute/manager.py", line 1423, in _spawn
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]     block_device_info)
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2084, in spawn
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]     admin_pass=admin_password)
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2514, in _create_image
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]     instance=instance)
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2508, in _create_image
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]     mandatory=('files',))
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]   File "/opt/stack/old/nova/nova/virt/disk/api.py", line 329, in inject_data
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]     fs.setup()
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]   File "/opt/stack/old/nova/nova/virt/disk/vfs/guestfs.py", line 129, in setup
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]     {'imgfile': self.imgfile, 'e': e})
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e] NovaException: Error mounting /opt/stack/data/nova/instances/a9a923e6-5748-465e-8ad3-84077b15447e/disk with libguestfs (external command failed, see earlier error messages)
2014-03-03 18:41:39.709 6130 TRACE nova.compute.manager [instance: a9a923e6-5748-465e-8ad3-84077b15447e]