ceph: TestVolumeBootPattern.test_create_ebs_image_and_check_boot times out failing to delete volume due to snapshot is busy
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
In Progress
|
Undecided
|
Jon Bernard |
Bug Description
Seen in the ceph job here:
2016-12-09 19:33:41.200131 | tempest.
2016-12-09 19:33:41.200170 | -------
2016-12-09 19:33:41.200185 |
2016-12-09 19:33:41.200196 | Captured traceback:
2016-12-09 19:33:41.200206 | ~~~~~~~~~~~~~~~~~~~
2016-12-09 19:33:41.200218 | Traceback (most recent call last):
2016-12-09 19:33:41.200241 | File "tempest/
2016-12-09 19:33:41.200257 | raise exceptions.
2016-12-09 19:33:41.200274 | tempest.
2016-12-09 19:33:41.200309 | Details: (TestVolumeBoot
2016-12-09 19:33:41.200315 |
2016-12-09 19:33:41.200320 |
2016-12-09 19:33:41.200329 | Captured traceback-1:
2016-12-09 19:33:41.200337 | ~~~~~~~
2016-12-09 19:33:41.200349 | Traceback (most recent call last):
2016-12-09 19:33:41.200372 | File "tempest/
2016-12-09 19:33:41.200385 | return func(*args, **kwargs)
2016-12-09 19:33:41.200407 | File "tempest/
2016-12-09 19:33:41.200424 | resp, body = self.delete(
2016-12-09 19:33:41.200442 | File "tempest/
2016-12-09 19:33:41.200461 | return self.request(
2016-12-09 19:33:41.200479 | File "tempest/
2016-12-09 19:33:41.200492 | self._error_
2016-12-09 19:33:41.200512 | File "tempest/
2016-12-09 19:33:41.200527 | raise exceptions.
2016-12-09 19:33:41.200542 | tempest.
2016-12-09 19:33:41.200592 | Details: {u'code': 400, u'message': u'Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group or have snapshots.'}
From the cinder-volume logs:
2016-12-09 19:14:00.798 ERROR cinder.
We likely have a race here so would need to trace the requests from the failure and see what's blowing up and in what order. Possibly we need to retry the snapshot delete until it's not busy, or timeout in c-vol.
Changed in cinder: | |
status: | New → Confirmed |
Changed in cinder: | |
assignee: | nobody → Jon Bernard (jbernard) |
Changed in cinder: | |
status: | Confirmed → In Progress |
Changed in cinder: | |
assignee: | nobody → Jon Bernard (jbernard) |
status: | Invalid → In Progress |
http:// logstash. openstack. org/#dashboard/ file/logstash. json?query= message% 3A%5C%22TestVol umeBootPattern% 5C%22%20AND% 20message% 3A%5C%22Delete% 20snapshot% 20failed% 2C%20due% 20to%20snapshot %20busy. %5C%22% 20AND%20tags% 3A%5C%22screen- c-vol.txt% 5C%22&from= 7d
5 hits in the last 7 days, check and gate, all failures and all in the ceph job.