OSTF test for Glance failed because of bug in Ceph backend

Bug #1605242 reported by Dmitrii
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Kairat Kushaev
8.0.x
Invalid
High
MOS Maintenance
9.x
Invalid
High
Kairat Kushaev

Bug Description

Detailed bug description:
OSTF test "Launch instance, create snapshot, launch instance from snapshot" failed at the step 3. Make snapshot of the created instance.

Steps to reproduce:
1. Install MOS 8.0 with standalone-ceph plugin
https://github.com/dmdmitriev/fuel-plugin-standalone-ceph
2. Deploy environment with 3 controllers, 3 Ceph controllers (plugin), 3 Ceph storage, Mongo for Ceilometer, compute.
3. Run OSTF.

Expected results:
Successfull finish of OSTF testing.

Actual result:
OSTF run failed.

Description of the environment:
- Network model: Neutron with tunneling segmentation

Additional information:
After OSTF test fails, there is a VM snapshot, that left in Glance:
# glance image-list
+--------------------------------------+-------------------------------+
| ID | Name |
+--------------------------------------+-------------------------------+
| c9396969-c3df-40e9-8f2c-948d6de486de | ost1_test-snapshot-1961534293 |
| cb94978e-8946-4fa2-8737-ff6bf75410e9 | TestVM |
+--------------------------------------+-------------------------------+

# glance -d image-show ost1_test-snapshot-1961534293
Request returned failure status 404.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/glanceclient/shell.py", line 688, in main
    args.func(client, args)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v2/shell.py", line 193, in do_image_show
    image = gc.images.get(args.id)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 181, in get
    resp, body = self.http_client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 333, in request
    return self._handle_response(resp)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 89, in _handle_response
    raise exc.from_response(resp, resp.content)
HTTPNotFound: 404 Not Found: No image found with ID ost1_test-snapshot-1961534293 (HTTP 404)
404 Not Found: No image found with ID ost1_test-snapshot-1961534293 (HTTP 404)

# glance -d image-delete ost1_test-snapshot-1961534293
Request returned failure status 404.
No image with an ID of 'ost1_test-snapshot-1961534293' exists.

But I can check image's details in Horizon:
Image Details: ost1_test-snapshot-1961534293
Image Overview
Information

Name ost1_test-snapshot-1961534293
ID c9396969-c3df-40e9-8f2c-948d6de486de
Owner 6abccf8c0b2f45a29eb754c9b7f6b30f
Status Active
Public No
Protected No
Checksum 6b7a6d92ad4f87d2882654008079286a
Created July 21, 2016, 9:42 a.m.
Updated July 21, 2016, 9:44 a.m.

Specs
Size 39.2 MB
Container Format BARE
Disk Format RAW
Min RAM 64MB

Custom Properties
base_image_ref cb94978e-8946-4fa2-8737-ff6bf75410e9
image_location snapshot
image_state available
image_type snapshot
instance_uuid 79cce771-23c7-45d9-ad0f-9845f82c8359
owner_id 6abccf8c0b2f45a29eb754c9b7f6b30f
user_id 881bbbf93d3d4b0bbf92708e60b15458

If I check Ceph for that snapshot directly, I can see this:
# rbd -p images list
4284f796-3eef-4ccb-9c4f-77053ff3f438
5180c7f4-bb79-4f6f-ae1d-9e4499d5c26d
64e036e8-b314-4484-8f31-bd3601035dc9
c9396969-c3df-40e9-8f2c-948d6de486de
cb94978e-8946-4fa2-8737-ff6bf75410e9
# rbd snap ls -p images c9396969-c3df-40e9-8f2c-948d6de486de
SNAPID NAME SIZE
    16 snap 40162 kB
# rbd children images/c9396969-c3df-40e9-8f2c-948d6de486de@snap
compute/de6027ce-1d48-4706-9bad-4da8c228a876_disk

There is no VMs, running at this moment, so I don't know why there is a compute/..._disk still in Ceph.

If I'll try to do everything manually (run a VM from image, make a snapshot from VM, run another Vm from snapshot, check that second VM works well, delete second VM, delete snapshot), there is no errors at all.

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

So glance v2 shows images only by ID. The same is applicable for image-delete.
That's why you got 404 errors: "No image found with ID...".
Could you please provide the following info:
what tests failed?
glance api log files
ostf logs.

Revision history for this message
Dmitrii (dsivaev) wrote :

Failed test called "Launch instance, create snapshot, launch instance from snapshot".

glance-api.log when I'm trying to delete image:
2016-07-22 19:50:19.000 11755 WARNING glance.api.v2.images [req-cae7ba28-0dcc-4a3e-bedf-20a3d174a33d 034dd5f68d354667bd9c355f1e2f8d04 f46d54cad5a3488193d7ca4d649e54e1 - - -] Image 8adbfae5-111f-44ac-83cd-e359e94aeee1 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.

# glance image-list
+--------------------------------------+-------------------------------+
| ID | Name |
+--------------------------------------+-------------------------------+
| 8adbfae5-111f-44ac-83cd-e359e94aeee1 | ost1_test-snapshot-1090752148 |
| c9df4f1c-8b94-47e4-b5b0-4dc29ce7ed73 | TestVM |
+--------------------------------------+-------------------------------+
# glance image-delete 8adbfae5-111f-44ac-83cd-e359e94aeee1
409 Conflict: Image 8adbfae5-111f-44ac-83cd-e359e94aeee1 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance. (HTTP 409)

OSTF log in attachment

Revision history for this message
Dmitrii (dsivaev) wrote :

Log file, attached to the previous comment, is for the Glance image named "ost1_test-snapshot-64334680".

But, after a second run of that test CLI command "glance image-show" become useless:
# glance image-show ost1_test-snapshot-64334680
404 Not Found: No image found with ID ost1_test-snapshot-64334680 (HTTP 404)

But I still can check image's info using Horizon:
Image Details: ost1_test-snapshot-64334680
Image Overview
Information
Name ost1_test-snapshot-64334680
ID 23eca076-fcf7-40b9-982f-f79b7d9edcba
Owner f46d54cad5a3488193d7ca4d649e54e1
Status Active
Public No
Protected No
Checksum 7a17095ed1ddc0ce5a0f62e068a9641a
Created July 22, 2016, 7:58 p.m.
Updated July 22, 2016, 8:01 p.m.

Specs
Size 39.2 MB
Container Format BARE
Disk Format RAW
Min RAM 64MB

Custom Properties
base_image_ref c9df4f1c-8b94-47e4-b5b0-4dc29ce7ed73
image_location snapshot
image_state available
image_type snapshot
instance_uuid 54bf0786-f153-4f2d-868d-221997230caa
owner_id f46d54cad5a3488193d7ca4d649e54e1
user_id 034dd5f68d354667bd9c355f1e2f8d04

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

Apparently this is the duplicate of https://bugs.launchpad.net/mos/9.x/+bug/1591081.
Upstream bug is here: https://bugs.launchpad.net/glance-store/+bug/1473953.

So looks like it has been fixed for MOS 9.0 but it was not applied to 8.0 because minimal glance_store version is 0.8 AFAIK.

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

Could you please check that it was not fixed for 9.0/9.1 to ensure that the problem is the same?

Revision history for this message
Dmitrii (dsivaev) wrote :

> Apparently this is the duplicate of https://bugs.launchpad.net/mos/9.x/+bug/1591081.
Kairat, I don't think so. The bug you mentioned is about uploading BIG images, but our image is just 40MB.

> Could you please check that it was not fixed for 9.0/9.1 to ensure that the problem is the same?
No, at this moment I don't have free hardware to check this. At the virtual MOS 8.0 environment (with a bit different network scheme) problem doesn't appear.

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

The referenced bug maybe the root cause of the problem if there is http connection broken and users try to re-upload the image. Please attach a snapshot to analyze root cause of the problem.

Revision history for this message
Kairat Kushaev (kkushaev) wrote :

The bug cannot be reproduced for now. Please re-open it when it will be reproduced.
It would be good to have debug turned on when you will manage to repeat the failed scenario.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.