Comment 0 for bug 1625489

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote : Glance fails to retrieve image from rbd backend

We have a deployment with three controller nodes running glance version 2:12.0.0-0ubuntu2 with a Ceph backend. About 50% of the image download requests from compute nodes are failing with an error like this:

2016-09-20 07:37:32.014 29606 WARNING glance.location [req-78e5191c-e31b-428e-82c5-30ef87feef4e a73f3a58c6ad478a838c3ca9353704f7 7a17e2b80f274591aec96baec3f6bf68 - - -] Get image 41a6aadf-c655-49c8-b77d-eedd356f5349 data failed: 'utf8' codec can't decode byte 0x82 in position 24: invalid start byte.

Looking at the code, it looks like the error is occuring in glance_store/_drivers/rbd.py while reading the image, but it seems very strange that glance is trying to utf8-decode a raw image, which pretty likely is bound to fail. But then it is unclear why only every other request is failing.

The error can be reproduced by just doing a "glance image-download", so it's nothing special happening from the Nova side.