glance_store

BackendException: Failed to add object to Swift. - While trying to upload image size of around 400 MB

Bug #1673280 reported by Sukesh Behera
22
This bug affects 5 people
Affects Status Importance Assigned to Milestone
glance_store
Incomplete
High
Erno Kuvaja

Bug Description

2017-03-15 22:36:29.268 418 ERROR glance_store._drivers.swift.store [req-96a417c4-7dcf-4406-a2d0-54681b54cca7 33efe02c6fbf41afaaf911f811444fd0 ff56560910c64b68b540eb0bd47d76aa - default default] Failed
to add object to Swift.
Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents for reupload..
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data [req-96a417c4-7dcf-4406-a2d0-54681b54cca7 33efe02c6fbf41afaaf911f811444fd0 ff56560910c64b68b540eb0bd47d76aa - default default] Failed to upload
 image data due to internal error
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data Traceback (most recent call last):
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data image.set_data(data, size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.base.set_data(data, size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data _send_notification(notify_error, 'image.upload', msg)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.force_reraise()
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.repo.set_data(data, size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data self.image.set_data(data, size=size)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data verifier=verifier)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 447, in add_to_backend
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data verifier)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 420, in store_add_to_backend
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data verifier=verifier)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data return store_op_fun(store, *args, **kwargs)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py", line 966, in add
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data raise glance_store.BackendException(msg)
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data BackendException: Failed to add object to Swift.
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents f
or reupload..
2017-03-15 22:36:29.271 418 ERROR glance.api.v2.image_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi [req-96a417c4-7dcf-4406-a2d0-54681b54cca7 33efe02c6fbf41afaaf911f811444fd0 ff56560910c64b68b540eb0bd47d76aa - default default] Caught error: Failed t
o add object to Swift.
Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents for reupload..
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi Traceback (most recent call last):
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/common/wsgi.py", line 1113, in __call__
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi request, **action_args)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/common/wsgi.py", line 1152, in dispatch
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return method(*args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/common/utils.py", line 363, in wrapped
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 250, in upload
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self._restore(image_repo, image)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.force_reraise()
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi image.set_data(data, size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.base.set_data(data, size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.force_reraise()
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.repo.set_data(data, size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi self.image.set_data(data, size=size)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi verifier=verifier)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 447, in add_to_backend
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi verifier)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/backend.py", line 420, in store_add_to_backend
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi verifier=verifier)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi File "/openstack/venvs/glance-14.0.7/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py", line 966, in add
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi raise glance_store.BackendException(msg)
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi BackendException: Failed to add object to Swift.
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '00840e88-2589-4543-a5dd-e2b35d0f0440-00001', ...) failure and no ability to reset contents for reu
pload..
2017-03-15 22:36:29.396 418 ERROR glance.common.wsgi

When I try to upload an image of lesser size (around 30 MB), it get thru successfully. I am not sure what might be causing this failure. Any timeout set anywhere?

This is my glance-api.conf looks like :

[glance_store]
default_store = swift

stores = swift,http,cinder

swift_store_config_file = /etc/glance/glance-swift-store.conf
default_swift_reference = swift1
swift_store_auth_insecure = True
swift_store_region = RegionOne
swift_store_container = glance_images
swift_store_create_container_on_put = True
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 2048
swift_store_retry_get_count = 5
swift_store_endpoint_type = publicURL

Tags: swift ironic
Sukesh Behera (sbopenstack)
tags: added: ceph
Sukesh Behera (sbopenstack)
tags: added: swift
removed: ceph
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Is this still occurring? We haven't heard about it from anyone else, and have not seen it ourselves. Have been able to upload 2GB images in recent devstack with swift backend.

Changed in glance-store:
status: New → Incomplete
Revision history for this message
Rajiv Mucheli (rajiv.mucheli) wrote :

I keep getting this error irrespective image sizes, logs are similar as well. Any suggestions on how to fix this ?

Revision history for this message
Oliver (ciwi) wrote :

I'm getting this error for snapshots above a certain size, e.g. no error when taking a snapshot of a 'fresh' Ubuntu 20.04 instance but it occurs after 'apt upgrade' was run on the instance. I'm using OpenStack Victoria.

Erno Kuvaja (jokke)
Changed in glance-store:
importance: Undecided → High
assignee: nobody → Erno Kuvaja (jokke)
Revision history for this message
Harald Kornmayer (hakor) wrote :

Somehow connected to the error above.
When creating an Image, we reach the function add() of class class BaseStore(driver.Store) in the file store.py.
- We get for the parameter "image_size" the value 0.

- We use swift as the backend for our image/snapshot files.

- We can create images with a size below swift_store_large_object_size, as the service will save the new image as one file/chunk

- But for image with a size above, the function fails, as it can not determine the number of chunks due to image_size parameter of 0(zero)

- We did not find a "workaround" to determine the correct image_size within the function add().

As we need to fix this urgently.

* which function is calling the BaseStore.add() function?
* were is the image_size calculated? where does the value originate from?

Any hint is welcome!

Revision history for this message
Erno Kuvaja (jokke) wrote :

I'm pretty sure this is related to https://bugs.launchpad.net/glance/+bug/1916052 and the whole trust mechanism being broken is causing any over 1 segment uploads to swift fail.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.