Glance

Image remains in queued state for web-download when node_staging_uri end with "/"

  1. Series queens
  2. Bug #1753964
Bug #1753964 reported by Abhishek Kekane
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
High
Abhishek Kekane
Glance rocky-1 "17.0.0.0b1"
Queens
Fix Committed
High
Abhishek Kekane
Glance queens-stable-1 "16.0.1"

Bug Description

The node_staging_uri has a default of file:///tmp/staging/
see: https://github.com/openstack/glance/blob/a0aaa614712090e7cab19dc35a155c32ea8f2190/glance/common/config.py#L680

If operator does not set 'node_staging_uri' in glance-api.conf then importing image using web-download remains in queued state.

Steps to reproduce:
1. Ensure glance-api is running under mod_wsgi (add WSGI_MODE=mod_wsgi in local.conf and run stack.sh)
2. Do not set node_staging_uri in glance-api.conf

3. Create image using below curl command:
curl -i -X POST -H "x-auth-token: <auth-token>" http://192.168.0.13:9292/v2/images -d '{"container_format":"bare","disk_format":"raw","name":"Import web-download"}'

4. Import image using below curl command:
curl -i -X POST -H "Content-type: application/json" -H "x-auth-token: <auth-token>" http://192.168.0.13:9292/v2/images/<id-of-image-created-in-step-3>/import -d '{"method":{"name":"web-download","uri":"https://www.openstack.org/assets/openstack-logo/2016R/OpenStack-Logo-Horizontal.eps.zip"}}'

Expected result:
Image should be in active state.

Actual result:
Image remains in queued state.

API Logs:
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: DEBUG glance_store.backend [-] Attempting to import store file {{(pid=3506) _load_store /usr/local/lib/python2.7/dist-packages/glance_store/backend.py:231}}
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: DEBUG glance_store.capabilities [-] Store glance_store._drivers.filesystem.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. {{(pid=3506) update_capabilities /usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py:97}}
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: Traceback (most recent call last):
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: func(*args, **kwargs)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/proxy.py", line 238, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self.base.run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/notifier.py", line 581, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: super(TaskProxy, self).run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/proxy.py", line 238, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self.base.run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/proxy.py", line 238, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self.base.run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/__init__.py", line 438, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: executor.begin_processing(self.task_id)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/taskflow_executor.py", line 144, in begin_processing
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: super(TaskExecutor, self).begin_processing(task_id)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/__init__.py", line 63, in begin_processing
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self._run(task_id, task.type)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/taskflow_executor.py", line 165, in _run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: flow = self._get_flow(task)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/taskflow_executor.py", line 134, in _get_flow
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: invoke_kwds=kwds).driver
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 61, in __init__
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: warn_on_missing_entrypoint=warn_on_missing_entrypoint
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 81, in __init__
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: verify_requirements)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 203, in _load_plugins
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self._on_load_failure_callback(self, ep, err)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 195, in _load_plugins
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: verify_requirements,
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 158, in _load_one_plugin
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: verify_requirements,
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 227, in _load_one_plugin
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: obj = plugin(*invoke_args, **invoke_kwds)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/flows/api_image_import.py", line 323, in get_flow
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: file_uri = separator.join((CONF.node_staging_uri, str(image_id)))
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: UnboundLocalError: local variable 'separator' referenced before assignment

See original description
Abhishek Kekane (abhishek-kekane)
Changed in glance:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
summary: - 500 Error for web-download if node_staging_uri is not set
+ Image remains in queued state for web-download if node_staging_uri is
+ not set
Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote : Re: Image remains in queued state for web-download if node_staging_uri is not set

Fix proposed to Master :-> https://review.openstack.org/#/c/550417/1

Revision history for this message
Erno Kuvaja (jokke) wrote :

So, the actual bug is invalid, there is clearly a bug in the side effect though, which is it throwing 500.

The node_staging_uri is used as local cache for the taskflow to store the data it downloads before the rest of the tasks are ran. So it is needed for the 'web-download' (In 'web-download' only use case it does not need to be shared between the glance nodes). If the methods are incorrectly configured, the image should stay in 'queued' but we should fail the import call gracefully.

Changed in glance:
status: New → Invalid
Brian Rosmaita (brian-rosmaita)
summary: - Image remains in queued state for web-download if node_staging_uri is
- not set
+ Image remains in queued state for web-download when node_staging_uri
+ uses default value
description: updated
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote : Re: Image remains in queued state for web-download when node_staging_uri uses default value

The problem isn't that the operator has configured None for node_staging_uri ... the problem occurs when the operator runs with the default value, which is 'file:///tmp/staging/', and someone tries a web-download import.

This is actually a bad bug. I think the normal way to write a directory is with a '/' at the end, and doing so will cause this failure to occur.

Brian Rosmaita (brian-rosmaita)
Changed in glance:
status: Invalid → In Progress
importance: Undecided → High
Brian Rosmaita (brian-rosmaita)
tags: added: queens-backport-potential
Erno Kuvaja (jokke)
summary: - Image remains in queued state for web-download when node_staging_uri
- uses default value
+ Image remains in queued state for web-download when node_staging_uri end
+ with "/"
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/550417
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=ff1ba4f3f5afcbd06a7bbb4cf4363ac8f368ee4c
Submitter: Zuul
Branch: master

commit ff1ba4f3f5afcbd06a7bbb4cf4363ac8f368ee4c
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 7 09:56:50 2018 +0000

    Fix web-download fails with default node_staging_uri

    Web-download import method fails to import image to backend if
    'node_staging_uri' is not set in glance-api.conf (or when the value
    ends with a '/', as the default setting does) due to "local variable
    'separator' referenced before assignment" error.

    Co-authored-by: Abhishek Kekane <email address hidden>
    Co-authored-by: Brian Rosmaita <email address hidden>

    Change-Id: I056fb7a1f7a7ef6bc14ea54133cf083b2b9d9920
    Closes-Bug: #1753964

Changed in glance:
status: In Progress → Fix Released
Brian Rosmaita (brian-rosmaita)
Changed in glance:
milestone: none → rocky-1
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Change proposed to stable/queens: https://review.openstack.org/#/c/559073/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/queens)

Reviewed: https://review.openstack.org/559073
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=40c886662e4cc6a633b858978b702b7fc257a579
Submitter: Zuul
Branch: stable/queens

commit 40c886662e4cc6a633b858978b702b7fc257a579
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 7 09:56:50 2018 +0000

    Fix web-download fails with default node_staging_uri

    Web-download import method fails to import image to backend if
    'node_staging_uri' is not set in glance-api.conf (or when the value
    ends with a '/', as the default setting does) due to "local variable
    'separator' referenced before assignment" error.

    Co-authored-by: Abhishek Kekane <email address hidden>
    Co-authored-by: Brian Rosmaita <email address hidden>

    Change-Id: I056fb7a1f7a7ef6bc14ea54133cf083b2b9d9920
    Closes-Bug: #1753964
    (cherry picked from commit ff1ba4f3f5afcbd06a7bbb4cf4363ac8f368ee4c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 17.0.0.0b1

This issue was fixed in the openstack/glance 17.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 16.0.1

This issue was fixed in the openstack/glance 16.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.