cinder-manage service list shows happy for uninitialized driver

Bug #1446750 reported by Walt Boring
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Low
Michal Dulko
Kilo
Won't Fix
Undecided
Unassigned

Bug Description

If there is a problem with a cinder driver not initializing correctly, but the c-vol service is up,
cinder-manage shows the cinder-volume service State as ':-)'

For example,
  If I try and configure the lefthand driver with an improper setting, such as lefthand_api_url = http://bogus.url

The service comes up, but the driver is left uninitialized.

2015-04-21 09:30:51.894 ERROR cinder.volume.manager [req-fe220083-3d7a-4bb1-a8b9-a058e14eb7fb None None] Error encountered during initialization of driver: HPLeftHandISCSIDriver
2015-04-21 09:30:51.895 DEBUG cinder.openstack.common.service [req-efc2cc5a-f7b9-4d80-a392-85dcdbd648a2 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'o
slo.messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'rou
tes.middleware=WARN', 'stevedore=WARN'] from (pid=30669) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2187
2015-04-21 09:30:51.896 DEBUG cinder.openstack.common.service [req-efc2cc5a-f7b9-4d80-a392-85dcdbd648a2 None None] default_volume_type = 3parfc from (pid=30669) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/
cfg.py:2187
2015-04-21 09:30:51.897 DEBUG cinder.openstack.common.service [req-efc2cc5a-f7b9-4d80-a392-85dcdbd648a2 None None] driver_client_cert = None from (pid=30669) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cf
g.py:2187
2015-04-21 09:30:51.895 ERROR cinder.volume.manager [req-fe220083-3d7a-4bb1-a8b9-a058e14eb7fb None None] Bad request (HTTP 400)
2015-04-21 09:30:51.895 TRACE cinder.volume.manager Traceback (most recent call last):
2015-04-21 09:30:51.895 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 302, in init_host
2015-04-21 09:30:51.895 TRACE cinder.volume.manager self.driver.check_for_setup_error()
2015-04-21 09:30:51.895 TRACE cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-04-21 09:30:51.895 TRACE cinder.volume.manager return f(*args, **kwargs)
2015-04-21 09:30:51.895 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_iscsi.py", line 78, in check_for_setup_error
2015-04-21 09:30:51.895 TRACE cinder.volume.manager self.proxy.check_for_setup_error()
2015-04-21 09:30:51.895 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_rest_proxy.py", line 162, in check_for_setup_error
2015-04-21 09:30:51.895 TRACE cinder.volume.manager client = self._login()
2015-04-21 09:30:51.895 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_rest_proxy.py", line 125, in _login
2015-04-21 09:30:51.895 TRACE cinder.volume.manager client = self.do_setup(None)
2015-04-21 09:30:51.895 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_rest_proxy.py", line 158, in do_setup
2015-04-21 09:30:51.895 TRACE cinder.volume.manager raise exception.DriverNotInitialized(ex)
2015-04-21 09:30:51.895 TRACE cinder.volume.manager DriverNotInitialized: Bad request (HTTP 400)
2015-04-21 09:30:51.895 TRACE cinder.volume.manager

And later when get_volume_stats() is called:

2015-04-21 09:32:02.021 WARNING cinder.volume.manager [req-a09699a4-3eb9-47cf-9ce8-50ca5ce5c832 None None] Unable to update stats, HPLeftHandISCSIDriver -1.0.6 (config name lefthand) driver is uninitialized.

The c-vol service is up, but the driver can't accept provisioning requests and can't do get_volume_stats().

cinder-manage service list shows it's happy.

cinder-volume trusty-devstack@lefthand nova enabled :-) 2015-04-21 16:34:41

Shouldn't we say it's somewhere between XXX and :-) ??

Changed in cinder:
assignee: nobody → Vilobh Meshram (vilobhmm)
Changed in cinder:
importance: Undecided → Low
Mike Perez (thingee)
Changed in cinder:
milestone: none → liberty-1
Mike Perez (thingee)
tags: added: kilo-backport-potential
Revision history for this message
Vilobh Meshram (vilobhmm) wrote :

Here along with the checking whether the service_is_up or not we should also check if we were able to successfully initialize the driver.

so if ( initialize driver && service_is_up)
          :)
      else
          XXX

Changed in cinder:
status: New → In Progress
Revision history for this message
Vilobh Meshram (vilobhmm) wrote :
Revision history for this message
Jay Bryant (jsbryant) wrote :

I have seen this problem as well. We should get it fixed.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/185730

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/187670

Changed in cinder:
assignee: Vilobh Meshram (vilobhmm) → Michal Dulko (michal-dulko-f)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/187670
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2a84ae0fc015a43617521727ca31152066839d3c
Submitter: Jenkins
Branch: master

commit 2a84ae0fc015a43617521727ca31152066839d3c
Author: Michal Dulko <email address hidden>
Date: Thu Jun 4 17:25:24 2015 +0200

    Don't send heartbeats if Manager reports a problem

    This commit adds is_working method to base Manager class that can be
    used to indicate that service has some problems and isn't running
    correctly. This is used to block refreshing Service heartbeats if
    Manager will return False from is_working.

    The method is implemented in c-vol Manager to return initialization
    state of the underlying volume driver.

    Closes-Bug: 1446750
    DocImpact

    Change-Id: I809b2b777a0f84d12f39f2233920ec961f74f846

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/185730
Reason: Over a month with no update.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/216728

Thierry Carrez (ttx)
Changed in cinder:
milestone: liberty-1 → 7.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (master)

Reviewed: https://review.openstack.org/216728
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9889f96afdc81442fd4b1966777a8dcd143e243e
Submitter: Jenkins
Branch: master

commit 9889f96afdc81442fd4b1966777a8dcd143e243e
Author: Szymon Wroblewski <email address hidden>
Date: Thu Aug 20 14:13:31 2015 +0200

    Support initialization state in Backup Manager

    This commit adds is_working method to Backup Manager class to
    return initialization state of the underlying volume drivers and
    indicate that service has some problems and isn't running correctly.
    This is used to block refreshing Service heartbeats if Manager will
    return False from is_working.

    Related-Bug: 1446750
    Change-Id: I3165d01f60bd80ead5163d8678886bd2828af3f9

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.