Volume creation failed , according there is no any cinder services on node with cinder role

Bug #1247939 reported by Tatyanka
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Triaged
Critical
Vladimir Kuklin

Bug Description

Reproduced on 319 iso:
http://srv11-msk.msk.mirantis.net/fuelweb-iso/fuel-319-2013-11-04_14-52-21.iso

Configuration:
KVM
CentOs simple (1 controller + 1 compute + 1 cinder) + Neutron Vlan

Steps to reproduce:
Deploy env
Execute OSTF

Actual result:
Volume goes to error state when we try to create it. I have node-3 with cinder role
ssh on node-3 from admin node and execute:
[root@node-3 ~]# ps uax| grep api
root 18692 0.0 0.0 103236 848 pts/0 S+ 18:52 0:00 grep api
[root@node-3 ~]# ps uax| grep cinder
root 18694 0.0 0.0 103236 848 pts/0 S+ 18:52 0:00 grep cinder
[root@node-3 ~]# ps uax| grep volume
root 18696 0.0 0.0 103236 848 pts/0 S+ 18:52 0:00 grep volume
But services are included in init.d, just not run here:
[root@node-3 ~]# /etc/init.d/openstack-cinder-backup status
openstack-cinder-backup is stopped
[root@node-3 ~]# /etc/init.d/openstack-cinder-api status
openstack-cinder-api is stopped
also cinder config sample is placed in /etc/cinder/cinder.conf with all data commented
As soon as service try to start with 2 config I try to see config from /usr/share/cinder-dist.conf

and here it is
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes
iscsi_helper = tgtadm
sql_connection = mysql://cinder:cinder@localhost/cinder
rpc_backend = cinder.openstack.common.rpc.impl_qpid
rootwrap_config = /etc/cinder/rootwrap.conf
auth_strategy = keystone

[keystone_authtoken]
admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http

As we can see there is no defined cvolume group(to succesfuul start of lvm backend) also why we have here rpc backend defined as qpid if we have rabbit on Centos?
also there is missed parameters like:
use_syslog=true
api_paste_config=/etc/cinder/api-paste.ini
volume_group=cinder
rabbit_userid=nova
bind_host=0.0.0.0
osapi_volume_listen=0.0.0.0
iscsi_ip_address=

More over if we ssh on controller node we can see running cinder services here
cinder 8138 0.1 5.0 414348 51360 ? S 17:41 0:05 /usr/bin/python /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
cinder 8190 1.2 0.3 251956 3924 ? S 17:41 0:58 /usr/bin/python /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
cinder 8226 0.1 1.7 372448 17368 ? S 17:41 0:05 /usr/bin/python /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
cinder 8331 0.0 2.2 375768 22676 ? S 17:42 0:04 /usr/bin/python /usr/bin/cinder-scheduler --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile
But with stopped backup
[root@node-1 ~]# /etc/init.d/openstack-cinder-backup status
openstack-cinder-backup is stopped
And missed data (and qpid for rpc)in /etc/cinder/cinder.conf and /usr/share/cinder-dist.conf

As result we can see next error in the cinder-lvm logs (and others)
cindcinder-cinder.brick.local_dev.lvm.log

2013-11-04T17:42:00.997489+00:00 emerg: ERROR: Unable to locate Volume Group cinder
in volume scheduler:
2013-11-04T18:06:58.026527+00:00 emerg: ERROR: Failed to schedule_create_volume: No valid host was found.
2013-11-04T18:14:35.606772+00:00 emerg: ERROR: Failed to schedule_create_volume: No valid host was found.
2013-11-04T18:16:01.290510+00:00 emerg: ERROR: Failed to schedule_create_volume: No valid host was found.
2013-11-04T18:16:18.931886+00:00 emerg: ERROR: Failed to schedule_create_volume: No valid host was found.
2013-11-04T18:18:14.417042+00:00 emerg: ERROR: Failed to schedule_create_volume: No valid host was found.
2013-11-04T17:42:00.998480+00:00 emerg: ERROR: Error encountered during initialization of driver: LVMISCSIDriver
2013-11-04T17:42:01.015836+00:00 err: ERROR: Bad or unexpected response from the storage volume backend API: Volume Group cinder does not exist
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 190, in init_host
    self.driver.check_for_setup_error()
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 94, in check_for_setup_error
    raise exception.VolumeBackendAPIException(data=message)

Full logs find in attachment

Tags: cinder library
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
status: New → Confirmed
Changed in fuel:
importance: Undecided → Critical
assignee: nobody → Vladimir Kuklin (vkuklin)
Changed in fuel:
status: Confirmed → New
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.