Fuel for OpenStack

image could not be stored in rbd in case if ceph is using as backend for glance

Bug #1319106 reported by Tatyanka
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Low
Dmitry Borodaenko
Fuel for OpenStack 5.1
5.0.x
Won't Fix
Low
Dmitry Borodaenko
Fuel for OpenStack 5.0.2

Bug Description

Steps to reproduce:
1. Run system test ceph_multinode_with_cinder
or revert snapshot from CI http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.thread_1/46/console
2. Run tests ostf create/snapshot

Expected Result:
should pass

Actual result:
Fail with 502 error from server,
I look at the glance logs and there is no error and we can see that call (saving image in rbd) was redirect to rbd backend (we use ceph in this test for image)
I go to the ceph nodes and find next traces here:

http://paste.openstack.org/show/80237/

and on the other ceph node from ceph cluster:
http://paste.openstack.org/show/80238/

Tags: ceph
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Tatyanka (tatyana-leontovich)
summary: - image could not be stored in rbd in case if cepth is using for images
+ image could not be stored in rbd in case if ceph is using as backend for
+ glance
Dmitry Borodaenko (angdraug)
tags: added: ceph
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Looks like reocurrence of this python-ceph bug:
https://github.com/ceph/ceph/pull/1787

When rbd_ceph_conf value is not set, it defaults to '', and python-ceph needs None to trigger lookup of ceph.conf in default locations such as /etc/ceph/ceph.conf.

Cinder should be patched to set rbd_ceph_conf default to None instead of ''.

Changed in fuel:
status: New → Confirmed
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/93523

Changed in cinder:
assignee: nobody → Dmitry Borodaenko (dborodaenko)
status: New → In Progress
Mike Scherbakov (mihgen)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Borodaenko (dborodaenko)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

My previous comment is wrong, incorrect behaviour is triggered by None being passed to python-ceph, not ''.

Changed in cinder:
status: In Progress → Invalid
Dmitry Borodaenko (angdraug)
no longer affects: cinder
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Download full text (4.2 KiB)

Guys may bu issue is invalid, but we receive on latest test run the same error but for now for volume:
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.581 15666 DEBUG cinder.openstack.common.service [-] migration_create_volume_timeout_secs = 300 log_opt_values /usr/lib/python2.6/site-packages/oslo/config/cfg.py:1945
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 ERROR cinder.volume.drivers.rbd [req-6f1e1169-5670-4665-bccf-b7353c193e75 - - - - -] error connecting to ceph cluster
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd Traceback (most recent call last):
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 263, in check_for_setup_error
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd with RADOSClient(self):
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 235, in __init__
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd self.cluster, self.ioctx = driver._connect_to_rados(pool)
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 281, in _connect_to_rados
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd client = self.rados.Rados(rados_id=ascii_user, conffile=ascii_conf)
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.6/site-packages/rados.py", line 221, in __init__
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd self.conf_read_file(conffile)
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.6/site-packages/rados.py", line 272, in conf_read_file
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd raise make_ex(ret, "error calling conf_read_file")
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd Error: error calling conf_read_file: errno EINVAL
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.578 15679 TRACE cinder.volume.drivers.rbd
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.582 15679 ERROR cinder.volume.manager [req-6f1e1169-5670-4665-bccf-b7353c193e75 - - - - -] Error encountered during initialization of driver: RBDDriver
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.582 15679 ERROR cinder.volume.manager [req-6f1e1169-5670-4665-bccf-b7353c193e75 - - - - -] Bad or unexpected response from the storage volume backend API: error connecting to ceph cluster
2014-05-14T06:47:02.892934+00:00 debug: 2014-05-14 06:47:01.582 15679 TRACE cinder....

Read more...

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

details can be found http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.thread_1/47/ - so please, help someone with triage

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Downgrading to High: since there's no bug in Cinder itself, this looks like a timing issue, cinder-volume tries to create a rados connection before ceph.conf is created by puppet/ceph-deploy.

Changed in fuel:
importance: Critical → High
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Downgrading further to Low, this test covers a corner use case of adding Cinder LVM nodes to a cluster that is configured to use Ceph RBD for Cinder volumes.

Changed in fuel:
importance: High → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/93600

Changed in fuel:
status: Confirmed → In Progress
Dmitry Borodaenko (angdraug)
Changed in fuel:
milestone: 5.0 → 5.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-main (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/93689

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/93689
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=483770194fb4fe0a84f169f5644c604b87a3b2a6
Submitter: Jenkins
Branch: master

commit 483770194fb4fe0a84f169f5644c604b87a3b2a6
Author: Tatyana Leontovich <email address hidden>
Date: Thu May 15 13:18:07 2014 +0000

    Fix for test ceph multinode with cinder

    In this tests we add cinder node
    but for valumes and images set ceph in settings
    It is wrong behavior, so this patch just enable
    volume lvm, set ceph for volumes in false
    to get more useful test caase

    Change-Id: Id46bd7b72f92980ff682f2db9f69128e855cdc65
    Related-Bug: #1319106

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/93600
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1da3d70c997ec9375618f87b05f6502510e50bca
Submitter: Jenkins
Branch: master

commit 1da3d70c997ec9375618f87b05f6502510e50bca
Author: Dmitry Borodaenko <email address hidden>
Date: Wed May 14 12:25:17 2014 -0400

    Pull ceph.conf for cinder role

    In a rare scenario of having cinder nodes (providing storage for Cinder LVM
    backend) in an OpenStack environment configured to use Ceph RBD as backend for
    Cinder volumes, cinder-volume on a cinder node would fail to connect to Ceph
    because ceph.conf isn't there.

    Change-Id: I1c4119fbfca3db5f234b18e1e03b7e31b971287e
    Closes-Bug: 1319106

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/113911

Revision history for this message
Andrew Woodward (xarses) wrote :

We won't be fixing low or medium bugs for 5.0.x any longer. If this needs to be addressed we need to increase priority

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/5.0)

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/113911

Tatyanka (tatyana-leontovich)
tags: removed: system-tests
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.