Can't create volume backup with ceph backend. ObjectNotFound: error opening pool 'volumebackups

Bug #1622561 reported by Sergei Chipiga
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
High
Timur Sufiev

Bug Description

Steps to reproduce:

- Login as admin
- Go to Project -> Compute -> Volumes
- Create a volume
- For created volume in dropdown menu select item `Create backup`
- Fill name and submit form

Expected result:
- Backup has `Available` status

Actual result:
- Backup has status `Error`

cinder-backup.log has error:
2016-09-12 10:41:16.481 20625 DEBUG oslo_db.sqlalchemy.engines [req-3110d07c-80fe-403d-803a-a983dd64538c 88d0c4a7834c47eea1e3661294d8e1fb 0e7352a68e1549608b873c8ac6450104 - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher [req-3110d07c-80fe-403d-803a-a983dd64538c 88d0c4a7834c47eea1e3661294d8e1fb 0e7352a68e1549608b873c8ac6450104 - - -] Exception during message handling: error opening pool 'volumebackups'
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 405, in create_backup
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err))
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 399, in create_backup
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher self._run_backup(context, backup, volume)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 447, in _run_backup
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher backup_service.backup(backup, device_path)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 869, in backup
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher self._full_backup(backup, volume_file, volume.name, length)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 696, in _full_backup
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher with rbd_driver.RADOSClient(self, backup.container) as client:
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 248, in __init__
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher self.cluster, self.ioctx = driver._connect_to_rados(pool)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 242, in _connect_to_rados
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher ioctx = client.open_ioctx(pool_to_open)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/rados.py", line 667, in open_ioctx
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher raise make_ex(ret, "error opening pool '%s'" % ioctx_name)
2016-09-12 10:41:16.694 20625 ERROR oslo_messaging.rpc.dispatcher ObjectNotFound: error opening pool 'volumebackups'

env:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-notify-9.0.0-1.mos8583.noarch
 fuel-nailgun-9.0.0-1.mos8816.noarch
 fuelmenu-9.0.0-1.mos275.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8816.noarch
 fuel-setup-9.0.0-1.mos6355.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos350.noarch
 fuel-agent-9.0.0-1.mos290.noarch
 python-packetary-9.0.0-1.mos149.noarch
 fuel-bootstrap-cli-9.0.0-1.mos290.noarch
 fuel-ui-9.0.0-1.mos2804.noarch
 nailgun-mcagents-9.0.0-1.mos764.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-utils-9.0.0-1.mos8583.noarch
 fuel-misc-9.0.0-1.mos8583.noarch
 fuel-mirror-9.0.0-1.mos149.noarch
 fuel-ostf-9.0.0-1.mos944.noarch
 fuel-openstack-metadata-9.0.0-1.mos8816.noarch
 rubygem-astute-9.0.0-1.mos764.noarch
 fuel-release-9.0.0-1.mos6355.noarch
 fuel-library9.0-9.0.0-1.mos8583.noarch
 fuel-9.0.0-1.mos6355.noarch
 fuel-migrate-9.0.0-1.mos8583.noarch

nodes:
3 controllers, 2 computes, 2 ceph-osd with replication factor 2.

Revision history for this message
Sergei Chipiga (schipiga) wrote :
Changed in mos:
milestone: none → 9.1
importance: Undecided → High
Revision history for this message
Sergei Chipiga (schipiga) wrote :

It's not fixed still

Changed in mos:
status: New → Confirmed
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

It's a horizon and/or configuration issue. Horizon creates backups with incorrect container name. Cinder CLI works as expected

Revision history for this message
Maksym Shalamov (mshalamov) wrote :

Hello,

Issue confirmed on MOS 9.1 snapshot #315

The root cause of the issue is following:

When you try to create volume backup via horizon by default you try to store it into volumebackups container, but default container should be backups(configured by default for cinder volume backups).

Changed in mos:
assignee: Maksym Shalamov (mshalamov) → MOS Horizon (mos-horizon)
Revision history for this message
Sergei Chipiga (schipiga) wrote :

There is hardcode inside horizon:

    if not data['container_name']:
        data['container_name'] = 'volumebackups'

I seem, if horizon uses cinder client inside, that hardcode doesn't need, because cinder client will substitute actual default value, if it is absent.

Revision history for this message
Paul Karikh (pkarikh) wrote :

Yes, that's because of hardcode in Horizon.

After discussion with Ivan Kolodyazhny I think we have two ways:
1) Remove hardcode from Horizon and let Cinder (not cinderclient) substitute actual default value.
2) Add new config value to the Horizon settings. It should be syncronised with Cinder config value (which should be implemented too).

Fisrt approach is more simple and elegant and I'm going to try it.

Also since we already passed HCF for 9.1 and we have working workaround (cinder cli) I think we could move this bug to milestone 9.2, for example.

Paul Karikh (pkarikh)
Changed in mos:
milestone: 9.1 → 9.2
Revision history for this message
Paul Karikh (pkarikh) wrote :
Changed in mos:
assignee: MOS Horizon (mos-horizon) → Paul Karikh (pkarikh)
status: Confirmed → In Progress
Revision history for this message
Paul Karikh (pkarikh) wrote :
Igor Marnat (imarnat)
Changed in mos:
assignee: Paul Karikh (pkarikh) → Timur Sufiev (tsufiev-x)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Paul Karikh <email address hidden>
Review: https://review.fuel-infra.org/29054

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/horizon (stable/mitaka)

Change abandoned by Timur Sufiev <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/27333

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/horizon (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/29054
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 06f6d136a5b0997988f6c4f3661792425fe2813e
Author: Paul Karikh <email address hidden>
Date: Wed Nov 30 14:24:51 2016

Remove hardcode of container_name

If user doesn't set container name
for volume backup, Horizon sets default
value 'volumebackups'. If Cinder is configured
with different default value, this behaviour
will cause an error.

Since Cinder is able to handle case when
container name for backup is not set,
we should remove this hardcode from Horizon.

Change-Id: I091f632eae6df2d004d39eea25371ca1137d28d6
Closes-Bug: #1622561

Changed in mos:
status: In Progress → Fix Committed
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.2 snapshot #580.

tags: removed: on-verification
Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (11.0/ocata)

Fix proposed to branch: 11.0/ocata
Change author: Paul Karikh <email address hidden>
Review: https://review.fuel-infra.org/34184

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (mcp/newton)

Fix proposed to branch: mcp/newton
Change author: Paul Karikh <email address hidden>
Review: https://review.fuel-infra.org/34225

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/horizon (mcp/ocata)

Fix proposed to branch: mcp/ocata
Change author: Paul Karikh <email address hidden>
Review: https://review.fuel-infra.org/34870

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/horizon (11.0/ocata)

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34184
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/horizon (mcp/ocata)

Reviewed: https://review.fuel-infra.org/34870
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 6c847a3208344bd306036c17b2d4a92e4524c562
Author: Paul Karikh <email address hidden>
Date: Thu Apr 27 10:33:26 2017

Remove hardcode of container_name

If user doesn't set container name
for volume backup, Horizon sets default
value 'volumebackups'. If Cinder is configured
with different default value, this behaviour
will cause an error.

Since Cinder is able to handle case when
container name for backup is not set,
we should remove this hardcode from Horizon.

Change-Id: I091f632eae6df2d004d39eea25371ca1137d28d6
Closes-Bug: #1622561

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/horizon (mcp/newton)

Reviewed: https://review.fuel-infra.org/34225
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: 9b0c7d558f48f0b1a0786704cd5d6503d17600c2
Author: Paul Karikh <email address hidden>
Date: Mon May 8 12:33:59 2017

Remove hardcode of container_name

If user doesn't set container name
for volume backup, Horizon sets default
value 'volumebackups'. If Cinder is configured
with different default value, this behaviour
will cause an error.

Since Cinder is able to handle case when
container name for backup is not set,
we should remove this hardcode from Horizon.

Change-Id: I091f632eae6df2d004d39eea25371ca1137d28d6
Closes-Bug: #1622561

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.