devstack-plugin-glusterfs

GlusterFS CI for Cinder fails for Volume Backup tests

Bug #1483126 reported by Deepak C Shetty on 2015-08-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	devstack-plugin-glusterfs	Fix Committed	Critical	Deepak C Shetty

Bug Description

Issue:

GlusterFS CI for Cinder is failing consistently for all VolumeBackup related tests.
Details about the failure is will be attached to this bug. For now , these are the tests that are failing ...

2015-08-10 00:16:31.667 | tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test.test_volume_backup_create_get_detailed_list_restore_delete[id-a66eb488-8ee1-47d4-8e9f-575a095728c6]

2015-08-10 00:16:31.672 | tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test.test_volume_backup_export_import[id-a99c54a1-dd80-4724-8a13-13bf58d4068d]

2015-08-10 00:16:31.677 | tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test.test_volume_backup_create_get_detailed_list_restore_delete[id-a66eb488-8ee1-47d4-8e9f-575a095728c6]

2015-08-10 00:16:31.682 | tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test.test_volume_backup_export_import[id-a99c54a1-dd80-4724-8a13-13bf58d4068d]

All the failures are due to the below reason ...

2015-08-10 00:16:31.681 | Body: {"backup": {"status": "error", "description": null, "links": [{"href": "http://127.0.0.1:8776/v1/459b2148bb9f4c29b2c1d9a227b032e3/backups/fd093fbb-1666-49e8-ac21-b17b12a5457f", "rel": "self"}, {"href": "http://127.0.0.1:8776/459b2148bb9f4c29b2c1d9a227b032e3/backups/fd093fbb-1666-49e8-ac21-b17b12a5457f", "rel": "bookmark"}], "availability_zone": null, "volume_id": "2b68ca64-5616-41fb-b030-56d688b32c27", "fail_reason": "Backup is only supported for raw-formatted GlusterFS volumes.", "id": "fd093fbb-1666-49e8-ac21-b17b12a5457f", "size": 1, "object_count": 0, "container": null, "name": "tempest-Backup-163585353", "created_at": "2015-08-10T00:13:22.000000"}}

Root Cause:
It seems the failure started happening after the below patch was merged

https://review.openstack.org/#/c/164527/14
( GlusterFS: Using 'fallocate' instead of 'dd' )

This patch introduces a new config option nas_volume_prov_type which by default make the volume type 'thin', which means we create qcow2 files by default, this causes the volume backup tests to fail as we don't yet support backup for 'qcow2' format files, we only support backup for raw files.

Solution:
Fix is to set this new option to 'thick' in the CI job so that it creates raw files as part of the tempest runs and thus
volume backup tests should pass (again). Will post a patch soon.

Tags:

Revision history for this message

Deepak C Shetty (dpkshetty) wrote on 2015-08-10:

glusterfs cinder CI failure for volumebackup tests Edit (42.6 KiB, text/plain)

Revision history for this message

Deepak C Shetty (dpkshetty) wrote on 2015-08-10:

@Self, @Bharat,
Lesson learnt, we should have created a devstack plugin patch that depends on the above cinder fallocate patch, that ways, the CI would have kept in sycn with the changes on the Cinder side.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-10: Fix proposed to devstack-plugin-glusterfs (master)

Fix proposed to branch: master
Review: https://review.openstack.org/210999

Changed in devstack-plugin-glusterfs:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-10: Fix merged to devstack-plugin-glusterfs (master)

Reviewed: https://review.openstack.org/210999
Committed: https://git.openstack.org/cgit/stackforge/devstack-plugin-glusterfs/commit/?id=4a012088b2d3404f39066b575339bf780612c868
Submitter: Jenkins
Branch: master

commit 4a012088b2d3404f39066b575339bf780612c868
Author: Deepak C Shetty <email address hidden>
Date: Mon Aug 10 12:13:15 2015 +0530

Cinder: Make the default volume prov type as thick

    Patch @ https://review.openstack.org/#/c/164527/14 changed the default
    prov type to thin, which causes the volume backup tests in tempest
    to fail, as we don't yet support backup for thin(qcow2) files.

This patch fixes the issue by making the default prov type as thick
for the CI job

Change-Id: I9edd0e0d0a82f08305b62951a87cf45b4df8d2f7
Closes-bug: #1483126