Cinder

nfs: failed to extend volume after blockRebase

Bug #1903319 reported by Rajat Dhasmana
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Low
Rajat Dhasmana

Bug Description

Followings are the steps to reproduce the issue:

1) create a bootable volume
2) launch an instance from the volume
3) create snapshot of instance (using nova image-create)
4) delete the snapshot (blockRebase operation)
5) delete the VM
6) try to perform extend operation on volume

it fails with the following error in cinder-volume:

ERROR cinder.volume.manager [req-5f939fc7-f4a5-4f6a-9b70-6dcb52ee7f2e req-a2702c72-fc40-41df-a600-903a5293b898 admin None] Extend volume failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img resize /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe 2G
Exit code: 1
Stdout: ''
Stderr: "qemu-img: Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': No such file or directory\n"
ERROR cinder.volume.manager Traceback (most recent call last):
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2767, in extend_volume
ERROR cinder.volume.manager self.driver.extend_volume(volume, new_size)
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 373, in extend_volume
ERROR cinder.volume.manager extend_by = int(new_size) - volume.size
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/image/image_utils.py", line 413, in resize_image
ERROR cinder.volume.manager utils.execute(*cmd, run_as_root=run_as_root)
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 126, in execute
ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
ERROR cinder.volume.manager File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 431, in execute
ERROR cinder.volume.manager raise ProcessExecutionError(exit_code=_returncode,
ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img resize /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe 2G
ERROR cinder.volume.manager Exit code: 1
ERROR cinder.volume.manager Stdout: ''
ERROR cinder.volume.manager Stderr: "qemu-img: Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': Could not open '/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-2a351c27-abb0-4965-a16e-5378bd2c2afe': No such file or directory\n"
ERROR cinder.volume.manager

stack@ubuntu2004:~/cinder$ cinder create --image 4921910a-d230-4933-b773-4af90544024f 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-06T15:18:43.000000 |
| description | None |
| encrypted | False |
| id | 2a351c27-abb0-4965-a16e-5378bd2c2afe |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | None |
| os-vol-host-attr:host | ubuntu2004@nfs#nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 90d7287d062c4ef99c0624af9023fd13 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2020-11-06T15:18:44.000000 |
| user_id | 0cd3910ea1ca40528da47cf255320a76 |
| volume_type | nfs |
+--------------------------------+--------------------------------------+
stack@ubuntu2004:~/cinder$ nova boot --flavor 42 --block-device source=volume,id=2a351c27-abb0-4965-a16e-5378bd2c2afe,size=1,shutdown=preserve,bootindex=0 ints1 --nic net-id=d6f03ab9-4849-4c4a-8d84-9d508b9c8919
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | ints1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-gfhqe76n |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | zxeQ5n3n2RaE |
| config_drive | |
| created | 2020-11-06T15:19:07Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw_rng:allowed": "True"} |
| flavor:original_name | m1.nano |
| flavor:ram | 128 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| host_status | |
| id | 3a922fe9-bae1-4d80-aa96-654ed0b8b110 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | ints1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 90d7287d062c4ef99c0624af9023fd13 |
| trusted_image_certificates | - |
| updated | 2020-11-06T15:19:06Z |
| user_id | 0cd3910ea1ca40528da47cf255320a76 |
+--------------------------------------+-------------------------------------------------+
stack@ubuntu2004:~/cinder$ nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------------------------+
| 3a922fe9-bae1-4d80-aa96-654ed0b8b110 | ints1 | ACTIVE | - | Running | private=fdc8:7eef:b74b:0:f816:3eff:fe75:196d, 10.0.0.48 |
+--------------------------------------+-------+--------+------------+-------------+---------------------------------------------------------+
stack@ubuntu2004:~/cinder$ nova image-create 3a922fe9-bae1-4d80-aa96-654ed0b8b110 snap1
cinder snapshstack@ubuntu2004:~/cinder$ cinder snapshot-list
cinder snapsho+--------------------------------------+--------------------------------------+-----------+--------------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+-----------+--------------------+------+
| 546e3016-8f6f-4e13-9099-53c26d454890 | 2a351c27-abb0-4965-a16e-5378bd2c2afe | available | snapshot for snap1 | 1 |
+--------------------------------------+--------------------------------------+-----------+--------------------+------+
tstack@ubuntu2004:~/cinder$ cinder snapshot-delete 546e3016-8f6f-4e13-9099-53c26d454890
stack@ubuntu2004:~/cinder$ cinder snapshot-list
+----+-----------+--------+------+------+
| ID | Volume ID | Status | Name | Size |
+----+-----------+--------+------+------+
+----+-----------+--------+------+------+
stack@ubuntu2004:~/cinder$ nova delete 3a922fe9-bae1-4d80-aa96-654ed0b8b110
Request to delete server 3a922fe9-bae1-4d80-aa96-654ed0b8b110 has been accepted.
stack@ubuntu2004:~/cinder$ nova list
cinder +----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
lstack@ubuntu2004:~/cinder$ cinder list
cinde+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 2a351c27-abb0-4965-a16e-5378bd2c2afe | available | - | 1 | nfs | true | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
stack@ubuntu2004:~/cinder$ cinder extend 2a351c27-abb0-4965-a16e-5378bd2c2afe 2
stack@ubuntu2004:~/cinder$ cinder list
+--------------------------------------+-----------------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------------+------+------+-------------+----------+-------------+
| 2a351c27-abb0-4965-a16e-5378bd2c2afe | error_extending | - | 1 | nfs | true | |
+--------------------------------------+-----------------+------+------+-------------+----------+-------------+

Tags: extend nfs volume
michael-mcaleer (mmcaleer)
Changed in cinder:
status: New → Triaged
Rajat Dhasmana (whoami-rajat)
Changed in cinder:
assignee: nobody → Rajat Dhasmana (whoami-rajat)
michael-mcaleer (mmcaleer)
tags: added: extend nfs volume
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/762462

Changed in cinder:
status: Triaged → In Progress
Revision history for this message
Lucio Seki (lseki) wrote :

Hi Rajat, I reproduced the issue with NetApp ONTAP NFS driver [0].

The error message is slightly different, though [1]:

---
Stderr: "qemu-img: Could not open '/opt/stack/data/cinder/mnt/83c3167af2464949d83cddf33977804e/volume-ac6a5bbd-ae96-490c-a1f9-04f744d5e4ba': Could not open '/
opt/stack/data/cinder/mnt/83c3167af2464949d83cddf33977804e/volume-ac6a5bbd-ae96-490c-a1f9-04f744d5e4ba': Permission denied\n".
---

[0] http://paste.openstack.org/show/799996/
[1] http://paste.openstack.org/show/799997/

Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

Hi Lucio,

Looks pretty much the same issue.
Can you manually check if the volume file it's trying to access exists in the directory /opt/stack/data/cinder/mnt/<share-hash>/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/763181

Revision history for this message
Lucio Seki (lseki) wrote :

Hi Rajat, the file does exist [0].

The failing qemu-img info command succeeds when manually executed as root [1].

[0] http://paste.openstack.org/show/800387/
[1] http://paste.openstack.org/show/800388/

Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

The code seems strange to me for deleting snapshot, It doesn't check if the volume is attached, doesn't perform any rebase of data in the base file to the snapshot file while deleting. It is just locating the file and deleting it.[1]
Looks very wrong to me or I'm just missing something here.

[1] https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/netapp/dataontap/nfs_base.py#L266-L280

Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Should we move this to won't fix?

Changed in cinder:
importance: Undecided → Low
Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

Hi Sofia,

We already have a fix proposed[1] and IMO this is an important issue since we don't keep track of nfs file changing names. So if we attach, perform snapshot operations, detach, then cinder won't be able to locate the volume file.

[1] https://review.opendev.org/c/openstack/cinder/+/763181/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.