Hi,
While using restoring a volume from ceph -> Nimble storage array with a target volume larger than the original backup the padding fails due to a TypeError:
2021-04-23 09:43:02.208 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Transferred chunk 63 of 80 (476724K/s) _transfer_data /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:418
2021-04-23 09:43:02.486 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Transferred chunk 64 of 80 (472558K/s) _transfer_data /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:418
2021-04-23 09:43:02.494 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Discarding 2147483648 bytes from offset 8589934592 _discard_bytes /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:364
2021-04-23 09:43:02.567 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Writing zeroes chunk 0 _discard_bytes /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:379
2021-04-23 09:43:02.594 20 DEBUG oslo_concurrency.processutils [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:379
2021-04-23 09:43:02.802 20 DEBUG oslo_concurrency.processutils [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc" returned: 0 in 0.208s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:416
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Restoring backup 36da785e-9e59-4d70-8dd9-058bb970b40f to volume a61e038b-999d-4446-9624-cfe4c1efca63 failed.: TypeError: a bytes-like object is required, not 'str'
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager Traceback (most recent call last):
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 654, in _run_restore
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager tpool.Proxy(device_file))
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 1269, in restore
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager self._restore_volume(backup, target_volume, volume_file)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 1236, in _restore_volume
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager length, src_snap=restore_point)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 1038, in _full_restore
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager dest_file, dest_name, length)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 406, in _transfer_data
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager length - dest.tell())
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 380, in _discard_bytes
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager volume.write(zeroes)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager rv = execute(f, *args, **kwargs)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager six.reraise(c, e, tb)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager raise value
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager rv = meth(*args, **kwargs)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager TypeError: a bytes-like object is required, not 'str'
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager
https://github.com/openstack/cinder/blob/stable/ussuri/cinder/backup/drivers/ceph.py#L372
This can be fixed by encoding the zero stream like:
else:
zeroes = '\0' * self.chunk_size
zeroes = zeroes.encode('utf-8')
though I'm not 100% sure if that does anything untowards or properly zeroes the remaining space.
The cinder version this is on:
cinder (16.2.1) (though I believe it will affect all versions including master)
Looks like a RBD related issue, i'll take a look.