We just had a list of backups left in the "creating" state even though cinder-backup was not working on them anymore (is there not check-loop to see if the individual backup process is still alive?).
I restarted cinder-backup to have it move all those backups to the "error" state on startup.
But some volumes were already deleted in the meantime, causing SQL error:
2023-06-21 08:24:35.353 1345602 INFO cinder.backup.manager [req-cfd983eb-a261-495f-bada-ad52ac04f866 - - - - -] Resetting backup fe1b42f6-49ee-4f24-9220-841c37c25928 to error (was creating).
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager [req-cfd983eb-a261-495f-bada-ad52ac04f866 - - - - -] Problem cleaning up backup fe1b42f6-49ee-4f24-9220-841c37c25928.: cinder.exception.VolumeNotFound: Volume f7b070c4-daea-4f0b-a8d9-95470f750e83 could not be found.
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager Traceback (most recent call last):
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 202, in _cleanup_incomplete_backup_operations
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager self._cleanup_one_backup(ctxt, backup)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 235, in _cleanup_one_backup
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager volume = objects.Volume.get_by_id(ctxt, backup.volume_id)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 339, in get_by_id
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 109, in get_by_id
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager return IMPL.get_by_id(context, model, id, *args, **kwargs)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 187, in wrapper
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager return f(*args, **kwargs)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 427, in get_by_id
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager return _GET_METHODS[model](context, id, *args, **kwargs)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 187, in wrapper
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager return f(*args, **kwargs)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 2367, in volume_get
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager return _volume_get(context, volume_id)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 187, in wrapper
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager return f(*args, **kwargs)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 2126, in _volume_get
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager raise exception.VolumeNotFound(volume_id=volume_id)
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager cinder.exception.VolumeNotFound: Volume f7b070c4-daea-4f0b-a8d9-95470f750e83 could not be found.
2023-06-21 08:24:35.359 1345602 ERROR cinder.backup.manager
We just had a list of backups left in the "creating" state even though cinder-backup was not working on them anymore (is there not check-loop to see if the individual backup process is still alive?).
I restarted cinder-backup to have it move all those backups to the "error" state on startup.
But some volumes were already deleted in the meantime, causing SQL error:
2023-06-21 08:24:35.353 1345602 INFO cinder. backup. manager [req-cfd983eb- a261-495f- bada-ad52ac04f8 66 - - - - -] Resetting backup fe1b42f6- 49ee-4f24- 9220-841c37c259 28 to error (was creating). backup. manager [req-cfd983eb- a261-495f- bada-ad52ac04f8 66 - - - - -] Problem cleaning up backup fe1b42f6- 49ee-4f24- 9220-841c37c259 28.: cinder. exception. VolumeNotFound: Volume f7b070c4- daea-4f0b- a8d9-95470f750e 83 could not be found. backup. manager Traceback (most recent call last): backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ backup/ manager. py", line 202, in _cleanup_ incomplete_ backup_ operations backup. manager self._cleanup_ one_backup( ctxt, backup) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ backup/ manager. py", line 235, in _cleanup_one_backup backup. manager volume = objects. Volume. get_by_ id(ctxt, backup.volume_id) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ objects/ base.py" , line 339, in get_by_id backup. manager orm_obj = db.get_ by_id(context, cls.model, id, *args, **kwargs) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/api. py", line 109, in get_by_id backup. manager return IMPL.get_ by_id(context, model, id, *args, **kwargs) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/sqlalchemy/ api.py" , line 187, in wrapper backup. manager return f(*args, **kwargs) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/sqlalchemy/ api.py" , line 427, in get_by_id backup. manager return _GET_METHODS[ model]( context, id, *args, **kwargs) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/sqlalchemy/ api.py" , line 187, in wrapper backup. manager return f(*args, **kwargs) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/sqlalchemy/ api.py" , line 2367, in volume_get backup. manager return _volume_ get(context, volume_id) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/sqlalchemy/ api.py" , line 187, in wrapper backup. manager return f(*args, **kwargs) backup. manager File "/usr/lib/ python3/ dist-packages/ cinder/ db/sqlalchemy/ api.py" , line 2126, in _volume_get backup. manager raise exception. VolumeNotFound( volume_ id=volume_ id) backup. manager cinder. exception. VolumeNotFound: Volume f7b070c4- daea-4f0b- a8d9-95470f750e 83 could not be found. backup. manager
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.
2023-06-21 08:24:35.359 1345602 ERROR cinder.