Comment 48 for bug 1806156

We upgraded openstack-ansible from rocky to stain and cinder-manage also fails (see below to fix/workaround).

~# cinder-manage --debug db online_data_migrations --max_count 1
 ; echo $?
Error attempting to run volume_service_uuids_online_data_migration
+--------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+--------------------------------------------+--------------+-----------+
| attachment_specs_online_data_migration | 0 | 0 |
| backup_service_online_migration | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
| shared_targets_online_data_migration | 0 | 0 |
| volume_service_uuids_online_data_migration | 0 | 0 |
+--------------------------------------------+--------------+-----------+
Some migrations failed unexpectedly. Check log for details.
2

~# journalctl --since "1 hour ago" | grep cinder.cmd.manage | grep Key
Aug 07 13:59:26 ik01-cinder-api-container-ebba9597 cinder-manage[326]: 2019-08-07 13:59:26.692 326 ERROR cinder.cmd.manage [req-d0a34065-a288-4b2b-9fad-1b915c1f4eea - - - - -] Error attempting to run volume_service_uuids_online_data_migration: KeyError: u'ik03@rbd-ec'

In our case database was inconsistent eg. 'service_uuid' field was sometimes NULL:
MariaDB [cinder]> select id,volume_type_id,status,service_uuid from volumes where host like '%ik03@rbd-ec%' and status!='deleted' and service_uuid is NULL;

Our workaround/fix was:
MariaDB [cinder]> update volumes set service_uuid='81d03e44-21ab-41ea-81b6-a3b5aac8a3b6' where host like '%ik03@rbd-ec%' and status!='deleted' and service_uuid is NULL;

~# /openstack/venvs/cinder-19.0.0/bin/cinder-manage --debug db online_data_migrations ; echo $?
Running batches of 50 until complete.
49 rows matched query volume_service_uuids_online_data_migration, 49 migrated
+--------------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+--------------------------------------------+--------------+-----------+
| attachment_specs_online_data_migration | 0 | 0 |
| backup_service_online_migration | 0 | 0 |
| service_uuids_online_data_migration | 0 | 0 |
| shared_targets_online_data_migration | 0 | 0 |
| volume_service_uuids_online_data_migration | 49 | 49 |
+--------------------------------------------+--------------+-----------+
0

Hope it helps someone to save few hours.