Comment 0 for bug 1588404

I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:

2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.

The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).

swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%

I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.

And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.

It's not clear how to debug this should it happen in a live system again.