| 2016-06-02 15:04:25 |
Andreas Hasenack |
bug |
|
|
added bug |
| 2016-06-02 15:04:25 |
Andreas Hasenack |
attachment added |
|
dashboard-swift-jump.png https://bugs.launchpad.net/bugs/1588404/+attachment/4675268/+files/dashboard-swift-jump.png |
|
| 2016-06-02 15:04:46 |
Andreas Hasenack |
attachment added |
|
swift-recon-full.txt https://bugs.launchpad.net/landscape-client/+bug/1588404/+attachment/4675269/+files/swift-recon-full.txt |
|
| 2016-06-02 15:05:01 |
Andreas Hasenack |
attachment added |
|
swift-df.txt https://bugs.launchpad.net/landscape-client/+bug/1588404/+attachment/4675270/+files/swift-df.txt |
|
| 2016-06-02 15:07:37 |
Andreas Hasenack |
description |
I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).
swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%
I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.
And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.
It's not clear how to debug this should it happen in a live system again. |
I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).
swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%
This was all seen several hours after the deployment finished, almost half a day.
I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.
And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.
It's not clear how to debug this should it happen in a live system again. |
|
| 2016-06-02 15:08:58 |
Andreas Hasenack |
description |
I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).
swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%
This was all seen several hours after the deployment finished, almost half a day.
I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.
And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.
It's not clear how to debug this should it happen in a live system again. |
I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).
swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%
This was all seen several hours after the deployment finished, almost half a day.
I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.
monitor log covering the time when it was broken, and after my restart where at first I ran it in the foreground, and then in the background with a shorter reporting interval:
# grep Swift monitor.log
2016-06-01 22:08:46,449 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-01 23:08:46,451 WARNING [MainThread] 1 of 720 expected Swift device usage snapshot events (0.14%) occurred in the last 3600.00s.
2016-06-02 00:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 01:08:46,451 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 03:08:46,451 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 04:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 05:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 06:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 07:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 08:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 09:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 10:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 11:08:46,450 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 12:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 12:54:05,236 WARNING [MainThread] 0 of 543 expected Swift device usage snapshot events (0.00%) occurred in the last 2718.79s.
2016-06-02 12:57:02,272 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 12:57:29,322 INFO [MainThread] 5 of 5 expected Swift device usage snapshot events (100.00%) occurred in the last 27.05s.
2016-06-02 12:58:10,375 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 13:02:04,883 INFO [MainThread] 46 of 46 expected Swift device usage snapshot events (100.00%) occurred in the last 234.51s.
2016-06-02 13:02:07,217 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 13:04:07,218 INFO [MainThread] 23 of 24 expected Swift device usage snapshot events (95.83%) occurred in the last 120.00s.
2016-06-02 13:06:07,218 INFO [MainThread] 24 of 23 expected Swift device usage snapshot events (104.35%) occurred in the last 120.00s.
And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.
It's not clear how to debug this should it happen in a live system again. |
|
| 2016-06-02 15:11:39 |
Andreas Hasenack |
attachment added |
|
juju-status-tabular.txt https://bugs.launchpad.net/landscape-client/+bug/1588404/+attachment/4675285/+files/juju-status-tabular.txt |
|
| 2016-06-02 15:13:04 |
Andreas Hasenack |
description |
I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).
swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%
This was all seen several hours after the deployment finished, almost half a day.
I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.
monitor log covering the time when it was broken, and after my restart where at first I ran it in the foreground, and then in the background with a shorter reporting interval:
# grep Swift monitor.log
2016-06-01 22:08:46,449 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-01 23:08:46,451 WARNING [MainThread] 1 of 720 expected Swift device usage snapshot events (0.14%) occurred in the last 3600.00s.
2016-06-02 00:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 01:08:46,451 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 03:08:46,451 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 04:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 05:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 06:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 07:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 08:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 09:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 10:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 11:08:46,450 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 12:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 12:54:05,236 WARNING [MainThread] 0 of 543 expected Swift device usage snapshot events (0.00%) occurred in the last 2718.79s.
2016-06-02 12:57:02,272 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 12:57:29,322 INFO [MainThread] 5 of 5 expected Swift device usage snapshot events (100.00%) occurred in the last 27.05s.
2016-06-02 12:58:10,375 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 13:02:04,883 INFO [MainThread] 46 of 46 expected Swift device usage snapshot events (100.00%) occurred in the last 234.51s.
2016-06-02 13:02:07,217 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 13:04:07,218 INFO [MainThread] 23 of 24 expected Swift device usage snapshot events (95.83%) occurred in the last 120.00s.
2016-06-02 13:06:07,218 INFO [MainThread] 24 of 23 expected Swift device usage snapshot events (104.35%) occurred in the last 120.00s.
And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.
It's not clear how to debug this should it happen in a live system again. |
landscape-client 16.04~bzr841-0ubuntu0~ubuntu14.04.1
I had a ceph/swift cloud deploy where for some reason just 1/3 of the swift units were reporting swift data. Two of them were saying this:
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
The openstack dashboard in landscape was reporting just 1/3 of the swift storage (see screenshot).
swift recon was showing all the storage (also see attached):
Disk usage: space used: 1996472320 of 199605878784
Disk usage: space free: 197609406464 of 199605878784
Disk usage: lowest: 0.11%, highest: 3.48%, avg: 1.00020717434%
This was all seen several hours after the deployment finished, almost half a day.
I then decided to restart landscape-client in the foreground, to see if there were any backtraces (that's the usual trick, because backtraces in the swift plugin are lost, see bug #1563565). To my surprise, the swift plugin started reporting data.
monitor log covering the time when it was broken, and after my restart where at first I ran it in the foreground, and then in the background with a shorter reporting interval:
# grep Swift monitor.log
2016-06-01 22:08:46,449 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-01 23:08:46,451 WARNING [MainThread] 1 of 720 expected Swift device usage snapshot events (0.14%) occurred in the last 3600.00s.
2016-06-02 00:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 01:08:46,451 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 02:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 03:08:46,451 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 04:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 05:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 06:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 07:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 08:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 09:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 10:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 11:08:46,450 WARNING [MainThread] 0 of 720 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 12:08:46,450 WARNING [MainThread] 0 of 719 expected Swift device usage snapshot events (0.00%) occurred in the last 3600.00s.
2016-06-02 12:54:05,236 WARNING [MainThread] 0 of 543 expected Swift device usage snapshot events (0.00%) occurred in the last 2718.79s.
2016-06-02 12:57:02,272 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 12:57:29,322 INFO [MainThread] 5 of 5 expected Swift device usage snapshot events (100.00%) occurred in the last 27.05s.
2016-06-02 12:58:10,375 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 13:02:04,883 INFO [MainThread] 46 of 46 expected Swift device usage snapshot events (100.00%) occurred in the last 234.51s.
2016-06-02 13:02:07,217 INFO [MainThread] Registering plugin landscape.monitor.swiftusage.SwiftUsage.
2016-06-02 13:04:07,218 INFO [MainThread] 23 of 24 expected Swift device usage snapshot events (95.83%) occurred in the last 120.00s.
2016-06-02 13:06:07,218 INFO [MainThread] 24 of 23 expected Swift device usage snapshot events (104.35%) occurred in the last 120.00s.
And indeed, after I restarted the clients on the two broken units, it all worked as it should. You can see the jump in the graph in the attached screenshot.
It's not clear how to debug this should it happen in a live system again. |
|
| 2016-06-02 15:15:41 |
Andreas Hasenack |
attachment added |
|
swift-landscape-logs.tar.bz2 https://bugs.launchpad.net/landscape-client/+bug/1588404/+attachment/4675306/+files/swift-landscape-logs.tar.bz2 |
|
| 2016-06-02 15:18:15 |
🤖 Landscape Builder |
tags |
kanban |
|
|
| 2016-07-15 15:31:57 |
Simon Poirier |
landscape-client: assignee |
|
Simon Poirier (simpoir) |
|
| 2016-07-18 17:29:17 |
Simon Poirier |
landscape-client: status |
New |
In Progress |
|
| 2016-07-21 16:26:56 |
Simon Poirier |
branch linked |
|
lp:~simpoir/landscape-client/bug_1588404_swift_usage_report |
|
| 2016-07-25 15:43:06 |
🤖 Landscape Builder |
landscape-client: status |
In Progress |
Fix Committed |
|
| 2018-02-22 20:56:56 |
Andreas Hasenack |
bug task added |
|
landscape-client (Ubuntu) |
|
| 2018-02-22 20:57:02 |
Andreas Hasenack |
landscape-client (Ubuntu): assignee |
|
Andreas Hasenack (ahasenack) |
|
| 2018-02-22 20:57:05 |
Andreas Hasenack |
landscape-client (Ubuntu): status |
New |
In Progress |
|
| 2018-02-28 00:08:20 |
Launchpad Janitor |
landscape-client (Ubuntu): status |
In Progress |
Fix Released |
|
