Multiple gnocchi-metricd with Redis show "ObjectNotFound"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Gnocchi |
Fix Released
|
High
|
Julien Danjou | ||
1.3 |
Fix Released
|
High
|
Julien Danjou |
Bug Description
I have four separate gnocchi-metricd instances (branch master, pulled from git on Jan 21) with a single redis-server for coordination, the log files show many of the following errors:
-----
2016-01-25 13:09:11.839 22257 DEBUG gnocchi.storage [-] Processing new and to delete measures process_
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage [-] Unexpected error during measures processing
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage Traceback (most recent call last):
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/local/
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage self.process_
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/local/
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage self._delete_
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/local/
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage ctx.remove_
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/lib/
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage raise make_ex(ret, "Failed to remove '%s'" % key)
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage ObjectNotFound: Failed to remove 'measure_
-----
2016-01-25 11:26:10.444 22257 DEBUG gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
2016-01-25 11:46:24.291 22257 ERROR gnocchi.
I'm using tooz installed from git (version stable/kilo, pulled on Jan 20).
Please advice. Thanks, regards.
Changed in gnocchi: | |
status: | New → Triaged |
importance: | Undecided → High |
assignee: | nobody → Julien Danjou (jdanjou) |
Changed in gnocchi: | |
status: | Triaged → In Progress |
Changed in gnocchi: | |
milestone: | none → 2.0.0 |
status: | Fix Committed → Fix Released |
Reviewed: https:/ /review. openstack. org/272475 /git.openstack. org/cgit/ openstack/ gnocchi/ commit/ ?id=115c275a667 bf60c694d02f128 00de52c857593e
Committed: https:/
Submitter: Jenkins
Branch: master
commit 115c275a667bf60 c694d02f12800de 52c857593e
Author: Julien Danjou <email address hidden>
Date: Tue Jan 26 12:18:43 2016 +0100
ceph, swift: do not fail if unprocess measures already deleted
If several workers are trying to delete the same unprocessed metric at
the same time, they might fail to do so. Ignore these error, as it's not
a big deal in the end: they're deleted.
Closes-Bug: #1537767 c6601bc63614cc0 c63a18f2e83
Change-Id: Ib8002b757698a8