Multiple gnocchi-metricd with Redis show "ObjectNotFound"

Bug #1537767 reported by Nicolas Vila
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi
Fix Released
High
Julien Danjou
1.3
Fix Released
High
Julien Danjou

Bug Description

I have four separate gnocchi-metricd instances (branch master, pulled from git on Jan 21) with a single redis-server for coordination, the log files show many of the following errors:

-----
2016-01-25 13:09:11.839 22257 DEBUG gnocchi.storage [-] Processing new and to delete measures process_background_tasks /usr/local/lib/python2.7/dist-packages/gnocchi/storage/__init__.py:182
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage [-] Unexpected error during measures processing
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage Traceback (most recent call last):
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/__init__.py", line 184, in process_background_tasks
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage self.process_measures(index, sync)
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 170, in process_measures
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage self._delete_unprocessed_measures_for_metric_id(metric_id)
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/ceph.py", line 106, in _delete_unprocessed_measures_for_metric_id
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage ctx.remove_object(n)
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage File "/usr/lib/python2.7/dist-packages/rados.py", line 1561, in remove_object
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage raise make_ex(ret, "Failed to remove '%s'" % key)
2016-01-25 13:33:22.405 22257 ERROR gnocchi.storage ObjectNotFound: Failed to remove 'measure_0dccfff7-30ff-4f63-8e86-ae9ee5f17b68_0018f4fa-90c8-478c-b64a-1aa681829caa_20162912_21:29:54'

-----

2016-01-25 11:26:10.444 22257 DEBUG gnocchi.storage._carbonara [-] Processing measures for 0d6be6f7-f49b-4611-add5-19fd74f52d67 process_measures /usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py:184
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara [-] Error processing new measures
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara Traceback (most recent call last):
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/_carbonara.py", line 185, in process_measures
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara with self._process_measure_for_metric(metric) as measures:
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara return self.gen.next()
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/ceph.py", line 117, in _process_measure_for_metric
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara data = self._get_object_content(ctx, n)
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara File "/usr/local/lib/python2.7/dist-packages/gnocchi/storage/ceph.py", line 215, in _get_object_content
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara data = ioctx.read(name, offset=offset)
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara File "/usr/lib/python2.7/dist-packages/rados.py", line 1489, in read
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara raise make_ex(ret, "Ioctx.read(%s): failed to read %s" % (self.name, key))
2016-01-25 11:46:24.291 22257 ERROR gnocchi.storage._carbonara ObjectNotFound: Ioctx.read(gnocchi): failed to read measure_0d6be6f7-f49b-4611-add5-19fd74f52d67_003ddbac-44f2-4e7b-9bbe-97cad254ce2b_20160015_08:00:46

I'm using tooz installed from git (version stable/kilo, pulled on Jan 20).

Please advice. Thanks, regards.

Revision history for this message
Nicolas Vila (nvlan) wrote :
Julien Danjou (jdanjou)
Changed in gnocchi:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Julien Danjou (jdanjou)
Changed in gnocchi:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to gnocchi (master)

Reviewed: https://review.openstack.org/272475
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=115c275a667bf60c694d02f12800de52c857593e
Submitter: Jenkins
Branch: master

commit 115c275a667bf60c694d02f12800de52c857593e
Author: Julien Danjou <email address hidden>
Date: Tue Jan 26 12:18:43 2016 +0100

    ceph, swift: do not fail if unprocess measures already deleted

    If several workers are trying to delete the same unprocessed metric at
    the same time, they might fail to do so. Ignore these error, as it's not
    a big deal in the end: they're deleted.

    Closes-Bug: #1537767
    Change-Id: Ib8002b757698a8c6601bc63614cc0c63a18f2e83

Changed in gnocchi:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to gnocchi (stable/1.3)

Fix proposed to branch: stable/1.3
Review: https://review.openstack.org/272996

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to gnocchi (stable/1.3)

Reviewed: https://review.openstack.org/272996
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=cffcc4d5082c29efc789f0cf4bc81100b82c2d33
Submitter: Jenkins
Branch: stable/1.3

commit cffcc4d5082c29efc789f0cf4bc81100b82c2d33
Author: Julien Danjou <email address hidden>
Date: Tue Jan 26 12:18:43 2016 +0100

    ceph, swift: do not fail if unprocess measures already deleted

    If several workers are trying to delete the same unprocessed metric at
    the same time, they might fail to do so. Ignore these error, as it's not
    a big deal in the end: they're deleted.

    Closes-Bug: #1537767
    Change-Id: Ib8002b757698a8c6601bc63614cc0c63a18f2e83
    (cherry picked from commit 115c275a667bf60c694d02f12800de52c857593e)

Julien Danjou (jdanjou)
Changed in gnocchi:
milestone: none → 2.0.0
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.