occasion error reading unprocessed measure

Bug #1595519 reported by gordon chung
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi
Incomplete
Medium
Unassigned

Bug Description

i get this error maybe once every few thousands metrics. it is not consistent but not rare. just to describe my test pattern, i post 10 measures for ~20k metrics... and then 10 minute later, i post 10 new measures for same 20k metrics. between posts and processing, services do sit idle for a bit.

2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara [-] Error processing new measures
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara Traceback (most recent call last):
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara File "/root/dev/gnocchi/gnocchi/storage/_carbonara.py" line 354 in process_measures
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara with self._process_measure_for_metric(metric) as measures:
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara File "/usr/lib64/python2.7/contextlib.py" line 17 in __enter__
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara return self.gen.next()
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara File "/root/dev/gnocchi/gnocchi/storage/ceph.py" line 219 in _process_measure_for_metric
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara data = self._get_object_content(ctx n)
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara File "/root/dev/gnocchi/gnocchi/storage/ceph.py" line 333 in _get_object_content
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara data = ioctx.read(name offset=offset)
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara File "rados.pyx" line 425 in rados.requires.wrapper.validate_func (rados.c:4106)
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara File "rados.pyx" line 2281 in rados.Ioctx.read (rados.c:25439)
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara ObjectNotFound: Ioctx.read(gnocchi): failed to read measure_fd407743-db04-42d3-a129-9a3a5b36f7dd_fb931995-e01a-4943-9d72-a2f0d67783fc_20160623_01:00:22
2016-06-22 21:06:34.828 18484 ERROR gnocchi.storage._carbonara

Julien Danjou (jdanjou)
Changed in gnocchi:
status: New → Triaged
Revision history for this message
Julien Danjou (jdanjou) wrote :

Looks like a locking issue where you would have 2 metricd processing the same measure at the same time.

Changed in gnocchi:
status: Triaged → Incomplete
importance: Undecided → Medium
Revision history for this message
gordon chung (chungg) wrote :

yeah, i think it's more a tooz thing. or just the fact that i'm using redis as my coordination driver.

Revision history for this message
Julien Danjou (jdanjou) wrote :

Do you still see this gordon?

Revision history for this message
gordon chung (chungg) wrote :

yep. i have no idea exactly what triggers it though

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.