Gnocchi

Aggregation start/stop flags with timezone %2B or + issue

Bug #1523549 reported by Gaël Lambert on 2015-12-07

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Gnocchi	Fix Released	High	Julien Danjou	Gnocchi 2.0.0
	1.3	Fix Released	High	Julien Danjou	Gnocchi 1.3.4

Bug Description

Refering #1522434

I send datas with my timezone "2015-12-03T13:19:15+0100". In gnocchi this point is converted to UTC "2015-12-03T12:19:15+00:00".

But for the start and stop flag in aggregat. To get my value added with 13:19:15+0100, I need to query start 12:19:15.

It's not a big deal but I added value with timezone, using gnocchi, I was expected to be able to query aggregate range with timezone also.

I tried to give the timezone with (+ encoded to %2B)
curl -H "Accept: application/json" -H "Content-Type: application/json" "http://localhost:8041/v1/aggregation/metric?metric=$metric1&metric=$metric2&start=2015-12-03T13:19:15%2B0100&stop=2015-12-03T13:30:15%2B0100&needed_overlap=0&aggregation=sum" | json_pp
[]

And keeping the + char :
curl -H "Accept: application/json" -H "Content-Type: application/json" "http://localhost:8041/v1/aggregation/metric?metric=$metric1&metric=$metric2&start=2015-12-03T13:19:15+0100&stop=2015-12-03T13:30:15+0100&needed_overlap=0&aggregation=sum" | json_pp

   File "/share/gnocchi/gnocchi/carbonara.py", line 388, in aggregated
  timeserie_raw = timeserie.fetch(from_timestamp, to_timestamp)
   File "/share/gnocchi/gnocchi/carbonara.py", line 344, in fetch
  points = self[from_timestamp:to_timestamp]
   File "/share/gnocchi/gnocchi/carbonara.py", line 97, in __getitem__
  return self.ts[key]
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 597, in __getitem__
  return self._get_with(key)
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 602, in _get_with
  indexer = self.index._convert_slice_indexer(key, kind='getitem')
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 950, in _convert_slice_indexer
  key, is_index_slice=is_index_slice)
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 885, in _convert_slice_indexer_getitem
  return self._convert_slice_indexer(key)
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 972, in _convert_slice_indexer
  indexer = self.slice_indexer(start, stop, step)
   File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.py", line 1436, in slice_indexer
  return Index.slice_indexer(self, start, end, step)
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 2570, in slice_indexer
  start_slice, end_slice = self.slice_locs(start, end, step=step, kind=kind)
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 2714, in slice_locs
  start_slice = self.get_slice_bound(start, 'left', kind)
   File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 2652, in get_slice_bound
  label = self._maybe_cast_slice_bound(label, side, kind)
   File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.py", line 1396, in _maybe_cast_slice_bound
  _, parsed, reso = parse_time_string(label, freq)
   File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/tools.py", line 478, in parse_time_string
  yearfirst=yearfirst)
   File "pandas/tslib.pyx", line 1651, in pandas.tslib.parse_datetime_string_with_reso (pandas/tslib.c:29906)

DateParseError: Unknown datetime string format, unable to parse: 2015-12-03T13:19:15 0100

Julien Danjou (jdanjou) on 2015-12-09

Changed in gnocchi:
status:	New → Triaged
importance:	Undecided → Wishlist

Julien Danjou (jdanjou) on 2015-12-18

Changed in gnocchi:
importance:	Wishlist → Low

Julien Danjou (jdanjou) on 2015-12-18

Changed in gnocchi:
assignee:	nobody → Julien Danjou (jdanjou)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-12-18: Fix proposed to gnocchi (master)

Fix proposed to branch: master
Review: https://review.openstack.org/259394

Changed in gnocchi:
status:	Triaged → In Progress

Revision history for this message

Julien Danjou (jdanjou) wrote on 2015-12-18:

I've pushed a patch with a test that shows that it works correctly with %2B. In your example here, with %2B version, it converts to 12:30:15 UTC, but you talk about points at 12:19, so maybe your start value was indeed invalid. :)

Julien Danjou (jdanjou) on 2016-01-05

no longer affects:

gnocchi/1.3

Revision history for this message

Gaël Lambert (gael-lambert) wrote on 2016-01-05:

In my exemple, I tried with :

start=2015-12-03T13:19:15%2B0100
stop=2015-12-03T13:30:15%2B0100

start=2015-12-03T13:19:15+0100
stop=2015-12-03T13:30:15+0100

The range seems good for me regarding the data from https://bugs.launchpad.net/gnocchi/+bug/1522434/comments/5

curl -H "Accept: application/json" -H "Content-Type: application/json" "http://localhost:8041/v1/aggregation/metric?metric=$metric1&metric=$metric2&start=2015-12-03T12:19:15&stop=2015-12-03T12:25:15&needed_overlap=0&aggregation=sum" | json_pp
[
    [
    "2015-12-03T12:19:15+00:00",
    1,
    1
    ],
    [
    "2015-12-03T12:20:15+00:00",
    1,
    1
    ],
    [
    "2015-12-03T12:21:15+00:00",
    1,
    11
    ],
    [
    "2015-12-03T12:22:15+00:00",
    1,
    11
    ],
    [
    "2015-12-03T12:23:15+00:00",
    1,
    10
    ],
    [
    "2015-12-03T12:24:15+00:00",
    1,
    10
    ]
]

But for this test I don't use stop=2015-12-03T12:25:15 (25) but stop=2015-12-03T13:30:15%2B0100
(30) a bigger range

If I have time this weekend, I will check again to validate that.

Thanks

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-07: Fix merged to gnocchi (master)

Reviewed: https://review.openstack.org/259394
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=cbd6b38fcbac3ac060a39ac99dab134f779bdd22
Submitter: Jenkins
Branch: master

commit cbd6b38fcbac3ac060a39ac99dab134f779bdd22
Author: Julien Danjou <email address hidden>
Date: Fri Dec 18 12:37:44 2015 +0100

metric: make sure start/stop can be used with timezone

Change-Id: I374a2b16cece3c6b464af9d5a4de214150402716
Closes-Bug: #1523549

Changed in gnocchi:
status:	In Progress → Fix Committed

Revision history for this message

Gaël Lambert (gael-lambert) wrote on 2016-01-08:

I tried with last code from master and still have [].

(all commands I used to reproduce my issue https://paste.netwiki.fr/?7ac22e20e4a4b358#taQ+P5VZwutW3w1VLtCYIYaq3pSMJXfx8JoifLumyIQ=)

I posted measures with "2015-12-03T13:19:15+0100". I can found them, example for metric1 :

    curl -H "Accept: application/json" -H "Content-Type: application/json" "http://localhost:8041/v1/metric/$metric1/measures" | json_pp
    [
       [
          "2015-12-03T12:19:15+00:00",
          1,
          1
       ],
       [
          "2015-12-03T12:20:15+00:00",
          1,
          1
       ],
       [
          "2015-12-03T12:21:15+00:00",
          1,
          1
       ],
       [
          "2015-12-03T12:22:15+00:00",
          1,
          1
       ]
    ]

So internally converted to "2015-12-03T12:19:15+00:00"

Now I tried to get aggregate with :
start=2015-12-03T13:00:15%2B0100 (2015-12-03T13:00:15+0100)
stop=2015-12-03T13:30:15%2B0100 (2015-12-03T13:30:15+0100)

Seems don't work. If I change start by start=2015-12-03T12:00:15%2B0100 (2015-12-03T12:00:15+0100) it works :

    curl -H "Accept: application/json" -H "Content-Type: application/json" "http://localhost:8041/v1/aggregation/metric?metric=$metric1&metric=$metric2&start=2015-12-03T12:00:15%2B0100&stop=2015-12-03T13:30:15%2B0100&needed_overlap=0&aggregation=sum" | json_pp
    [
       [
          "2015-12-03T12:19:15+00:00",
          1,
          1
       ],
...
       [
          "2015-12-03T12:24:15+00:00",
          1,
          10
       ]
    ]

Like if %2B0100 (+0100) was ignored. Did I miss something ?

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-11: Fix proposed to gnocchi (master)

Fix proposed to branch: master
Review: https://review.openstack.org/265824

Revision history for this message

Julien Danjou (jdanjou) wrote on 2016-01-11:

Thank for digging Gaël. I realize I only checked the non-aggregate part in my tests, and that is a different code path. So there's indeed a bug, where start and stop were not parsed at all by the API, and thus passed a string to Carbonara and Pandas, which were not able to read the timezone properly I guess. I've fixed that in https://review.openstack.org/265824

Changed in gnocchi:
status:	Fix Committed → In Progress
importance:	Low → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-26: Fix proposed to gnocchi (stable/1.3)

Fix proposed to branch: stable/1.3
Review: https://review.openstack.org/272433

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-26: Fix merged to gnocchi (master)

Reviewed: https://review.openstack.org/265824
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=7df853e61b6fc7feff206e95b542d51284220a62
Submitter: Jenkins
Branch: master

commit 7df853e61b6fc7feff206e95b542d51284220a62
Author: Julien Danjou <email address hidden>
Date: Mon Jan 11 14:43:10 2016 +0100

rest: fix timestamp parsing for aggregates

Currently, start/stop are not parsed as Timestamp like in the rest of
the measures retrieve requests. This patches fixes that.

Change-Id: If7ac90f9a01cffa4a09c2a9e267afe8be4dfe2c5
Closes-Bug: #1523549

Changed in gnocchi:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-26: Fix merged to gnocchi (stable/1.3)

#10

Reviewed: https://review.openstack.org/272433
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=6f74a33bec8e35edcaa898b680fff3b71bcc12b4
Submitter: Jenkins
Branch: stable/1.3

commit 6f74a33bec8e35edcaa898b680fff3b71bcc12b4
Author: Julien Danjou <email address hidden>
Date: Mon Jan 11 14:43:10 2016 +0100

rest: fix timestamp parsing for aggregates

Currently, start/stop are not parsed as Timestamp like in the rest of
the measures retrieve requests. This patches fixes that.

Closes-Bug: #1523549

(partially cherry-picked from 7df853e61b6fc7feff206e95b542d51284220a62)

Change-Id: Ie60f8f8140675964bb058ed733dfa7958eddd419

Julien Danjou (jdanjou) on 2016-02-19

Changed in gnocchi:
milestone:	none → 2.0.0
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1537792

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.