Fuel for OpenStack

OSTF 'Ceilometer test to list meters, alarms, resources and events' fails with InternalServerError

Bug #1611023 reported by ElenaRossokhina
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Ilya Tyaptin
Fuel for OpenStack 11.0
Mitaka
Fix Released
High
Ilya Tyaptin
Fuel for OpenStack 9.2-mu-2
Newton
Invalid
High
MOS Ceilometer
Fuel for OpenStack 10.1
Ocata
Fix Committed
High
Ilya Tyaptin
Fuel for OpenStack 11.0

Bug Description

Detailed bug description:
Found on ci: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.command_line/20/testReport/(root)/cli_selected_nodes_deploy_huge/cli_selected_nodes_deploy_huge/

OSTF 'Ceilometer test to list meters, alarms, resources and events' fails with foolowing exception
InternalServerError: 'NoneType' object has no attribute 'get_meters' (HTTP 500) (Request-ID: req-f3e7d5c0-fef5-4646-b6d3-576348c834d2)

Steps to reproduce:
Run cli_selected_nodes_deploy_huge

Expected results:
All ostf suites are passed.

Actual result:
fuel_health.common.test_mixins: INFO: STEP:1, verify action: 'getting list of meters'
ceilometerclient.openstack.common.apiclient.client: DEBUG: Request returned failure status: 500
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/meters.py", line 38, in list
    return self._list(options.build_url(path, q, params))
  File "/usr/lib/python2.7/site-packages/ceilometerclient/common/base.py", line 63, in _list
    resp = self.api.get(url)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 201, in request
    raise exceptions.from_response(resp, method, url)
InternalServerError: 'NoneType' object has no attribute 'get_meters' (HTTP 500) (Request-ID: req-f3e7d5c0-fef5-4646-b6d3-576348c834d2)

diagnostic snapshot: https://drive.google.com/open?id=0B2ag_Bf-ShtTN1hQODJhY2lmY1E

Dmitry Guryanov (dguryanov)
Changed in fuel:
importance: Undecided → High
status: New → Confirmed
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
milestone: none → 9.1
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.1 → 10.0
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → MOS Ceilometer (mos-ceilometer)
Nadya Privalova (nprivalova)
tags: added: move-to-9.2
Revision history for this message
Ilya Tyaptin (ityaptin) wrote :

Hello,

Issue with no storage connection in ceilometer-api caused by running out of connection retries count.
In this ceilometer-api.log you could see what ceilometer tries to connect MongoDB 120 times:
There are 10 retries to call ceilometer.storage.get_connection (this count defined by database.max_retries default config in ceilometer.conf) and 12 retries by attempt in impl_mongodb.

So, ceilometer-api has tried to connect from 2016-08-08T09:57:26 to 2016-08-08T11:15:25, but MongoDB started at 2016-08-08T11:18:07 only.

This issue could be fixed by changing database.max_retries to -1 by default. In this case ceilometer-services will try to connect forever and it will return 503 while connection is not established.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to puppet-modules/puppet-ceilometer (mos-9.0)

Fix proposed to branch: mos-9.0
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/29578

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on puppet-modules/puppet-ceilometer (mos-9.0)

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mos-9.0
Review: https://review.fuel-infra.org/29578

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/420727

Changed in fuel:
assignee: MOS Ceilometer (mos-ceilometer) → Ilya Tyaptin (ityaptin)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422758

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Ilya Tyaptin (ityaptin) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/mitaka)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/422758
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/420727
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a0bbe2cc46cdef576409c83d67972f921f026c9b
Submitter: Jenkins
Branch: master

commit a0bbe2cc46cdef576409c83d67972f921f026c9b
Author: Ilya Tyaptin <email address hidden>
Date: Mon Jan 16 17:59:22 2017 +0400

    Change default db reconnection tries count to -1

    This change is needed for cases with big time interval
    between Ceilometer services which need a db and
    database backend starting.
    Otherwise, Ceilometer API defines storage connection to None
    and all storage requests fail permanently.

    Change-Id: Ic442ff42e225f98a7a06469b2ff77d95fac84e2b
    Closes-bug: #1611023

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/442433

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/ocata)

Reviewed: https://review.openstack.org/442433
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1c243f7424ca4c84cf789e5aaa8bc8fa72616f82
Submitter: Jenkins
Branch: stable/ocata

commit 1c243f7424ca4c84cf789e5aaa8bc8fa72616f82
Author: Ilya Tyaptin <email address hidden>
Date: Mon Jan 16 17:59:22 2017 +0400

    Change default db reconnection tries count to -1

    This change is needed for cases with big time interval
    between Ceilometer services which need a db and
    database backend starting.
    Otherwise, Ceilometer API defines storage connection to None
    and all storage requests fail permanently.

    Change-Id: Ic442ff42e225f98a7a06469b2ff77d95fac84e2b
    Closes-bug: #1611023
    (cherry picked from commit a0bbe2cc46cdef576409c83d67972f921f026c9b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc2

This issue was fixed in the openstack/fuel-library 11.0.0.0rc2 release candidate.

Michael Kraynov (mkraynov)
tags: added: sla1
Alexander Rubtsov (arubtsov)
tags: added: customer-found
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change restored on puppet-modules/puppet-ceilometer (mos-9.0)

Change restored by Ilya Tyaptin <email address hidden> on branch: mos-9.0
Review: https://review.fuel-infra.org/29578

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on puppet-modules/puppet-ceilometer (mos-9.0)

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mos-9.0
Review: https://review.fuel-infra.org/29578

Revision history for this message
Ilya Tyaptin (ityaptin) wrote :

For MOS 9.x fix has been already commited: https://review.openstack.org/#/c/420727/
So, bug for 9.x should be moved to "Fix commited" status.

Revision history for this message
Ilya Tyaptin (ityaptin) wrote :

Sorry, for misinformation. Mentioned change request has been committed to branch master. Backport to stable/mitaka for MOS 9.x is in progress now: https://review.openstack.org/#/c/422758/.

Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Since we don't support Ceilo in 10.0 issue was moved to Invalid.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/422758
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c83c49928b0a30328b80aa2a3ce4c54870213892
Submitter: Jenkins
Branch: stable/mitaka

commit c83c49928b0a30328b80aa2a3ce4c54870213892
Author: Ilya Tyaptin <email address hidden>
Date: Thu Jan 19 19:08:36 2017 +0400

    Change default db reconnection tries count to -1

    This change is needed for cases with big time interval
    between Ceilometer services and database backend starting.
    Otherwise, Ceilometer API defines storage connection to None
    and all storage requests fail permanently.

    cherry-picked from bd53bfb

    Change-Id: Ic442ff42e225f98a7a06469b2ff77d95fac84e2b
    Closes-bug: #1611023

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on 9.2 MU 2:
https://patching-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.command_line/14/testReport/(root)/cli_selected_nodes_deploy_huge/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.