[R4.0 build 15]ceilometer api calls were failing

Bug #1694368 reported by aswani kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Megh Bhatt
Trunk
Fix Committed
High
Megh Bhatt

Bug Description

R4.0 build 15 mitaka ubuntu14-04

Not able to make ceilometer api calls

root@nodei31:~# ceilometer meter-list
HTTPConnectionPool(host='10.204.217.143', port=8777): Max retries exceeded with url: /v2/meters (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd822301290>: Failed to establish a new connection: [Errno 111] Connection refused',))

openstack-status

== Ceilometer services ==
ceilometer-api: active
ceilometer-agent-central: active
ceilometer-agent-compute: inactive (disabled on boot)
ceilometer-collector: active
ceilometer-agent-notification:active

seeing these errors in the api log

 ERROR ceilometer Traceback (most recent call last):
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/bin/ceilometer-api", line 10, in <module>
2017-05-30 11:20:10.034 22741 ERROR ceilometer sys.exit(main())
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/api.py", line 23, in main
2017-05-30 11:20:10.034 22741 ERROR ceilometer app.build_server()
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 105, in build_server
2017-05-30 11:20:10.034 22741 ERROR ceilometer app = load_app()
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 101, in load_app
2017-05-30 11:20:10.034 22741 ERROR ceilometer return deploy.loadapp("config:" + cfg_file)
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2017-05-30 11:20:10.034 22741 ERROR ceilometer return loadobj(APP, uri, name=name, **kw)
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2017-05-30 11:20:10.034 22741 ERROR ceilometer return context.create()
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-05-30 11:20:10.034 22741 ERROR ceilometer return self.object_type.invoke(self)
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 203, in invoke
2017-05-30 11:20:10.034 22741 ERROR ceilometer app = context.app_context.create()
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-05-30 11:20:10.034 22741 ERROR ceilometer return self.object_type.invoke(self)
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2017-05-30 11:20:10.034 22741 ERROR ceilometer return fix_call(context.object, context.global_conf, **context.local_conf)
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2017-05-30 11:20:10.034 22741 ERROR ceilometer val = callable(*args, **kw)
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 126, in app_factory
2017-05-30 11:20:10.034 22741 ERROR ceilometer return setup_app()
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 58, in setup_app
2017-05-30 11:20:10.034 22741 ERROR ceilometer hooks.DBHook(),
2017-05-30 11:20:10.034 22741 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/hooks.py", line 53, in __init__
2017-05-30 11:20:10.034 22741 ERROR ceilometer ', '.join(['metering', 'event']))
2017-05-30 11:20:10.034 22741 ERROR ceilometer Exception: Api failed to start. Failed to connect to databases, purpose: metering, event
2017-05-30 11:20:10.034 22741 ERROR ceilometer

again in 10 seconds.
2017-05-30 11:16:49.712 22741 WARNING ceilometer.storage.mongo.utils [-] Unable to reconnect to the primary mongodb: 10.204.217.143:27017: [Errno 111] Connection refused. Trying again in 10 seconds.
2017-05-30 11:17:29.775 22741 WARNING ceilometer.storage.mongo.utils [-] Unable to reconnect to the primary mongodb: 10.204.217.143:27017: [Errno 111] Connection refused. Trying again in 10 seconds.
2017-05-30 11:18:09.841 22741 WARNING ceilometer.storage.mongo.utils [-] Unable to reconnect to the primary mongodb: 10.204.217.143:27017: [Errno 111] Connection refused. Trying again in 10 seconds.
2017-05-30 11:18:49.905 22741 WARNING ceilometer.storage.mongo.utils [-] Unable to reconnect to the primary mongodb: 10.204.217.143:27017: [Errno 111] Connection refused. Trying again in 10 seconds.
2017-05-30 11:19:29.965 22741 WARNING ceilometer.storage.mongo.utils [-] Unable to reconnect to the primary mongodb: 10.204.217.143:27017: [Errno 111] Connection refused. Trying again in 10 seconds.

tags: added: sanity
Changed in juniperopenstack:
importance: Undecided → Critical
assignee: nobody → Megh Bhatt (meghb)
Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
Nikhil Bansal (nikhilb-u) wrote :

mongodb is not running, need to figure out who should bring it up

Revision history for this message
Megh Bhatt (meghb) wrote :

This looks like missing feature in ansible provisioning. We were adding mongodb provisioning as part of database node in fabric and SM. We will have to document that ceilometer provisioning is not supported with ansible.

Revision history for this message
Jeba Paulaiyan (jebap) wrote :

Release Notes:

Ceilometer provisioning is not supported through ansible in R4.0.0.0

tags: added: releasenote
removed: blocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/33852
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/33856
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/33852
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/33907
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/33852
Committed: http://github.com/Juniper/contrail-packaging/commit/0de7e0158b43f60a98a7af1a8d456d9ccdb4f37c
Submitter: Zuul (<email address hidden>)
Branch: master

commit 0de7e0158b43f60a98a7af1a8d456d9ccdb4f37c
Author: Megh Bhatt <email address hidden>
Date: Thu Jul 20 15:14:54 2017 -0700

Changes to install mongodb on openstack node

Package mongodb dependents into correct contrail package so that
mongodb-server and mongodb-client packages can be installed on
the openstack node

On ub16 newton, package mongodb-server also in addition to the
above changes

Change-Id: Ib97a84464c8c0e25143cc553cc81001d0d1494b1
Partial-Bug: #1694368

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/33856
Committed: http://github.com/Juniper/contrail-puppet/commit/8dbc38c20d577446f68846c80fcd0955192276bf
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8dbc38c20d577446f68846c80fcd0955192276bf
Author: Megh Bhatt <email address hidden>
Date: Thu Jul 20 15:49:16 2017 -0700

Changes to install mongodb on openstack node

Move installation of mongodb to openstack node
instead of analyticsdb/database node

Change-Id: I7a7e53bf6551cc3add8712386e6bfaa2cdd59ebd
Closes-Bug: #1694368

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/33907
Committed: http://github.com/Juniper/contrail-packaging/commit/8a5c57d15cdffbc822571fb0c4eec7c2b3b55b67
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 8a5c57d15cdffbc822571fb0c4eec7c2b3b55b67
Author: Megh Bhatt <email address hidden>
Date: Thu Jul 20 15:14:54 2017 -0700

Changes to install mongodb on openstack node

Package mongodb dependents into correct contrail package so that
mongodb-server and mongodb-client packages can be installed on
the openstack node

On ub16 newton, package mongodb-server also in addition to the
above changes

Change-Id: Ib97a84464c8c0e25143cc553cc81001d0d1494b1
Partial-Bug: #1694368

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/33940
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/33940
Committed: http://github.com/Juniper/contrail-puppet/commit/fd3020ffbf04d49b82e50462cd7472596fa5031d
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit fd3020ffbf04d49b82e50462cd7472596fa5031d
Author: Megh Bhatt <email address hidden>
Date: Thu Jul 20 15:49:16 2017 -0700

Changes to install mongodb on openstack node

Move installation of mongodb to openstack node
instead of analyticsdb/database node

Change-Id: I7a7e53bf6551cc3add8712386e6bfaa2cdd59ebd
Closes-Bug: #1694368
(cherry picked from commit 8dbc38c20d577446f68846c80fcd0955192276bf)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.