[ocata][17.02] ceilometer errors due to incorrect swift API url usage with rados gateway: storage.containers.objects: Account GET failed

Bug #1712813 reported by Dmitrii Shcherbakov
This bug report is a duplicate of:  Bug #1481364: feature: ceilometer integration. Edit Remove
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
New
Undecided
Unassigned
OpenStack Ceilometer Charm
New
Undecided
Unassigned

Bug Description

ceilometer-agent-central log is full of errors:

ceilometer-agent-central.log.1:2017-08-14 12:36:55.689 76729 ERROR ceilometer.agent.manager [-] Continue after error from storage.containers.objects: Account GET failed: http://10.232.40.209:80/swift/v1/AUTH_099a6960844445f48b7ed795c8ffeac2?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_099a6960844445f48b
ceilometer-agent-central.log.1:2017-08-14 12:36:55.689 76729 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account GET failed', body)
ceilometer-agent-central.log.1:2017-08-14 12:36:55.689 76729 ERROR ceilometer.agent.manager ClientException: Account GET failed: http://10.232.40.209:80/swift/v1/AUTH_099a6960844445f48b7ed795c8ffeac2?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_099a6960844445f48b
ceilometer-agent-central.log.1:2017-08-14 12:36:56.949 76729 ERROR ceilometer.agent.manager [-] Continue after error from storage.containers.objects.size: Account GET failed: http://10.232.40.209:80/swift/v1/AUTH_099a6960844445f48b7ed795c8ffeac2?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_099a6960844445f48b
ceilometer-agent-central.log.1:2017-08-14 12:36:56.949 76729 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account GET failed', body)
ceilometer-agent-central.log.1:2017-08-14 12:36:56.949 76729 ERROR ceilometer.agent.manager ClientException: Account GET failed: http://10.232.40.209:80/swift/v1/AUTH_099a6960844445f48b7ed795c8ffeac2?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_099a6960844445f48b

Errors indicate that ceilometer is polling the swift API for an object size. And it does not use code available specifically for radosgw:

https://github.com/openstack/ceilometer/blob/stable/ocata/ceilometer/objectstore/swift.py#L127 -L142 - this code is used (swift-specific)

https://github.com/openstack/ceilometer/blob/stable/ocata/ceilometer/objectstore/rgw.py#L121-L137
 instead of this (rados-gateway specific)

The messages would have reported radosgw.containers.objects.size not storage.containers.objects.size

Related info:

https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage
https://review.openstack.org/#/c/151763/10/doc/source/measurements.rst

This is missing in our current implementation:

"In order to access Ceph from Telemetry, you need to specify a service group for radosgw in the ceilometer.conf configuration file along with access_key and secret_key of the admin user mentioned above."

Similar issue:
http://tracker.ceph.com/issues/19268

Tags: cpec
affects: charm-ceilometer → charm-ceph-radosgw
Revision history for this message
Luca Cervigni (cervigni) wrote :

Have anyone have any news about this ?
Is there a workaround? I can either have object store working or ceilometer working.

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Luca,

that's the latest:
https://bugs.launchpad.net/charms/+source/ceph-radosgw/+bug/1481364/comments/1

I wouldn't say "either or" - it's rather a configuration problem for object storage metering specifically.

While this is not fixed in the charm yet you could try to use manual configuration: http://tracker.ceph.com/issues/19268#note-12

Revision history for this message
Luca Cervigni (cervigni) wrote :

Dmitrii,

In ocata, we use ceph as object storage backend, and our rados conf is this:
[client.radosgw.gateway]
rgw_keystone_token_cache_size = 10
user = www-data
rgw_keystone_url = http://10.4.78.244:35357
rgw_s3_auth_use_keystone = True
rgw_frontends = civetweb port=7480
log_file = /var/log/ceph/radosgw.log
rgw_init_timeout = 360000
rgw_keystone_admin_token = xxxxxxxxxxx
host = xxxxxx
rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
keyring = /etc/ceph/client.radosgw.gateway

We actually had to remove, the ceph workaround of adding the tenant_id in the endpoints and changing the rados config to make rados work with ceph. Before removing that "custom" endpoints, the obj storage was not working. So for us is either one or the other. Or have you got another solution?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.