Hyper-V CPU pollster,could not get CPU time for instance: list index out of range

Bug #1249959 reported by Jun Xie
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Undecided
Claudiu Belu
Havana
Fix Released
Undecided
Claudiu Belu

Bug Description

environment: Windows Server 2012 Datacenter,with hyper-V

debugged, and found that in ceilometer\compute\virt\hyperv\utilsv2.py :
 element.associators(wmi_association_class=self._METRICS_ME) in _get_metrics() returns [ ].

http://paste.openstack.org/show/51563/

2013-11-11 05:45:10.569 1432 INFO ceilometer.compute.manager [-] Polling pollster cpu
2013-11-11 05:45:10.569 1432 INFO ceilometer.compute.pollsters.cpu [-] checking instance 144af1c4-1663-4070-88c8-db65ae13d96a
2013-11-11 05:45:10.944 1432 ERROR ceilometer.compute.pollsters.cpu [-] could not get CPU time for 144af1c4-1663-4070-88c8-db65ae13d96a: list index out of range
2013-11-11 05:45:10.944 1432 ERROR ceilometer.compute.pollsters.cpu [-] list index out of range
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu Traceback (most recent call last):
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu File "C:\Program Files (x86)\IBM\SmartCloud Entry\Hyper-V Agent\Python27\lib\site-packages\ceilometer\compute\pollsters\cpu.py", line 36, in get_samples
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu cpu_info = manager.inspector.inspect_cpus(instance_name)
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu File "C:\Program Files (x86)\IBM\SmartCloud Entry\Hyper-V Agent\Python27\lib\site-packages\ceilometer\compute\virt\hyperv\inspector.py", line 45, in inspect_cpus
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu cpu_count, uptime) = self._utils.get_cpu_metrics(instance_name)
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu File "C:\Program Files (x86)\IBM\SmartCloud Entry\Hyper-V Agent\Python27\lib\site-packages\ceilometer\compute\virt\hyperv\utilsv2.py", line 91, in get_cpu_metrics
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu cpu_metric_aggr = self._get_metrics(vm, cpu_metrics_def)[0]
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu IndexError: list index out of range
2013-11-11 05:45:10.944 1432 TRACE ceilometer.compute.pollsters.cpu

Revision history for this message
Jun Xie (junxiebj) wrote :

looks like setting: enable_instance_metrics_collection to true makes the error msg away, looks like this is not a problem , although this error msg is not seens again, but the CPU time used always stays 0(Expected: should be Cumulative). so this issue can be linked to the following one :
https://bugs.launchpad.net/ceilometer/+bug/1250336

Revision history for this message
Jun Xie (junxiebj) wrote :

ceilometer sample-list -m cpu|grep a7d2b685-cf54-4726-b535-c42870cfc7be

| a7d2b685-cf54-4726-b535-c42870cfc7be | cpu | cumulative | 0.0 | ns | 2013-11-12T16:01:52 |
| a7d2b685-cf54-4726-b535-c42870cfc7be | cpu | cumulative | 0.0 | ns | 2013-11-12T16:11:51 |
| a7d2b685-cf54-4726-b535-c42870cfc7be | cpu | cumulative | 0.0 | ns | 2013-11-12T16:21:51 |

Revision history for this message
Jun Xie (junxiebj) wrote :
Download full text (6.5 KiB)

the value of cpu is always 0,: causing memory error

http://paste.openstack.org/show/51941/

2013-11-12 08:21:51.101 2548 INFO ceilometer.compute.manager [-] Polling pollster cpu
2013-11-12 08:21:51.101 2548 INFO ceilometer.compute.pollsters.cpu [-] checking instance a7d2b685-cf54-4726-b535-c42870cfc7be
2013-11-12 08:21:51.492 2548 INFO ceilometer.compute.pollsters.cpu [-] CPUTIME USAGE: {'OS-EXT-STS:task_state': None, 'addresses': {u'network1': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:2e:4c:26', u'version': 4, u'addr': u'10.0.1.3', u'OS-EXT-IPS:type': u'fixed'}]}, 'links': [{u'href': u'http://10.1.1.151:8774/v2/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'self'}, {u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'bookmark'}], 'image': {u'id': u'fc79cd16-45f3-4f9c-9857-442385b7f955', u'links': [{u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/images/fc79cd16-45f3-4f9c-9857-442385b7f955', u'rel': u'bookmark'}], 'name': u'hyperv-image'}, 'ramdisk_id': None, 'manager': <novaclient.v1_1.servers.ServerManager object at 0x030DB1B0>, 'OS-EXT-STS:vm_state': u'active', 'OS-EXT-SRV-ATTR:instance_name': u'instance-00000004', 'OS-SRV-USG:launched_at': u'2013-11-12T15:56:26.000000', 'flavor': {'name': u'xj-flavor', u'links': [{u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/flavors/7', u'rel': u'bookmark'}], 'ram': 1024, 'ephemeral': 0, 'vcpus': 1, 'disk': 10, u'id': u'7'}, 'id': u'a7d2b685-cf54-4726-b535-c42870cfc7be', 'security_groups': [{u'name': u'default'}], 'user_id': u'49774643974642b5898facdea26c92c9', 'OS-DCF:diskConfig': u'MANUAL', 'accessIPv4': u'', 'accessIPv6': u'', 'progress': 0, 'OS-EXT-STS:power_state': 1, 'OS-EXT-AZ:availability_zone': u'nova', 'config_drive': u'', 'status': u'ACTIVE', 'updated': u'2013-11-12T08:56:03Z', 'hostId': u'bb9341a44b83789f578a5d5d87c260f5f7a593d50f0413c09a12a50c', 'OS-EXT-SRV-ATTR:host': u'WIN-RGVGHQAJUUB', 'OS-SRV-USG:terminated_at': None, 'kernel_id': None, 'key_name': None, 'OS-EXT-SRV-ATTR:hypervisor_hostname': u'WIN-RGVGHQAJUUB', 'name': u'hyperv8888', 'created': u'2013-11-12T08:55:53Z', 'tenant_id': u'0013b7b825174c869afa6a513493044b', 'os-extended-volumes:volumes_attached': [], '_info': {u'OS-EXT-STS:task_state': None, u'addresses': {u'network1': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:2e:4c:26', u'version': 4, u'addr': u'10.0.1.3', u'OS-EXT-IPS:type': u'fixed'}]}, u'links': [{u'href': u'http://10.1.1.151:8774/v2/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'self'}, {u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/servers/a7d2b685-cf54-4726-b535-c42870cfc7be', u'rel': u'bookmark'}], u'image': {u'id': u'fc79cd16-45f3-4f9c-9857-442385b7f955', u'links': [{u'href': u'http://10.1.1.151:8774/0013b7b825174c869afa6a513493044b/images/fc79cd16-45f3-4f9c-9857-442385b7f955', u'rel': u'bookmark'}], 'name': u'hyperv-image'}, u'OS-EXT-STS:vm_state': u'active', u'OS-EXT-SRV-ATTR:instance_name': u'instance-00000004', u'OS-SRV-USG:launched_at': u'2013-11-12T15:56:26.000000', u'flavor': {'name': u'xj-...

Read more...

Julien Danjou (jdanjou)
Changed in ceilometer:
status: New → Triaged
Revision history for this message
Alessandro Pilotti (alexpilotti) wrote :

Correct, enable_instance_metrics_collection must be enabled in Nova. It must anyway not raise an exception if not set.

We'll check right away into the "0" cumulative value that you are experiencing.

Revision history for this message
Jun Xie (junxiebj) wrote :

 thanks,Alessandro. BTW, is there any doc available for configuring ceilometer + hyper-v? I am now wondering whether I have missed some other configurations(in order to let ceilometer correctly collect the meters of the hyper-v node), if any.

I now know about this one: enable_instance_metrics_collection = true in nova.conf , anything else which are specific for hyper environment?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/57017

Changed in ceilometer:
assignee: nobody → Claudiu Belu (cbelu)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/57017
Committed: http://github.com/openstack/ceilometer/commit/f15e2cc9248c99e9b5c6b371ecce4f0ed49450cc
Submitter: Jenkins
Branch: master

commit f15e2cc9248c99e9b5c6b371ecce4f0ed49450cc
Author: Claudiu Belu <email address hidden>
Date: Mon Nov 18 05:26:17 2013 -0800

    Fixes Hyper-V Inspector cpu metrics bug

    Closes-Bug: 1249959

    Change-Id: I39513648ab25d46b2639b24a94e7344954ce3aeb

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: none → icehouse-1
status: Fix Committed → Fix Released
tags: added: hyper-v
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/83024

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/havana)

Reviewed: https://review.openstack.org/83024
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=6637acdd8d9da4a2cc9e977a0837cc25483372d6
Submitter: Jenkins
Branch: stable/havana

commit 6637acdd8d9da4a2cc9e977a0837cc25483372d6
Author: Claudiu Belu <email address hidden>
Date: Mon Nov 18 05:26:17 2013 -0800

    Checks for presence of Hyper-V cpu metrics before accessing

    Adds a default value if there are no cpu metrics to be retrieved.

    Closes-Bug: 1249959
    (cherry picked from commit f15e2cc9248c99e9b5c6b371ecce4f0ed49450cc)

    Change-Id: I39513648ab25d46b2639b24a94e7344954ce3aeb

tags: added: in-stable-havana
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: icehouse-1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.