[R5.0]- CI- VM verification fails in opserver intermittently

Bug #1770340 reported by alok kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Invalid
High
alok kumar
Trunk
Invalid
High
alok kumar

Bug Description

In CI as well as sanity, few cases fails intermittently while verifying VMs in opserver.

For failure case, It was seen that response code from analytics is 200 but the response size was very small, probably causing the failure.

For instance, test case SecurityGroupBasicRegressionTests1.test_sec_group_basic has failed at
http://logs.opencontrail.org/12/42712/2/check/contrail-sanity-centos7-kolla-ocata/6b0f35f/sanity-logs/contrail-test-runs/2018_05_03_07_51_49/reports/junit-noframes.html
and
http://logs.opencontrail.org/12/42912/1/check/contrail-sanity-centos7-kolla-ocata/f7e8b82/sanity-logs/contrail-test-runs/2018_05_10_01_27_43/reports/junit-noframes.html

Analytics logs for both of the failures:

http://logs.opencontrail.org/12/42712/2/check/contrail-sanity-centos7-kolla-ocata/6b0f35f/sanity-logs/contrail/contrail-analytics-api.log

05/03/2018 07:57:01 AM [contrail-analytics-api]: __default__ [SYS_INFO]: AnalyticsApiStats: api_stats = << operation_type = GET remote_ip = 172.17.0.2 object_type = virtual-machine request_url = http://172.16.8.9:8081/analytics/uves/virtual-machine/0dfcea49-a3c1-4aa1-a77e-34d4733eb6ba?flat response_time_in_usec = 99 response_size_objects = 1 response_size_bytes = 2 resp_code = 200 useragent = python-requests/2.11.1 node = c7-systest-c-juniper-contrail-ci-c-0000041899.novalocal >>

and
http://logs.opencontrail.org/12/42912/1/check/contrail-sanity-centos7-kolla-ocata/f7e8b82/sanity-logs/contrail/contrail-analytics-api.log

05/10/2018 01:33:42 AM [contrail-analytics-api]: __default__ [SYS_INFO]: AnalyticsApiStats: api_stats = << operation_type = GET remote_ip = 172.17.0.2 object_type = generator request_url = http://172.16.8.7:8081/analytics/uves/generator/c7-systest-c-juniper-contrail-ci-c-0000043929.novalocal%3ACompute%3Acontrail-vrouter-agent%3A0?flat response_time_in_usec = 139 response_size_objects = 1 response_size_bytes = 2 resp_code = 200 useragent = python-requests/2.11.1 node = c7-systest-c-juniper-contrail-ci-c-0000043929.novalocal >>
05/10/2018 01:33:42 AM [contrail-analytics-api]: AggUVE loading: [{u'type': u'NodeStatus', u'key': u'ObjectDatabaseInfo:c7-systest-c-juniper-contrail-ci-c-0000043929.novalocal'}]

Revision history for this message
Sundaresan Rajangam (srajanga) wrote :

What is the issue here? some VMs are missing in /analytics/uves/virtual-machines ?
If so, please check if collector has received the those virtual-machine UVEs using contrail-logs.
The information provided in the bug is not sufficient to triage the issue.

Revision history for this message
alok kumar (kalok) wrote :
Revision history for this message
Sundaresan Rajangam (srajanga) wrote :

Alok, Can't debug this issue with analytics logs. As mentioned in #1, please check the contrail-logs to check if vrouter-agent has sent the VMI UVE for default-domain:ctest-SecurityGroupBasicRegressionTests1-58558567:701d74d0-f0e2-4e44-b61b-10a727691e20

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/43717
Submitter: alok kumar (<email address hidden>)

Jeba Paulaiyan (jebap)
tags: added: contrail-networking
Jeba Paulaiyan (jebap)
tags: added: automation
Revision history for this message
alok kumar (kalok) wrote :

not reproducible easily and not seen the issue in recent ci, so closing for now.

Revision history for this message
Sudheendra Rao (sudheendra-k) wrote :

it shouldn't be incomplete if the problem is not reproducible, moving it to invalid.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.