Customer is testing analytics response time when one of the control node fails. Response time varies based on the number of VN and VMI’s. Greater the number VN’s & VMI it takes longer for the response.
Customer setup is as below
3 Control, config
3 collector
3 DB
1 openstack
6 compute nodes
2 TSN nodes
/etc/contrail/contrail-vrouter-agent.conf is modified to point to collector nodes on each compute node.
Customer has provided scripts to create VN & VMI and to query the analytics. They shutdown one of the control node and note down the time. They see a large difference in correct response for the analytics queries based on the number of VN’s and VMI
VN VMI Response time
303 600 5 Sec
1500 3000 50 secs
3000 6000 approx 2 min With one control node shutdown
The above delta time doubles when two control nodes are shutdown.
Is this intended behavior?
Why is this difference noticed in clustered scenario when collector nodes stop responding (to replicate the nodes are shutdown). The DB nodes are all up and running when performing this test.
Can the response time be reduced & consistent irrespective number of interfaces.
I could replicate the issue in lab up to 1500 VN and 3000 VMI. Due to resource constraints couldn’t scale this higher.
When querying fro VMI we were getting Http 200 K as response but nothing pertaining to interface or network (output of script)
Valid response
10.204.74.242:8081 default-domain:mock:vmi_ntt-comp5_0100_01 200 {"UveVMInterfaceAgent": {"ip6_active": f
Invalid response
10.204.74.242:8081 default-domain:mock:vmi_ntt-comp6_0100_01 200 {}
From LogsFrom contrail-alarm-gen.log sv-25_log-large_sv-24_down
09/05/2017 11:46:02 AM [contrail-alarm-gen]: -uve-3 An exception of type LeaderNotAvailableError occured. Arguments:
LeaderNotAvailableError: LeaderNotAvailableError - 5 - This error is thrown if we are in the middle of a leadership election and there is currently no leader for this partition and hence it is unavailable for writes.
09/05/2017 11:46:07 AM [contrail-alarm-gen]: -uve-23 An exception of type LeaderNotAvailableError occured. Arguments:
LeaderNotAvailableError: LeaderNotAvailableError - 5 - This error is thrown if we are in the middle of a leadership election and there is currently no leader for this partition and hence it is unavailable for writes.
09/05/2017 11:46:07 AM [contrail-alarm-gen]: Error: Consumer Failure LeaderNotAvailableError occured. Arguments:
LeaderNotAvailableError: LeaderNotAvailableError - 5 - This error is thrown if we are in the middle of a leadership election and there is currently no leader for this partition and hence it is unavailable for writ
09/05/2017 11:51:00 AM [contrail-alarm-gen]: redis-uve failed Error connecting to 192.168.0.124:6379. timed out. for key ObjectVRouter:sv-39: (u'192.168.0.124', 6379, 1149) tb Traceback (most recent call last):
ConnectionError: Error connecting to 192.168.0.124:6379. timed out.
09/05/2017 11:51:00 AM [contrail-alarm-gen]: redis-uve failed Error connecting to 192.168.0.124:6379. timed out. for key ObjectGeneratorInfo:sv-21:Control:contrail-dns:0: (u'192.168.0.124', 6379, 1149) tb Traceback (most recent call last):
ConnectionError: Error connecting to 192.168.0.124:6379. timed out.
09/05/2017 11:51:00 AM [contrail-alarm-gen]: Exception KeyError in notif worker. Arguments:
((u'192.168.0.124', 6379, 1149),) : traceback Traceback (most recent call last):
09/05/2017 11:51:04 AM [contrail-alarm-gen]: -uve-12 An exception of type KeyError occured. Arguments:
09/05/2017 11:51:40 AM [contrail-alarm-gen]: Starting part 2 collectors [u'192.168.0.126:6379', u'192.168.0.125:6379']
Hi,
Any Update?
Best Regards,
Vijay Kumar