[Mainline/Mitaka] Objects inconsistent between reads while trying to validate RI

In the latest CB mainline build 44, Objects inconsistent between reads while trying to validate RI.

   During a intf mirroring test, the script validates that a RI in contrail-api has a reference to two other RIs. The query is done via the LB IP

    Across multiple runs, sometimes, it is seen that routing_instance_read() returns an object which does not have routing_instance_refs populated.

   When we debugged further, it is seen that the same query, when run 3 times, does not return the same output. For instance below, every 3rd RI read returns wrongly. Probably, one of the api-servers is returning wrongly.

  Observation has been that if we check the object using contrail-ui on all the 3 config nodes, the RI refs would have got populated and the query in the script too starts passing.

  Checked Cassandra contents on all the 3 nodes and could see RI refs for this object on all 3 cassandras. But the reads in the test script continues to fail.

(Pdb) print vars(self.connections.vnc_lib_fixture.routing_instance_read(fq_name=ri_fq_name)) <<<< no routing_instance_refs
{'_type': 'routing-instance', '_static_route_entries': route = [], 'fq_name': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn1-45608846', u'ctest-vn1-45608846'], 'name': u'ctest-vn1-45608846', '_pending_field_updates': set([]), '_routing_instance_has_pnf': False, '_uuid': u'00acbca6-2084-4931-b0bb-5b3041e5608f', 'parent_type': u'virtual-network', '_pending_ref_updates': set([]), '_pending_field_map_updates': {}, '_id_perms': permissions = owner = contrail-api, owner_access = 7, group = admin, group_access = 7, other_access = 7, uuid = uuid_mslong = 48621117690366257, uuid_lslong = 12734872634142318735, enable = True, created = 2017-09-11T19:32:36.163293, last_modified = 2017-09-11T19:32:36.235709, description = None, user_visible = True, creator = None, '_perms2': owner = 1b8813c441f24c94aa06086a265a9837, owner_access = 7, global_access = 0, share = [], 'route_target_refs': [{u'to': [u'target:64510:8000009'], u'href': u'http://10.204.216.160:8082/route-target/ed8c55b7-5b5b-4955-9c34-7e3696610e15', u'attr': import_export = None, u'uuid': u'ed8c55b7-5b5b-4955-9c34-7e3696610e15'}], '_pending_field_list_updates': {}, '_display_name': u'ctest-vn1-45608846', '_server_conn': <vnc_api.vnc_api.VncApi object at 0x7fec842dedd0>, '_routing_instance_is_default': True, 'parent_uuid': u'0e315c3e-ed30-465a-afb2-058c51dbe1c6'}
(Pdb) print vars(self.connections.vnc_lib_fixture.routing_instance_read(fq_name=ri_fq_name))
{'_type': 'routing-instance', '_static_route_entries': route = [], 'fq_name': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn1-45608846', u'ctest-vn1-45608846'], 'name': u'ctest-vn1-45608846', '_pending_field_updates': set([]), '_routing_instance_has_pnf': False, '_uuid': u'00acbca6-2084-4931-b0bb-5b3041e5608f', 'parent_type': u'virtual-network', '_pending_ref_updates': set([]), '_pending_field_map_updates': {}, '_id_perms': permissions = owner = contrail-api, owner_access = 7, group = admin, group_access = 7, other_access = 7, uuid = uuid_mslong = 48621117690366257, uuid_lslong = 12734872634142318735, enable = True, created = 2017-09-11T19:32:36.163293, last_modified = 2017-09-11T19:32:36.235709, description = None, user_visible = True, creator = None, '_perms2': owner = 1b8813c441f24c94aa06086a265a9837, owner_access = 7, global_access = 0, share = [], 'route_target_refs': [{u'to': [u'target:64510:8000009'], u'href': u'http://10.204.216.160:8082/route-target/ed8c55b7-5b5b-4955-9c34-7e3696610e15', u'attr': import_export = None, u'uuid': u'ed8c55b7-5b5b-4955-9c34-7e3696610e15'}], '_pending_field_list_updates': {}, 'routing_instance_refs': [{u'to': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn3-78087498', u'ctest-vn3-78087498'], u'href': u'http://10.204.216.160:8082/routing-instance/8b713aa6-c79d-4e5a-858a-1b5dac77c221', u'attr': destination_instance = None, u'uuid': u'8b713aa6-c79d-4e5a-858a-1b5dac77c221'}, {u'to': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn2-61893951', u'ctest-vn2-61893951'], u'href': u'http://10.204.216.160:8082/routing-instance/e1d236b0-0589-41e2-88bc-2823c61acf53', u'attr': destination_instance = None, u'uuid': u'e1d236b0-0589-41e2-88bc-2823c61acf53'}], '_display_name': u'ctest-vn1-45608846', '_server_conn': <vnc_api.vnc_api.VncApi object at 0x7fec842dedd0>, '_routing_instance_is_default': True, 'parent_uuid': u'0e315c3e-ed30-465a-afb2-058c51dbe1c6'}
(Pdb) print vars(self.connections.vnc_lib_fixture.routing_instance_read(fq_name=ri_fq_name))
{'_type': 'routing-instance', '_static_route_entries': route = [], 'fq_name': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn1-45608846', u'ctest-vn1-45608846'], 'name': u'ctest-vn1-45608846', '_pending_field_updates': set([]), '_routing_instance_has_pnf': False, '_uuid': u'00acbca6-2084-4931-b0bb-5b3041e5608f', 'parent_type': u'virtual-network', '_pending_ref_updates': set([]), '_pending_field_map_updates': {}, '_id_perms': permissions = owner = contrail-api, owner_access = 7, group = admin, group_access = 7, other_access = 7, uuid = uuid_mslong = 48621117690366257, uuid_lslong = 12734872634142318735, enable = True, created = 2017-09-11T19:32:36.163293, last_modified = 2017-09-11T19:32:36.235709, description = None, user_visible = True, creator = None, '_perms2': owner = 1b8813c441f24c94aa06086a265a9837, owner_access = 7, global_access = 0, share = [], 'route_target_refs': [{u'to': [u'target:64510:8000009'], u'href': u'http://10.204.216.160:8082/route-target/ed8c55b7-5b5b-4955-9c34-7e3696610e15', u'attr': import_export = None, u'uuid': u'ed8c55b7-5b5b-4955-9c34-7e3696610e15'}], '_pending_field_list_updates': {}, 'routing_instance_refs': [{u'to': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn2-61893951', u'ctest-vn2-61893951'], u'href': u'http://10.204.216.160:8082/routing-instance/e1d236b0-0589-41e2-88bc-2823c61acf53', u'attr': destination_instance = None, u'uuid': u'e1d236b0-0589-41e2-88bc-2823c61acf53'}, {u'to': [u'default-domain', u'ctest-TestIntfMirror-65639139', u'ctest-vn3-78087498', u'ctest-vn3-78087498'], u'href': u'http://10.204.216.160:8082/routing-instance/8b713aa6-c79d-4e5a-858a-1b5dac77c221', u'attr': destination_instance = None, u'uuid': u'8b713aa6-c79d-4e5a-858a-1b5dac77c221'}], '_display_name': u'ctest-vn1-45608846', '_server_conn': <vnc_api.vnc_api.VncApi object at 0x7fec842dedd0>, '_routing_instance_is_default': True, 'parent_uuid': u'0e315c3e-ed30-465a-afb2-058c51dbe1c6'}
(Pdb) print ri_fq_name
['default-domain', 'ctest-TestIntfMirror-65639139', 'ctest-vn1-45608846', 'ctest-vn1-45608846']
(Pdb)

Testcase name:
==============
TestIntfMirror.test_intf_mirror_with_subintf_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3

Reference to running a testcase - https://github.com/Juniper/contrail-test-ci/wiki/How-to-use-contrail-test-ci-container

steps to repro:
===============
The issue is seen while trying to validate RI reference when running the above testcase.
So i added a pdb in validate_vn function in /contrail-test/common/servicechain/verify.py and started the script.
It checks for ri_obj for first time and if its present, scripts continues. If ri_obj is not there, then it goes into a loop and keeps checking for 15 times with delay of 5 secs.
Since i have put pdb, it will check for ri_obj once and if it is not present, script will pause and we can debug the issue.