SM provision failure in provision_control.py

Bug #1732740 reported by wenqing liang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
Critical
kamlesh parmar
R4.1
Fix Committed
Critical
kamlesh parmar
Trunk
Fix Committed
Critical
kamlesh parmar

Bug Description

Seen in CB R4.1-50 newton:

+----------+---------------------+------------+--------------------------------------------------------------------------+
| id | status | ip_address | roles |
+----------+---------------------+------------+--------------------------------------------------------------------------+
| server4 | provision_failed | 10.0.0.7 | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb'] |
| server6 | provision_failed | 10.0.0.9 | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb'] |
| server7 | provision_failed | 10.0.0.10 | [u'contrail-lb'] |
| server1 | provision_completed | 10.0.0.4 | [u'openstack'] |
| server2 | provision_completed | 10.0.0.5 | [u'openstack'] |
| server3 | provision_completed | 10.0.0.6 | [u'openstack'] |
| server10 | provision_failed | 10.0.0.13 | [u'contrail-compute'] |
| server5 | provision_failed | 10.0.0.8 | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb'] |
| server8 | provision_failed | 10.0.0.11 | [u'contrail-compute'] |
| server9 | provision_failed | 10.0.0.12 | [u'contrail-compute'] |
+----------+---------------------+------------+--------------------------------------------------------------------------+

"2017-11-16 09:54:15,514-INFO-sm_ansible_callback.py:53-append(): fatal: [10.0.0.9]Traceback (most recent call last):
  File "/usr/share/contrail-utils/provision_control.py", line 225, in <module>
    main()
  File "/usr/share/contrail-utils/provision_control.py", line 221, in main
    ControlProvisioner(args_str)
  File "/usr/share/contrail-utils/provision_control.py", line 64, in __init__
    self._args.address_families, self._args.md5)
  File "/usr/share/contrail-utils/provision_bgp.py", line 107, in add_bgp_router
    cur_id = vnc_lib.bgp_router_create(bgp_router_obj)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 483, in _object_create
    rest.OP_POST, obj_cls.create_uri, data=json_body)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 872, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 962, in _request
    raise RefsExistError(content)
cfgm_common.exceptions.RefsExistError: ['default-domain', 'default-project', 'ip-fabric', '__default__', 'server4'] already exists with uuid: 75ae2f55-1cb9-48c9-901a-b5302d4cad70"
"2017-11-16 09:54:15,532-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.7] => (item={'key': u'10.10.0.7', 'value': u'server4'})"
"2017-11-16 09:54:15,669-INFO-sm_ansible_callback.py:53-append(): changed: [10.0.0.8] => (item={'key': u'10.10.0.7', 'value': u'server4'})"
"2017-11-16 09:54:16,724-INFO-sm_ansible_callback.py:53-append(): fatal: [10.0.0.7]Traceback (most recent call last):

Revision history for this message
Abhay Joshi (abhayj) wrote :

please provide replication setup.

Revision history for this message
Abhay Joshi (abhayj) wrote :

Kamlesh moved the script outside of container to external ansible. The same task provision_control is also there for internal ansible (Nagendra can confirm if its removed from internal ansible).
The original checkin from Kamlesh passed the ci job. It means that provision_control ran twice in that job (external ansible and internal ansible). Below trace is seems for exception when trying add object again. Is that the behavior change in api-server?

Thanks,

Abhay

Revision history for this message
Sachin Bansal (sbansal) wrote :

From Nagendra:

provision_control.py is not run from within docker, so ran only once from SM.

Regards,
Nagendra Prasath

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37633
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37634
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/37635
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37634
Committed: http://github.com/Juniper/contrail-controller/commit/003e1c602665b109b9cfdbd6b35c6c8015c36561
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 003e1c602665b109b9cfdbd6b35c6c8015c36561
Author: Kamlesh Parmar <email address hidden>
Date: Fri Nov 17 11:46:28 2017 -0800

In multi control-node configuration, adding already existing control node
throws exception for Id already existing. need to handle that gracefully.

Closes-Bug: #1732740

Change-Id: I41ab61f6f90853302f4a51efffd046c54e8db780

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/37633
Committed: http://github.com/Juniper/contrail-controller/commit/ee27603f9618ee4d9f78c6f059f01504eb655796
Submitter: Zuul (<email address hidden>)
Branch: master

commit ee27603f9618ee4d9f78c6f059f01504eb655796
Author: Kamlesh Parmar <email address hidden>
Date: Fri Nov 17 11:46:28 2017 -0800

In multi control-node configuration, adding already existing control node
throws exception for Id already existing. need to handle that gracefully.

Closes-Bug: #1732740

Change-Id: I41ab61f6f90853302f4a51efffd046c54e8db780

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/37635
Committed: http://github.com/Juniper/contrail-controller/commit/83ae9699506192d2c40bc169082832aebe70b3f7
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 83ae9699506192d2c40bc169082832aebe70b3f7
Author: Kamlesh Parmar <email address hidden>
Date: Fri Nov 17 11:46:28 2017 -0800

In multi control-node configuration, adding already existing control node
throws exception for Id already existing. need to handle that gracefully.

Closes-Bug: #1732740

Change-Id: I41ab61f6f90853302f4a51efffd046c54e8db780

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.