Update XmppConnection data structure correctly after GR swap

Bug #1731834 reported by Ananth Suryanarayana
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Fix Committed
Critical
Ananth Suryanarayana
R4.0
Fix Committed
Critical
Ananth Suryanarayana
R4.1
Fix Committed
Critical
Ananth Suryanarayana
Trunk
Fix Committed
Critical
Ananth Suryanarayana

Bug Description

This issue was found by Venu Kolli during system testing for XMPP GR/LLGR.

During XMPP GR, Old and Newly formed XmppConnection objects are swapped in order to retain some critical data structures as is. There are two issues here.

1. During the swap, all related members to the new session must also be swapped, such as endpoints, proto_stats, etc. endpoint is also the key inside XmppServer::connection_map_. Hence this map also needs to be updated, when the endpoints are swapped

2. In the state machine, after connections are swapped, update connection must be used for further processing, such as to trigger keep-alive timer

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37452
Submitter: Ananth Suryanarayana (<email address hidden>)

Nischal Sheth (nsheth)
tags: added: blocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37452
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37452
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37452
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37452
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37452
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37449
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37449
Committed: http://github.com/Juniper/contrail-controller/commit/51f07ac6b602fd8276c636cdad4d5e31d508b4f9
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 51f07ac6b602fd8276c636cdad4d5e31d508b4f9
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Nov 12 22:23:30 2017 -0800

Swap XmppConnections correctly during XMPP Graceful Restart

During XMPP GR, Old and Newly formed XmppConnection objects are swapped in
order to retain some critical data structures as is. There are two issues here.

1. During the swap, all related members to the new session must also be swapped,
such as endpoints, proto_stats, disable_read_, etc. endpoint is also the key
inside XmppServer::connection_map_. Hence this map also needs to be updated,
when the endpoints are swapped. During this update, the map is protected using
a lock because multiple XmppState machine data structures can run and hence
update the map concurrently.

2. In the state machine, after connections are swapped, updated connection must
be used for further processing, such as to trigger keep-alive timer

o Add specific checks in GR tests to make sure that KeepAlive and HoldTImers are
always running in established session
o Run all GR tests with xmpp authentication enabled as well.
o Split GR test cases into separate tests so that they can be run in parallel
and also because each of them now gets a longer time to complete

3. Partly stablize GR tests by removing some of the outdated code in mock agent.
We should task_util::TaskFire() routines to safely access data structures
instead of using a separate work queue, conditional variable etc.

Change-Id: I5aab8b175c33870d8991058d79e4fe7dd5b72170
Closes-Bug: 1731834
Closes-Bug: 1732267

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/37452
Committed: http://github.com/Juniper/contrail-controller/commit/b89e149c80e165187524cf7772ffed356aa74022
Submitter: Zuul (<email address hidden>)
Branch: master

commit b89e149c80e165187524cf7772ffed356aa74022
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Nov 12 22:23:30 2017 -0800

Swap XmppConnections correctly during XMPP Graceful Restart

During XMPP GR, Old and Newly formed XmppConnection objects are swapped in
order to retain some critical data structures as is. There are two issues here.

1. During the swap, all related members to the new session must also be swapped,
such as endpoints, proto_stats, disable_read_, etc. endpoint is also the key
inside XmppServer::connection_map_. Hence this map also needs to be updated,
when the endpoints are swapped. During this update, the map is protected using
a lock because multiple XmppState machine data structures can run and hence
update the map concurrently.

2. In the state machine, after connections are swapped, updated connection must
be used for further processing, such as to trigger keep-alive timer

o Add specific checks in GR tests to make sure that KeepAlive and HoldTImers are
always running in established session
o Run all GR tests with xmpp authentication enabled as well.
o Split GR test cases into separate tests so that they can be run in parallel
and also because each of them now gets a longer time to complete

3. Partly stablize GR tests by removing some of the outdated code in mock agent.
We should task_util::TaskFire() routines to safely access data structures
instead of using a separate work queue, conditional variable etc.

Change-Id: I5aab8b175c33870d8991058d79e4fe7dd5b72170
Partial-Bug: 1731834 1732267

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/37866
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37866
Committed: http://github.com/Juniper/contrail-controller/commit/0edf7958b8e33be776e2629ebcd348c0a81ec52d
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 0edf7958b8e33be776e2629ebcd348c0a81ec52d
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Nov 12 22:23:30 2017 -0800

Swap XmppConnections correctly during XMPP Graceful Restart

During XMPP GR, Old and Newly formed XmppConnection objects are swapped in
order to retain some critical data structures as is. There are two issues here.

1. During the swap, all related members to the new session must also be swapped,
such as endpoints, proto_stats, disable_read_, etc. endpoint is also the key
inside XmppServer::connection_map_. Hence this map also needs to be updated,
when the endpoints are swapped. During this update, the map is protected using
a lock because multiple XmppState machine data structures can run and hence
update the map concurrently.

2. In the state machine, after connections are swapped, updated connection must
be used for further processing, such as to trigger keep-alive timer

o Add specific checks in GR tests to make sure that KeepAlive and HoldTImers are
always running in established session
o Run all GR tests with xmpp authentication enabled as well.
o Split GR test cases into separate tests so that they can be run in parallel
and also because each of them now gets a longer time to complete

3. Partly stablize GR tests by removing some of the outdated code in mock agent.
We should task_util::TaskFire() routines to safely access data structures
instead of using a separate work queue, conditional variable etc.

Change-Id: I5aab8b175c33870d8991058d79e4fe7dd5b72170
Closes-Bug: 1731834
Closes-Bug: 1732267
(cherry picked from commit 51f07ac6b602fd8276c636cdad4d5e31d508b4f9)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/37934
Submitter: Ananth Suryanarayana (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37934
Committed: http://github.com/Juniper/contrail-controller/commit/092ea4f1d19401ab42947813d175f1844d405ae3
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 092ea4f1d19401ab42947813d175f1844d405ae3
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Nov 12 22:23:30 2017 -0800

Swap XmppConnections correctly during XMPP Graceful Restart

During XMPP GR, Old and Newly formed XmppConnection objects are swapped in
order to retain some critical data structures as is. There are two issues here.

1. During the swap, all related members to the new session must also be swapped,
such as endpoints, proto_stats, disable_read_, etc. endpoint is also the key
inside XmppServer::connection_map_. Hence this map also needs to be updated,
when the endpoints are swapped. During this update, the map is protected using
a lock because multiple XmppState machine data structures can run and hence
update the map concurrently.

2. In the state machine, after connections are swapped, updated connection must
be used for further processing, such as to trigger keep-alive timer

o Add specific checks in GR tests to make sure that KeepAlive and HoldTImers are
always running in established session
o Run all GR tests with xmpp authentication enabled as well.
o Split GR test cases into separate tests so that they can be run in parallel
and also because each of them now gets a longer time to complete

3. Partly stablize GR tests by removing some of the outdated code in mock agent.
We should task_util::TaskFire() routines to safely access data structures
instead of using a separate work queue, conditional variable etc.

Change-Id: I5aab8b175c33870d8991058d79e4fe7dd5b72170
Closes-Bug: 1731834
Closes-Bug: 1732267

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.