Bug #1543901 “void boost::intrusive::list_impl<boost::intrusive:...” : Bugs : Juniper Openstack

OpenContrail Admin (ci-admin-f) on 2016-02-10

Changed in juniperopenstack:
assignee:	nobody → Hari Prasad Killi (haripk)
milestone:	none → r3.0-fcs

Hari Prasad Killi (haripk) on 2016-02-10

tags:

added: vrouter

Revision history for this message

Prabhjot Singh Sethi (prabhjot) wrote on 2016-02-15:

#1

Download full text (11.2 KiB)

issue happened due to agent route modification from two threads at the same time one in context of "Agent::RouteWalker" and other in context of "Agent::KSync", both of them trying to manage there paths in the same route for dynamic router peer.

before removal of path there was a previous front pointing to BGP path which got removed before calling sort on the pending path list.

This parallel access currently happens only when OVS and BGP connection is down at the same time.

#4 0x000000000084e865 in AgentRoute::RemovePath (this=0x7f4644d07580, path=0x7f46245d2850) at controller/src/vnsw/agent/oper/agent_route.cc:559
559 controller/src/vnsw/agent/oper/agent_route.cc: No such file or directory.
(gdb) p *path
$1 = (AgentPath) {
  <Path> = {
    _vptr.Path = 0xe36390 <vtable for AgentPath+16>,
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          prev_ = 0x0
        }, <No data fields>}, <No data fields>},
    time_stamp_usecs_ = 1455067642637189
  },
  members of AgentPath:
  peer_ = 0x7f4644002e90,
  nh_ = {
    px = 0x7f463001fa90
  },
  label_ = 4294967295,
  vxlan_id_ = 6,
  dest_vn_name_ = {
    static npos = <optimized out>,
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
      _M_p = 0x7f4694108c78 "default-domain:commonmax:commonmax-vn31-001"
    }
  },
  sync_ = false,
  force_policy_ = false,
  sg_list_ = {
    <std::_Vector_base<int, std::allocator<int> >> = {
      _M_impl = {
        <std::allocator<int>> = {
          <__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>},
        members of std::_Vector_base<int, std::allocator<int> >::_Vector_impl:
        _M_start = 0x0,
        _M_finish = 0x0,
        _M_end_of_storage = 0x0
      }
    }, <No data fields>},
  tunnel_dest_ = {
    addr_ = {
      s_addr = 688592812
    }
  },
  tunnel_bmap_ = 8,
  tunnel_type_ = TunnelType::VXLAN,
  vrf_name_ = {
    static npos = <optimized out>,
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
      _M_p = 0x132a898 <std::string::_Rep::_S_empty_rep_storage@@GLIBCXX_3.4+24> ""
    }
  },
  gw_ip_ = {
    addr_ = {
      s_addr = 0
    }
  },
  unresolved_ = false,
  is_stale_ = false,
  is_subnet_discard_ = false,
  dependant_rt_ = {
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          pr...

issue happened due to agent route modification from two threads at the same time one in context of "Agent::RouteWalker" and other in context of "Agent::KSync", both of them trying to manage there paths in the same route for dynamic router peer.

before removal of path there was a previous front pointing to BGP path which got removed before calling sort on the pending path list.

This parallel access currently happens only when OVS and BGP connection is down at the same time.

#4  0x000000000084e865 in AgentRoute::RemovePath (this=0x7f4644d07580, path=0x7f46245d2850) at controller/src/vnsw/agent/oper/agent_route.cc:559
559     controller/src/vnsw/agent/oper/agent_route.cc: No such file or directory.
(gdb) p *path
$1 = (AgentPath) {
  <Path> = {
    _vptr.Path = 0xe36390 <vtable for AgentPath+16>,
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          prev_ = 0x0
        }, <No data fields>}, <No data fields>},
    time_stamp_usecs_ = 1455067642637189
  },
  members of AgentPath:
  peer_ = 0x7f4644002e90,
  nh_ = {
    px = 0x7f463001fa90
  },
  label_ = 4294967295,
  vxlan_id_ = 6,
  dest_vn_name_ = {
    static npos = <optimized out>,
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
      _M_p = 0x7f4694108c78 "default-domain:commonmax:commonmax-vn31-001"
    }
  },
  sync_ = false,
  force_policy_ = false,
  sg_list_ = {
    <std::_Vector_base<int, std::allocator<int> >> = {
      _M_impl = {
        <std::allocator<int>> = {
          <__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>},
        members of std::_Vector_base<int, std::allocator<int> >::_Vector_impl:
        _M_start = 0x0,
        _M_finish = 0x0,
        _M_end_of_storage = 0x0
      }
    }, <No data fields>},
  tunnel_dest_ = {
    addr_ = {
      s_addr = 688592812
    }
  },
  tunnel_bmap_ = 8,
  tunnel_type_ = TunnelType::VXLAN,
  vrf_name_ = {
    static npos = <optimized out>,
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
      _M_p = 0x132a898 <std::string::_Rep::_S_empty_rep_storage@@GLIBCXX_3.4+24> ""
    }
  },
  gw_ip_ = {
    addr_ = {
      s_addr = 0
    }
  },
  unresolved_ = false,
  is_stale_ = false,
  is_subnet_discard_ = false,
  dependant_rt_ = {
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          prev_ = 0x0
        }, <No data fields>}, <No data fields>},
    self_ = 0x7f4644d07580,
    ptr_ = 0x0
  },
  path_preference_ = {
    sequence_ = 0,
    preference_ = PathPreference::HA_STALE,
    wait_for_traffic_ = false,
    ecmp_ = false,
    static_preference_ = false
  },
  local_ecmp_mpls_label_ = {
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          prev_ = 0x0
        }, <No data fields>}, <No data fields>},
    self_ = 0x7f4644d07580,
    ptr_ = 0x0
  },
  composite_nh_key_ = {
    px = 0x0
  },
  subnet_gw_ip_ = {
    type_ = boost::asio::ip::address::ipv4,
    ipv4_address_ = {
      addr_ = {
        s_addr = 0
      }
    },
    ipv6_address_ = {
      addr_ = {
        __in6_u = {
          __u6_addr8 = '\000' <repeats 15 times>,
          __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
          __u6_addr32 = {0, 0, 0, 0}
        }
      },
      scope_id_ = 0
    }
  },
  arp_mac_ = {
    static kZeroMac = {
      static kZeroMac = <same as static member of an already seen type>,
      static kBroadcastMac = {
        static kZeroMac = <same as static member of an already seen type>,
        static kBroadcastMac = <same as static member of an already seen type>,
        addr_ = {
          ether_addr_octet = "\377\377\377\377\377\377"
        }
      },
      addr_ = {
        ether_addr_octet = "\000\000\000\000\000"
      }
    },
    static kBroadcastMac = <same as static member of an already seen type>,
    addr_ = {
      ether_addr_octet = "\000\000\000\000\000"
    }
  },
  arp_interface_ = {
    px = 0x0
  },
  arp_valid_ = false,
  ecmp_suppressed_ = false
}
(gdb) p *path->peer_
$2 = (OvsPeer) {
  <Peer> = {
    _vptr.Peer = 0xe64c90 <vtable for OvsPeer+16>,
    type_ = Peer::OVS_PEER,
    name_ = {
      static npos = <optimized out>,
      _M_dataplus = {
        <std::allocator<char>> = {
          <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
        members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
        _M_p = 0x7f4644002cf8 "OVS-0.0.0.0"
      }
    },
    export_to_controller_ = true
  },
  members of OvsPeer:
  peer_ip_ = {
    type_ = boost::asio::ip::address::ipv4,
    ipv4_address_ = {
      addr_ = {
        s_addr = 0
      }
    },
    ipv6_address_ = {
      addr_ = {
        __in6_u = {
          __u6_addr8 = '\000' <repeats 15 times>,
          __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
          __u6_addr32 = {0, 0, 0, 0}
        }
      },
      scope_id_ = 0
    }
  },
  gen_id_ = 0,
  peer_manager_ = 0x7f46a0000ef0,
  ha_stale_export_ = true
}
(gdb) p *prev_front
$3 = (AgentPath) {
  <Path> = {
    _vptr.Path = 0xe36390 <vtable for AgentPath+16>,
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x7f46a518bd58,
          prev_ = 0x7f46a518bd58
        }, <No data fields>}, <No data fields>},
    time_stamp_usecs_ = 1455067567383942
  },
  members of AgentPath:
  peer_ = 0x7f4680001ff0,
  nh_ = {
    px = 0x7f463001fa90
  },
  label_ = 4294967295,
  vxlan_id_ = 6,
  dest_vn_name_ = {
    static npos = <optimized out>,
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
      _M_p = 0x7f4678e14ed8 "default-domain:commonmax:commonmax-vn31-001"
    }
  },
  sync_ = false,
  force_policy_ = false,
  sg_list_ = {
    <std::_Vector_base<int, std::allocator<int> >> = {
      _M_impl = {
        <std::allocator<int>> = {
          <__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>},
        members of std::_Vector_base<int, std::allocator<int> >::_Vector_impl:
        _M_start = 0x0,
        _M_finish = 0x0,
        _M_end_of_storage = 0x0
      }
    }, <No data fields>},
  tunnel_dest_ = {
    addr_ = {
      s_addr = 688592812
    }
  },
  tunnel_bmap_ = 8,
  tunnel_type_ = TunnelType::VXLAN,
  vrf_name_ = {
    static npos = <optimized out>,
    _M_dataplus = {
      <std::allocator<char>> = {
        <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
      members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
      _M_p = 0x132a898 <std::string::_Rep::_S_empty_rep_storage@@GLIBCXX_3.4+24> ""
    }
  },
  gw_ip_ = {
    addr_ = {
      s_addr = 0
    }
  },
  unresolved_ = false,
  is_stale_ = false,
  is_subnet_discard_ = false,
  dependant_rt_ = {
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          prev_ = 0x0
        }, <No data fields>}, <No data fields>},
    self_ = 0x7f4644d07580,
    ptr_ = 0x0
  },
  path_preference_ = {
    sequence_ = 1,
    preference_ = PathPreference::LOW,
    wait_for_traffic_ = false,
    ecmp_ = false,
    static_preference_ = false
  },
  local_ecmp_mpls_label_ = {
    node_ = {
      <boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {
        <boost::intrusive::detail::no_default_definer> = {<No data fields>},
        <boost::intrusive::list_node<void*>> = {
          next_ = 0x0,
          prev_ = 0x0
        }, <No data fields>}, <No data fields>},
    self_ = 0x7f4644d07580,
    ptr_ = 0x0
  },
  composite_nh_key_ = {
    px = 0x0
  },
  subnet_gw_ip_ = {
    type_ = boost::asio::ip::address::ipv4,
    ipv4_address_ = {
      addr_ = {
        s_addr = 0
      }
    },
    ipv6_address_ = {
      addr_ = {
        __in6_u = {
          __u6_addr8 = '\000' <repeats 15 times>,
          __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
          __u6_addr32 = {0, 0, 0, 0}
        }
      },
      scope_id_ = 0
    }
  },
  arp_mac_ = {
    static kZeroMac = {
      static kZeroMac = <same as static member of an already seen type>,
      static kBroadcastMac = {
        static kZeroMac = <same as static member of an already seen type>,
        static kBroadcastMac = <same as static member of an already seen type>,
        addr_ = {
          ether_addr_octet = "\377\377\377\377\377\377"
        }
      },
      addr_ = {
        ether_addr_octet = "\000\000\000\000\000"
      }
    },
    static kBroadcastMac = <same as static member of an already seen type>,
    addr_ = {
      ether_addr_octet = "\000\000\000\000\000"
    }
  },
  arp_interface_ = {
    px = 0x0
  },
  arp_valid_ = false,
  ecmp_suppressed_ = false
}
(gdb) p *prev_front->peer_
$4 = (BgpPeer) {
  <Peer> = {
    _vptr.Peer = 0xe541b0 <vtable for BgpPeer+16>,
    type_ = Peer::BGP_PEER,
    name_ = {
      static npos = <optimized out>,
      _M_dataplus = {
        <std::allocator<char>> = {
          <__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
        members of std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider:
        _M_p = 0x7f4634000d48 "172.23.10.193"
      }
    },
    export_to_controller_ = false
  },
  members of BgpPeer:
  server_ip_ = {
    addr_ = {
      s_addr = 3238664108
    }
  },
  id_ = 2,
  setup_time_ = 2530899035,
  bgp_xmpp_peer_ = {
    px = 0x7f4634002660,
    pn = {
      pi_ = 0x7f46340039e0
    }
  },
  is_disconnect_walk_ = {
    <tbb::internal::atomic_impl<bool>> = {
      my_storage = {
        my_value = false
      }
    }, <No data fields>},
  route_walker_ = {
    px = 0x7f4680001ac0
  }
}

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-15: [Review update] master

#2

Review in progress for https://review.opencontrail.org/17233
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-15: A change has been merged

#4

Reviewed: https://review.opencontrail.org/17233
Committed: http://github.org/Juniper/contrail-controller/commit/f9981edb0627b600b0a664173b77e9e09079556c
Submitter: Zuul
Branch: master

commit f9981edb0627b600b0a664173b77e9e09079556c
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Feb 15 14:40:16 2016 +0530

Fix parallel access to route path list

Issue:
------
Route Path list is getting modified from two threads at
the same time, resulting in path list sanity issues and
causing crash

Fix:
----
Adding task exclusion between "Agent::KSync" and
"Agent::RouteWalker" to resolve this issue.
TODO: this exculsion can be removed once handling for
dynamic peer is complete

Closes-Bug: 1543901
Change-Id: Ia5380210adfe868bcdc0559c21c5e938b3d445d3

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-16: [Review update] R2.20

#5

Review in progress for https://review.opencontrail.org/17290
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-16: [Review update] R2.21.x

#7

Review in progress for https://review.opencontrail.org/17291
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-16: [Review update] R2.22.x

#9

Review in progress for https://review.opencontrail.org/17292
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-17: A change has been merged

#11

Reviewed: https://review.opencontrail.org/17292
Committed: http://github.org/Juniper/contrail-controller/commit/5e7d003089daac3501c9bdbc820de03b08645cef
Submitter: Zuul
Branch: R2.22.x

commit 5e7d003089daac3501c9bdbc820de03b08645cef
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Feb 15 14:40:16 2016 +0530

Fix parallel access to route path list

Issue:
------
Route Path list is getting modified from two threads at
the same time, resulting in path list sanity issues and
causing crash

Fix:
----
Adding task exclusion between "Agent::KSync" and
"Agent::RouteWalker" to resolve this issue.
TODO: this exculsion can be removed once handling for
dynamic peer is complete

Conflicts:
src/vnsw/agent/cmn/agent.cc

Closes-Bug: 1543901
Change-Id: Ia5380210adfe868bcdc0559c21c5e938b3d445d3
(cherry picked from commit f9981edb0627b600b0a664173b77e9e09079556c)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-17:

#12

Reviewed: https://review.opencontrail.org/17290
Committed: http://github.org/Juniper/contrail-controller/commit/a9480663472b317138ebf1374af3f0352150723d
Submitter: Zuul
Branch: R2.20

commit a9480663472b317138ebf1374af3f0352150723d
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Feb 15 14:40:16 2016 +0530

Fix parallel access to route path list

Issue:
------
Route Path list is getting modified from two threads at
the same time, resulting in path list sanity issues and
causing crash

Fix:
----
Adding task exclusion between "Agent::KSync" and
"Agent::RouteWalker" to resolve this issue.
TODO: this exculsion can be removed once handling for
dynamic peer is complete

Conflicts:
src/vnsw/agent/cmn/agent.cc

Closes-Bug: 1543901
Change-Id: Ia5380210adfe868bcdc0559c21c5e938b3d445d3
(cherry picked from commit f9981edb0627b600b0a664173b77e9e09079556c)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-17:

#13

Reviewed: https://review.opencontrail.org/17291
Committed: http://github.org/Juniper/contrail-controller/commit/34ad87ec8413d5ed9ae32bd4f23f71510011f8ff
Submitter: Zuul
Branch: R2.21.x

commit 34ad87ec8413d5ed9ae32bd4f23f71510011f8ff
Author: Prabhjot Singh Sethi <email address hidden>
Date: Mon Feb 15 14:40:16 2016 +0530

Fix parallel access to route path list

Issue:
------
Route Path list is getting modified from two threads at
the same time, resulting in path list sanity issues and
causing crash

Fix:
----
Adding task exclusion between "Agent::KSync" and
"Agent::RouteWalker" to resolve this issue.
TODO: this exculsion can be removed once handling for
dynamic peer is complete

Conflicts:
src/vnsw/agent/cmn/agent.cc

Closes-Bug: 1543901
Change-Id: Ia5380210adfe868bcdc0559c21c5e938b3d445d3
(cherry picked from commit f9981edb0627b600b0a664173b77e9e09079556c)

Ashish Ranjan (aranjan-n) on 2016-02-25

information type:

Proprietary → Public

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.20	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r2.23 "r2.23"
R2.21.x	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r2.21.2
R2.22.x	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r2.22.1
Trunk	Fix Committed	High	Prabhjot Singh Sethi	Juniper Openstack r3.0-fcs

Juniper Openstack

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches