L3 HA router doesn't work over VXLAN/L2 pop after rebooting the controller with primary router

This change is proposed for stable/mitaka https://review.openstack.org/#/c/382210/, so when it will be merged in upstream and we can get it with sync.

Removed 8.0-mu4 nomination, since we don't have a fix to backport and it is not clear what is the root cause of this issue (I have checked upstream's review comments).

Reviewed: https://review.fuel-infra.org/28774
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 61dc356a30d803b4c1ec3bbcbbf83f202427bc79
Author: Ann Kamyshnikova <email address hidden>
Date: Wed Nov 23 09:40:42 2016

Merge the tip of origin/stable/mitaka into origin/9.0/mitaka

2ee6e58 Fix "failed unplugging ha interface" error when deleting router
fd082cd Avoid trace in _notify_l3_agent_ha_port_update
8834107 LinuxBridge: Pass host into get_devices_details_list
0010301 Ensure there are fdb_entries before iterating
9dd4ad1 Update metadata proxy when subnet add/delete
c06ff65 l2pop fdb flows for HA router ports

Closes-Bug: #1639025

Change-Id: If47e63124d783bc1a2a2c0109547ae35b7358f5a

Fix landed in Upstream by https://review.openstack.org/#/c/255237/

Verified on:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-notify-9.0.0-1.mos8642.noarch
 fuel-nailgun-9.0.0-1.mos8911.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-ui-9.0.0-1.mos2842.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-utils-9.0.0-1.mos8642.noarch
 python-packetary-9.0.0-1.mos157.noarch
 fuel-library9.0-9.0.0-1.mos8642.noarch
 fuel-migrate-9.0.0-1.mos8642.noarch
 python-fuelclient-9.0.0-1.mos363.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-mirror-9.0.0-1.mos157.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8911.noarch
 fuel-openstack-metadata-9.0.0-1.mos8911.noarch
 rubygem-astute-9.0.0-1.mos782.noarch
 fuel-9.0.0-1.mos6359.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 fuel-misc-9.0.0-1.mos8642.noarch
 nailgun-mcagents-9.0.0-1.mos782.noarch
 fuel-setup-9.0.0-1.mos6359.noarch

Original upstream bug #1522980 is about general issue: it states that HA with l2pop is provided only if control plane is operational. Original bug's reporter says that we shouldn't rely on database, messaging server and neutron-server reliability when providing HA.

Upstream developers used 3 patches to solve reported issue:

1. https://review.openstack.org/#/c/255237/ is used to implement additional method to create flood flaws (not it can be done not only with l2pop) and to change the way neutron-server, DB and messenger are used to organize HA.
2. Patches https://review.openstack.org/#/c/323993/ and https://review.openstack.org/#/c/339982/ are used to rename existing functions and DB tables, but don't change any neutron's behaviour. No need to backport.

I have analyzed changes made in the first patch and it looks like they change neutron internal processes significantly. On the other hand, the overall request looks like a feature to me (we improve existing neutron's features). Since we have no reliable tests to cover all the variety of possible issues this patch can introduce, I think that this bug should be closed as Won't Fix for 8.0-updates.