DVR floatingip to rule priority association lost on Agent restart

Critical to have it in MU-4

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ryan Moats <email address hidden>
Review: https://review.fuel-infra.org/20308

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Adolfo Duarte <email address hidden>
Review: https://review.fuel-infra.org/20309

Customer found in MOS 7.0 - need fix and check for future versions.

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

This could affects every MOS 7 customer, with at least 4 customers already hit it. Raising the priority to Critical. Please include into MU 4 for MOS 7.

Check on 9-0:
[root@fuel ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 134
cat /etc/fuel_build_number:
 134
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 liberty-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6321.noarch
 rubygem-astute-9.0.0-1.mos730.noarch
 fuel-library9.0-9.0.0-1.mos8223.noarch
 shotgun-9.0.0-1.mos85.noarch
 fuel-agent-9.0.0-1.mos270.noarch
 fuel-ui-9.0.0-1.mos2626.noarch
 fuel-migrate-9.0.0-1.mos8223.noarch
 nailgun-mcagents-9.0.0-1.mos730.noarch
 fuel-misc-9.0.0-1.mos8223.noarch
 python-packetary-9.0.0-1.mos128.noarch
 fuel-bootstrap-cli-9.0.0-1.mos270.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8597.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos128.noarch
 fuel-openstack-metadata-9.0.0-1.mos8597.noarch
 fuel-notify-9.0.0-1.mos8223.noarch
 fuel-setup-9.0.0-1.mos6321.noarch
 python-fuelclient-9.0.0-1.mos299.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-9.0.0-1.mos6321.noarch
 fuel-utils-9.0.0-1.mos8223.noarch
 fuel-nailgun-9.0.0-1.mos8597.noarch
 fuel-ostf-9.0.0-1.mos919.noarch

Steps:
1) Boot vm
2) Associate floating ip to it
3) Restart l3 agent
4) Boot vm2
5) Associate floating ip to it
6) Check that floatings has different priorities
 sudo ip netns exec qrouter-a05dd452-4ff2-465c-a172-66916310bb0b ip rule s
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
57480: from 10.0.1.5 lookup 16
57481: from 10.0.1.3 lookup 16

Invalid for 8.0-updates as the fix is already present in 8.0 branch (consumed from stable/liberty)

Steps to reproduce (taken from original bug, worked for me):
1. Spawn a VM and associate a floatingip1
2. Restart the Agent
3. Spawn a second VM with floatingip2
4. Disassociate the floatingip2

--> Floatingip1 stops working sometimes

OK, proposed patches doesn't fix the issue described in message https://bugs.launchpad.net/neutron/+bug/1414779/comments/5 . We need detailed steps to reproduce for this issue.

After fix was applied and node was rebootedb:
root@node-4:~# ip netns exec qrouter-dfcb7f7e-1215-475b-8aa1-0891408bfc93 ip rule s
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
57480: from 192.168.111.14 lookup 16
57481: from 192.168.111.15 lookup 16
3232263937: from 192.168.111.1/24 lookup 3232263937

Changing assignee to Kevin Benton (the one who pushed a patch).

There are two patches required, please ensure that you are using both together:

Introduce ItemAllocator class: https://review.fuel-infra.org/#/c/20308/

Preserve DVR FIP rule priority over Agent restarts: https://review.fuel-infra.org/#/c/20309/

The second patch depends on the first one.

Also, this won't fix the issue if the floating IPs were allocated before applying the patch. The way that patch works it will only protect floating IPs allocated after that patch has been applied.

If you want it to recover floating IPs that were allocated before the patch, you will need to run the netns cleanup script.

I have installed package from temporary [1]. It looks like there is no second patch [2], so I haven't tested both patches. Sorry for mistake.

[1] repo http://perestroika-repo-tst.infra.mirantis.net/review/LP-1414779/mos-repos/ubuntu/7.0
[2] https://review.fuel-infra.org/#/c/20309/

No, both patches [1] and [2] are available in temporary repo.

OK, fix is confirmed:
  - FIP rule priorities are in correct order (new rule has correct priority) as in [1]
  - Old floating IP still works as in [2]

[1] https://bugs.launchpad.net/neutron/+bug/1414779/comments/5
[2] https://bugs.launchpad.net/neutron/+bug/1414779/comments/3

Reviewed: https://review.fuel-infra.org/20308
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 2d1edb6dfb9fcbe66c1ed4ae690f43a5f56b929d
Author: Ryan Moats <email address hidden>
Date: Tue May 3 23:45:08 2016

Introduce ItemAllocator class

The ItemAllocator class is used as the base class for
LinkLocalAllocator in preparation for adding
FipRulePriorityAllocator as a child class.

Change-Id: I2c77e5a895f750845b46d3e8a2326e01ea87ee78
Partial-Bug: #1414779
Partial-Bug: #1577985
Signed-off-by: Ryan Moats <email address hidden>
(cherry picked from commit 2093d8727070829064b1a6005a9b411a383d246d)

Reviewed: https://review.fuel-infra.org/20309
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 09b9730f68df23512fb6124fa1f707c167ba764c
Author: Adolfo Duarte <email address hidden>
Date: Tue May 3 23:45:08 2016

Preserve DVR FIP rule priority over Agent restarts

IP rule priorities assigned to DVR floating IPs need
to be preserved over L3 agent restarts. Reuse
the ItemAllocator class decomposed from Link Local IP
address allocation. Also move commn unit tests to
ItemAllocator class.

Conflicts:
 neutron/agent/l3/dvr_router.py

Closes-Bug: #1414779
Closes-Bug: #1577985
Change-Id: I6a75aa8ad612ee80b391f0a27a8a7e29519c3f8d
Co-Authored-By: Rajeev Grover <email address hidden>
Co-Authored-By: Ryan Moats <email address hidden>
(cherry picked from commit de81ab8385e8490b2320e23ee7dd86b43e22fd32)

Verified on MOS 7.0 + MU5 updates.

Steps to verify:
1) Boot VM1, create and associate floatingip1 to it
2) Restart l3 agent on appropriate compute node
3) Boot VM2, create and associate floatingip2 to it
4) Check floatings priorities
5) Disassociate the floatingip2
6) Check floatings rules

Actual results before MU5 updates:
1) VM1 with IP 192.168.111.15 and floatingip1 is created.
3) VM2 with IP 192.168.111.16 and floatingip2 is created.
4) floatings have the same priorities for both VMs:
root@node-2:~# ip netns exec qrouter-26a8635b-1ae5-47cc-b7c1-bdc8728829ad ip rule s
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
32768: from 192.168.111.15 lookup 16
32768: from 192.168.111.16 lookup 16
3232263937: from 192.168.111.1/24 lookup 3232263937
6) After floatingip2 was disassociated from VM2, VM1's lookup rule is deleted:
root@node-2:~# ip netns exec qrouter-26a8635b-1ae5-47cc-b7c1-bdc8728829ad ip rule s
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
32768: from 192.168.111.16 lookup 16
3232263937: from 192.168.111.1/24 lookup 3232263937

Actual results after MU5 updates:
1) VM1 with IP 192.168.111.7 and floatingip1 is created.
3) VM2 with IP 192.168.111.8 and floatingip2 is created.
4) Floatings have different priorities:
root@node-4:~# ip netns exec qrouter-97e3a6ff-78aa-4113-b4ad-26706acf675c ip rule s
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
57480: from 192.168.111.8 lookup 16
57481: from 192.168.111.7 lookup 16
3232263937: from 192.168.111.1/24 lookup 3232263937
6) After floatingip2 was disassociated from VM2, VM2's lookup rule is deleted:
root@node-4:~# ip netns exec qrouter-97e3a6ff-78aa-4113-b4ad-26706acf675c ip rule s
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
57481: from 192.168.111.7 lookup 16
3232263937: from 192.168.111.1/24 lookup 3232263937

Test case is automated: https://review.gerrithub.io/#/c/288824/