[eBay] Change contrail-vrouter rpm kernel module packaging

Additional feedback:
Upgrades of the contrail package do not work without manual intervention.
Removing the package and then installing the package works. Clean installs also work.

However there is big this way of packaging would not survive a kernel upgrade without manually creating the symlinks after that.
In that regard packaging is really broken.

eBay thinks they should be able to do the following without manual creation of symlinks, running depmod etc.
1) upgrade the package
2) upgrade the kernel

According to eBay, weak modules would solve this because they think that functionality intended for this problem.

Assuming the OS is centos7 and fab is used to provision, Could you please help with below questions...

Whats the kernel version being used?
Is this is a fresh installation or an upgrade? If upgrade, let us know from and to release please.

FYI: Contrail's recommended kernel version is 3.10.0-327.10.1.el7 for 3.0.2.x

Hi Nagendra,
please find the info bellow:

Kernel used: 3.10.0-327.28.3.el7.x86_64
OS: RHEL7
Packages: http://10.84.5.120/github-build/R3.0.2.x/8/redhat70/liberty/contrail-install-packages-3.0.2.1-8~liberty.el7.noarch.rpm
Installation: custom puppet scripts using standard Linux distro commands

All of the above hasn't been disapproved by Contrail Engineering, which is closely working with eBay on their project. Please check with Sree, if some of those are deemed unsupported.

Regards,
Mladen

Upgrade procedure:

* Put new rpms in repository
* Run yum clean all on compute node
* Run yum upgrade -y
In our case: The vrouter.ko / symlink is not put in /usr/lib/modules/3.10.0-327.28.3.el7.x86_64/extra/net/vrouter/vrouter.ko

(We used ansible for the commands, something like: ansible -i inventory.txt all -m shell -a 'yum clean all;yum upgrade -y' --become -f 20 )

If you look at the rpm script ( rpm -q --scripts contrail-vrouter ) you will see there is some scripting in place which normally creates the symlinks and runs a depmod -a.
Running this script manually after installation will make it work, also doing an uninstall / reinstall will work.

However this script will (also) not run when upgrading the kernel so you will also need a manual run of that script when upgrading kernels.
As mentioned before using "weak modules" in the packaging could be a solution to that

Ebay upgraded kernel from officially supported Contrail kernel, but it the future it can really be any (minor version) kernel, as they will always run latest version that's patched with security updates.

They upgraded from build 6 to build 8, when issue observed.

Here's exact upgrade procedure, for this particular node, "as is", provided by eBay:

The server was installed from scratch from our provisioning with 3.10.0-327.28.3.el7.x86_64 kernel (the kernel was installed/upgraded before contrail was installed) and it ran contrail-vrouter-agent-3.0.2.1-6

The following updates were installed:
Sep 14 08:21:20 Updated: python-contrail-3.0.2.1-8.el7.x86_64
Sep 14 08:21:20 Updated: contrail-lib-3.0.2.1-8.el7.x86_64
Sep 14 08:21:22 Updated: contrail-vrouter-agent-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: contrail-vrouter-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: contrail-nodemgr-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: contrail-utils-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: python-contrail-vrouter-api-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: python-opencontrail-vrouter-netns-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: contrail-vrouter-utils-3.0.2.1-8.el7.x86_64
Sep 14 08:21:24 Updated: contrail-vrouter-init-3.0.2.1-8.el7.x86_64
Sep 14 08:21:25 Updated: contrail-setup-3.0.2.1-8.el7.noarch
Sep 14 08:21:25 Updated: contrail-vrouter-common-3.0.2.1-8.el7.noarch
Sep 14 08:21:25 Updated: contrail-openstack-vrouter-3.0.2.1-8.el7.noarch

Issue has been triggered when they upgraded form build 6 to build 8, without kernel change (3.10.0-327.28.3.el7.x86_64).

Distro details can be found bellow – it’s Centos but for some reason they’ve been using EL packages from the start (~130 MB). Let us known whether that’s problem.

# cat /etc/contrail/agent_param
LOG=/var/log/contrail.log
CONFIG=/etc/contrail/contrail-vrouter-agent.conf
prog=/usr/bin/contrail-vrouter-agent
kmod=vrouter
pname=contrail-vrouter-agent
LIBDIR=/usr/lib64
DEVICE=vhost0
dev=em2
vgw_subnet_ip=__VGW_SUBNET_IP__
vgw_intf=__VGW_INTF_LIST__
LOGFILE=--log-file=/var/log/contrail/vrouter.log

# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

Review in progress for https://review.opencontrail.org/24466
Submitter: Nagendra Prasath (<email address hidden>)

Review in progress for https://review.opencontrail.org/24464
Submitter: Nagendra Prasath (<email address hidden>)

Review in progress for https://review.opencontrail.org/24463
Submitter: Nagendra Prasath (<email address hidden>)

Review in progress for https://review.opencontrail.org/24465
Submitter: Nagendra Prasath (<email address hidden>)

Upgraded package still creates the symlink to vrouter.ko but the preun of the old package removes them due to RPM scriptlet order which is not handled.

Reviewed: https://review.opencontrail.org/24466
Committed: http://github.org/Juniper/contrail-packages/commit/889a72a9b22f35f6321a4c22c6df67b5ca564c90
Submitter: Zuul
Branch: master

commit 889a72a9b22f35f6321a4c22c6df67b5ca564c90
Author: Nagendra Chandran <email address hidden>
Date: Mon Sep 26 16:21:53 2016 -0700

Remove symlink to vrouter.ko only during uninstallation

RPM Scriplets execution order during upgrade, installs the new package first
followed by new packages's postinst and then preun script of old package
Because of this, symlink gets removed during contrail upgrade

Closes-Bug: 1623413
Change-Id: Ie9f13c3025b58e159b78aee3250576b12616ca7a

Reviewed: https://review.opencontrail.org/24465
Committed: http://github.org/Juniper/contrail-packages/commit/a4f15182190ab52e2b8d4c55e8f0946cbd8af4d1
Submitter: Zuul
Branch: R3.0

commit a4f15182190ab52e2b8d4c55e8f0946cbd8af4d1
Author: Nagendra Chandran <email address hidden>
Date: Mon Sep 26 16:20:52 2016 -0700

Remove symlink to vrouter.ko only during uninstallation

RPM Scriplets execution order during upgrade, installs the new package first
followed by new packages's postinst and then preun script of old package
Because of this, symlink gets removed during contrail upgrade

Closes-Bug: 1623413
Change-Id: Ie88ae9662589256a5246883e77e3a4120850b68e

Reviewed: https://review.opencontrail.org/24464
Committed: http://github.org/Juniper/contrail-packages/commit/23da97bec55900cdbf4bd95bffc75677133154dc
Submitter: Zuul
Branch: R3.1

commit 23da97bec55900cdbf4bd95bffc75677133154dc
Author: Nagendra Chandran <email address hidden>
Date: Mon Sep 26 16:18:05 2016 -0700

Remove symlink to vrouter.ko only during uninstallation

RPM Scriplets execution order during upgrade, installs the new package first
followed by new packages's postinst and then preun script of old package
Because of this, symlink gets removed during contrail upgrade

Closes-Bug: 1623413
Change-Id: I9869b5642724fdaae7867e548ccdceeeaa9d0be2

Reviewed: https://review.opencontrail.org/24463
Committed: http://github.org/Juniper/contrail-packages/commit/319eba8deb0d776324d360a25d5cf241ca38cbee
Submitter: Zuul
Branch: R3.0.2.x

commit 319eba8deb0d776324d360a25d5cf241ca38cbee
Author: Nagendra Chandran <email address hidden>
Date: Mon Sep 26 16:13:56 2016 -0700

Remove symlink to vrouter.ko only during uninstallation

RPM Scriplets execution order during upgrade, installs the new package first
followed by new packages's postinst and then preun script of old package
Because of this, symlink gets removed during contrail upgrade

Closes-Bug: 1623413
Change-Id: I122129608e8fd4749968b610987131162c62e46e

I will be out of office till 16th October 2016, back in office on 17th October 2016.
For any assistance with Server Manager, please send email to “<email address hidden>”.

Thanks,

Abhay

On Oct 13, 2016, at 9:47 PM, Vinay Mahuli <email address hidden> wrote:

> ** Changed in: juniperopenstack/r3.0
>    Milestone: r3.0.3.0 => r3.0.4.0
>
> ** Also affects: juniperopenstack/r3.0.3.x
>   Importance: Undecided
>       Status: New
>
> ** Changed in: juniperopenstack/r3.0.3.x
>       Status: New => Fix Committed
>
> ** Changed in: juniperopenstack/r3.0.3.x
>   Importance: Undecided => High
>
> ** Changed in: juniperopenstack/r3.0.3.x
>     Assignee: (unassigned) => Nagendra Prasath (npchandran)
>
> ** Changed in: juniperopenstack/r3.0.3.x
>    Milestone: None => r3.0.3.1
>
> --
> You received this bug notification because you are a member of Contrail
> Systems engineering, which is subscribed to Juniper Openstack.
> https://bugs.launchpad.net/bugs/1623413
>
> Title:
>  [eBay] Change contrail-vrouter rpm kernel module packaging
>
> Status in Juniper Openstack:
>  Fix Committed
> Status in Juniper Openstack r3.0 series:
>  Fix Committed
> Status in Juniper Openstack r3.0.2.x series:
>  Fix Committed
> Status in Juniper Openstack r3.0.3.x series:
>  Fix Committed
> Status in Juniper Openstack r3.1 series:
>  Fix Committed
> Status in Juniper Openstack trunk series:
>  Fix Committed
>
> Bug description:
>  After installing contrail-install-packages-3.0.2.1-8-liberty.el7 in order to fix following bug:
>  https://bugs.launchpad.net/juniperopenstack/+bug/1621816
>  eBay reported that installer scripts are not working properly.
>
>  When installing updating to a new version of a contrail-vrouter.rpm
>  the kernel module can no longer be found.
>
>  It seems that the postinstall script which creates the symlink for the latest kernel is not executed.
>
>  Since they now run a more recent kernel modprobe cannot find the file and they need to manually fix this.
>
>  eBay suggests that proper way of handling this on RedHat machines is making use of weak-modules which should work across kernel versions.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juniperopenstack/+bug/1623413/+subscriptions

Review in progress for https://review.opencontrail.org/25067
Submitter: Nagendra Prasath (<email address hidden>)

Reviewed: https://review.opencontrail.org/25067
Committed: http://github.org/Juniper/contrail-packages/commit/bcee6ef9003529e3336f40eafeac04728844d7a6
Submitter: Zuul
Branch: R2.21.x

commit bcee6ef9003529e3336f40eafeac04728844d7a6
Author: Nagendra Chandran <email address hidden>
Date: Mon Sep 26 16:21:53 2016 -0700

Remove symlink to vrouter.ko only during uninstallation

RPM Scriplets execution order during upgrade, installs the new package first
followed by new packages's postinst and then preun script of old package
Because of this, symlink gets removed during contrail upgrade

Closes-Bug: 1623413
(cherry picked from commit 889a72a9b22f35f6321a4c22c6df67b5ca564c90)

Centos/RHEL: Support kernel version 3.10.0-327.10.1.el7.x86_64 in R2.21.x
Closes-Bug: 1635019
Change-Id: Ie9f13c3025b58e159b78aee3250576b12616ca7a