[Queens -> Rocky][19.04] neutron-l3-agent is not restarted automatically which can result in impl_idl import errors during l3 agent operation

[Problem Description]

Upgraded a deployment from Queens to Rocky (via charm actions) and tried to create a router after that - all of its ports were down https://paste.ubuntu.com/p/G6cw3KzX6Q/

This looks like py2 -> py3 conversion related error - the necessary python2 packages were removed, however, the agent still runs with python2.7 (see below).

As soon as I restart the l3 agents on all nodes, ports are "up" as shown in `openstack port list --router <routername>`

juju run --application neutron-openvswitch 'sudo systemctl restart neutron-l3-agent'

I would expect neutron-dhcp-agent.service, neutron-openvswitch-agent.service and neutron-metadata-agent.service to potentially have similar issues because they also run with python2.7 after upgrade.

[Analysis]

Based on the l3-agent log I can see that this was due to import errors:

https://paste.ubuntu.com/p/CZbJHCh5rm/ (larger output)

2019-07-05 14:53:26.544 1146080 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-07-05 14:53:26.544 1146080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 562, in _process_router_update
# ...
2019-07-05 14:53:26.544 1146080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/api.py", line 30, in from_config
2019-07-05 14:53:26.544 1146080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 73, in import_module
2019-07-05 14:53:26.544 1146080 ERROR neutron.agent.l3.agent ImportError: No module named impl_idl

There are no python2.7 packages left with that module:

find /usr/lib/python* -name 'impl_idl.py'
/usr/lib/python3/dist-packages/neutron/agent/ovsdb/impl_idl.py
/usr/lib/python3/dist-packages/ovsdbapp/schema/open_vswitch/impl_idl.py
/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py
/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/impl_idl.py

root@adze:~# dpkg -l | grep ovsdb
ii python3-ovsdbapp 0.12.0-0ubuntu1~cloud0 all library for creating OVSDB applications - Python 3.x

python-ovsdbapp was removed during the upgrade:

/var/log/apt/term.log:Preparing to unpack .../060-python-ovsdbapp_0.12.0-0ubuntu1~cloud0_all.deb ...
/var/log/apt/term.log:Unpacking python-ovsdbapp (0.12.0-0ubuntu1~cloud0) over (0.9.1-0ubuntu1) ...
/var/log/apt/term.log:Setting up python-ovsdbapp (0.12.0-0ubuntu1~cloud0) ...
/var/log/apt/term.log:Selecting previously unselected package python3-ovsdbapp.
/var/log/apt/term.log:Preparing to unpack .../119-python3-ovsdbapp_0.12.0-0ubuntu1~cloud0_all.deb ...
/var/log/apt/term.log:Unpacking python3-ovsdbapp (0.12.0-0ubuntu1~cloud0) ...
/var/log/apt/term.log:Setting up python3-ovsdbapp (0.12.0-0ubuntu1~cloud0) ...
/var/log/apt/term.log:Removing python-ovsdbapp (0.12.0-0ubuntu1~cloud0) ...

State before a manual neutron-l3-agent restart (python2.7):

systemctl status neutron-l3-agent
● neutron-l3-agent.service - OpenStack Neutron L3 agent
   Loaded: loaded (/lib/systemd/system/neutron-l3-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-07-03 21:48:25 UTC; 1 day 18h ago
 Main PID: 1411876 (neutron-l3-agen)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/neutron-l3-agent.service
           ├─1411876 /usr/bin/python2.7 /usr/bin/neutron-l3-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini --config-file=/etc/neutron/
           └─1413914 /usr/bin/python2.7 /usr/bin/privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent.ini --config-file /etc/neutron/fw

After (python3.6):

systemctl status neutron-l3-agent
● neutron-l3-agent.service - OpenStack Neutron L3 agent
   Loaded: loaded (/lib/systemd/system/neutron-l3-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-07-05 16:02:18 UTC; 1s ago
 Main PID: 2085206 (neutron-l3-agen)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/neutron-l3-agent.service
           └─2085206 /usr/bin/python3.6 /usr/bin/neutron-l3-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini --config-file=/etc/neutron/

Jul 05 16:02:18 adze systemd[1]: Started OpenStack Neutron L3 agent.