Charm is making sysctl calls in LXD containers (unsupported)

Bug #1829047 reported by Ryan Farrell
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Neutron Gateway Charm
Fix Released
High
Chris MacNaughton

Bug Description

I am following the guide outlined here: https://docs.openstack.org/charm-guide/latest/openstack-on-lxd.html

Steps
-----
# install deps <not shown>
# get openstack-on-lxd source configs
git clone https://github.com/openstack-charmers/openstack-on-lxd

# Configure LXD on Host
echo fs.inotify.max_queued_events=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_instances=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf
echo vm.swappiness=1 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
sudo lxd init
lxc profile device set default eth0 mtu 9000

# Bootstrap Juju
juju bootstrap --config config.yaml localhost lxd
cat lxd-profile.yaml | lxc profile edit juju-default

# Install Openstack
juju deploy bundle-bionic-queens.yaml
# Get coffee...
-----

Once deployment stabilizes'juju status' will show \
    neutron-gateway/0* error idle 15 10.0.8.251 hook failed: "config-changed"

From Log
--------
1879 2019-05-13 21:52:18 DEBUG juju-log Updating sysctl_file: /etc/sysctl.d/50-quantum-gateway.conf values: {'net.ipv4.neigh.default.gc_thresh1': 128, 'net.ipv4.neigh.default.gc_thresh2': 28672, 'net.ipv 4.neigh.default.gc_thresh3': 32768, 'net.ipv6.neigh.default.gc_thresh1': 128, 'net.ipv6.neigh.default.gc_thresh2': 28672, 'net.ipv6.neigh.default.gc_thresh3': 32768, 'net.nf_conntrack_max': 1000000, 'net.netfilter.nf_conntrack_buckets': 204800, 'net.netfilter.nf_conntrack_max': 1000000}
 1880 2019-05-13 21:52:18 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh1: No such file or directory
 1881 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh2: No such file or directory
 1882 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh3: No such file or directory
 1883 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory
 1884 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory
 1885 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory
 1886 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/nf_conntrack_max: No such file or directory
 1887 2019-05-13 21:52:19 DEBUG config-changed sysctl: setting key "net.netfilter.nf_conntrack_buckets": No such file or directory
 1888 2019-05-13 21:52:19 DEBUG config-changed net.netfilter.nf_conntrack_max = 1000000
 1889 2019-05-13 21:52:19 DEBUG config-changed Traceback (most recent call last):
 1890 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/config-changed", line 362, in <module>
 1891 2019-05-13 21:52:19 DEBUG config-changed hooks.execute(sys.argv)
 1892 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
 1893 2019-05-13 21:52:19 DEBUG config-changed self._hooks[hook_name]()
 1894 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1546, in wrapped_f
 1895 2019-05-13 21:52:19 DEBUG config-changed stopstart, restart_functions)
 1896 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/host.py", line 741, in restart_on_change_helper
 1897 2019-05-13 21:52:19 DEBUG config-changed r = lambda_f()
 1898 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1545, in <lambda>
 1899 2019-05-13 21:52:19 DEBUG config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
 1900 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
 1901 2019-05-13 21:52:19 DEBUG config-changed return f(*args, **kwargs)
 1902 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/config-changed", line 128, in config_changed
 1903 2019-05-13 21:52:19 DEBUG config-changed '/etc/sysctl.d/50-quantum-gateway.conf')
 1904 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/sysctl.py", line 65, in create
 1905 2019-05-13 21:52:19 DEBUG config-changed check_call(call)
 1906 2019-05-13 21:52:19 DEBUG config-changed File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
 1907 2019-05-13 21:52:19 DEBUG config-changed raise CalledProcessError(retcode, cmd)
 1908 2019-05-13 21:52:19 DEBUG config-changed subprocess.CalledProcessError: Command '['sysctl', '-p', '/etc/sysctl.d/50-quantum-gateway.conf']' returned non-zero exit status 255.
 1909 2019-05-13 21:52:19 ERROR juju.worker.uniter.operation runhook.go:132 hook "config-changed" failed: exit status 1

Full Logs attached

Revision history for this message
Ryan Farrell (whereisrysmind) wrote :
David Ames (thedac)
Changed in charm-neutron-gateway:
status: New → Triaged
importance: Undecided → High
milestone: none → 19.07
Revision history for this message
David Ames (thedac) wrote :

TRIAGE:

The sysctl commands fail when neutron-gateway is deployed in an LXC.

We have patterns in other charms to handle this. See hardening. We simply need a try/except for the sysctl -p or we need to gate based detecting containerization.

Revision history for this message
Jamon Camisso (jamon) wrote :

I ran into this bug when upgrading from 18.11 to 19.04 charms. I juju set sysctl="" and that resolved the issues.

Checking for container=lxc in /proc/1/environ seems straightforward enough. If possible a backport to 19.04 would save (me) some work and I suspect problems for others who haven't upgraded yet.

Changed in charm-neutron-gateway:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-gateway (master)

Fix proposed to branch: master
Review: https://review.opendev.org/659722

Changed in charm-neutron-gateway:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-gateway (master)

Reviewed: https://review.opendev.org/659722
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=a47f400e1513a1ca37ae9adfd84006c95acc2fb4
Submitter: Zuul
Branch: master

commit a47f400e1513a1ca37ae9adfd84006c95acc2fb4
Author: Chris MacNaughton <email address hidden>
Date: Fri May 17 07:56:31 2019 +0200

    Ensure we only try to set sysctls outside of a container

    THe sysctls are not available inside of a container, so we
    should not set them.

    Change-Id: I93e1052958a8c8fb44c0589caf7cb5d6a6019ab8
    Closes-Bug: #1829047

Changed in charm-neutron-gateway:
status: In Progress → Fix Committed
Revision history for this message
Dan Ackerson (dan.ackerson) wrote :

Confirm same issue when running `juju deploy bundle-bionic-rocky.yaml`.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

Dan, which bundle is that, and is it using the next or stable charms? This has been committed (and so released to cs:~openstack-charmers-next) but won't go to the stable charms for a few more weeks.

David Ames (thedac)
Changed in charm-neutron-gateway:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.