baremetal devstack kubelet can't do network probes to pods
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| kuryr-kubernetes |
Fix Released
|
High
|
Unassigned | ||
Bug Description
On baremetal devstack:
- The kubelet runs in the host and its networking
- The host networking can only access Neutron ports via FIPs
- The pods that are not marked as HostNetworking=True run on Neutron ports in the 'private-network'
The above facts mean that all network readiness and liveness probes from the kubelet to the pods will fail. As such, the containers in the pods will be endlessly restarted and end in crashloop backoff.
This problem also affects non-devstack baremetal deployments.
As a workaround you can:
1. Create a port for each kubelet in your baremetal cluster in the pod subnet
2. Create an OVS port on br-int on each worker node and bind the port created in step 1 to the newly created ovs port (if your deployment uses hybrid firewall driver, that means creating a bridge, one veth pair and adding one side of the veth pair as a port to ovs br-int.
3. Make sure the port is up and add the IP address to it.
The above steps will have created a link scoped route to the internal network that kubelet will automatically start using.
| Antoni Segura Puimedon (celebdor) wrote | #1 |
| Changed in kuryr-kubernetes: | |
| importance: | Undecided → High |
| milestone: | none → pike-3 |
| status: | New → Triaged |
| Antoni Segura Puimedon (celebdor) wrote | #2 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to kuryr-kubernetes (master) | #3 |
| Changed in kuryr-kubernetes: | |
| status: | Triaged → Fix Released |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/kuryr-kubernetes 0.2.0 | #4 |
As for a proper resolution to this bug. I suppose that we should do better than the workaround. My late in the night thought is as follows:
1. Create a subnet for baremetal worker nodes with its SG
2. Create a port and bind it per worker node
3. Add a Probe handler that watches pod events and adds SGs for the liveness and readiness probes allowing the whole SG of the worker nodes access to that specific port.