Comment 5 for bug 1803964

Revision history for this message
Javier Peña (jpena-c) wrote :

Could we get the /var/log/cleaning_stacks.log file from the te-broker on November 19? We do not have HAProxy logs from RDO Cloud, but checking back nodepool logs from one of the cases I have found the following:

2018-11-19 04:03:27,512 DEBUG nodepool.NodeLauncher-0000303194: Node 0000303194 is running [region: regionOne, az: nova, ip: 38.145.33.10 ipv4: 38.145.33.10, ipv6: ]

Instance 52c2c700-6a4e-4f1f-b156-0f18eff30301 was created at that time, and nodepool tried to delete it at 06:07

2018-11-19 06:07:21,098 INFO nodepool.DeletedNodeWorker: Deleting used instance 52c2c700-6a4e-4f1f-b156-0f18eff30301 from rdo-cloud-tripleo

It got IP 38.145.33.10, however that same IP was used elsewhere 1 hour before (!):

2018-11-19 05:01:38,812 DEBUG nodepool.NodeLauncher-0000299388: Node 0000299388 is running [region: regionOne, az: nova, ip: 38.145.33.10 ipv4: 38.145.33.10, ipv6: ]
2018-11-19 05:08:12,925 DEBUG nodepool.NodeLauncher-0000303241: Node 0000303241 is running [region: regionOne, az: nova, ip: 38.145.33.10 ipv4: 38.145.33.10, ipv6: ]

That means that either the VM was deleted or the FIP was released way earlier than it should have been. Nodepool did not do it (at least there are no traces of it in the logs), so let's check if the ovb cleanup script did it.