Activity log for bug #2039952

Date Who What changed Old value New value Message
2023-10-20 09:51:28 Marian Gasparovic bug added bug
2023-10-20 11:29:18 Marian Gasparovic description MAAS 3.3.5 We noticed in test runs that our staticaly defined VIP addresses for openstack services sometimes don't resolve and a minute later they are fine. Turns out bind servers restarts often, tens of times a day. ``` $ grep "starting BIND" named.log*|awk '{print $1}'|sed "s/.*://"|sort|uniq -c 15 12-Oct-2023 67 13-Oct-2023 175 14-Oct-2023 170 15-Oct-2023 34 16-Oct-2023 41 17-Oct-2023 36 18-Oct-2023 73 19-Oct-2023 15 20-Oct-2023 ``` Christian had a look and he found that BIND is getting restarted frequently because MAAS checks the serial for each zone it updates, if it fails the check on 2 additional retries, BIND is restarted. But reason is unknown. To show how it is hitting us I ran a script which digs six statically defined records once a minute, it failed 56 times during 12 hours, there were several cases when they were not resolving for two minutes. MAAS 3.3.5 We noticed in test runs that our staticaly defined VIP addresses for openstack services sometimes don't resolve and a minute later they are fine. Turns out bind server restarts often, tens of times a day. ``` $ grep "starting BIND" named.log*|awk '{print $1}'|sed "s/.*://"|sort|uniq -c      15 12-Oct-2023      67 13-Oct-2023     175 14-Oct-2023     170 15-Oct-2023      34 16-Oct-2023      41 17-Oct-2023      36 18-Oct-2023      73 19-Oct-2023      15 20-Oct-2023 ``` Christian had a look and he found that BIND is getting restarted frequently because MAAS checks the serial for each zone it updates, if it fails the check on 2 additional retries, BIND is restarted. But reason is unknown. To show how it is hitting us I ran a script which digs six statically defined records once a minute, it failed 56 times during 12 hours, there were several cases when they were not resolving for two minutes.
2023-10-20 11:34:45 Marian Gasparovic attachment added sosreport-maas-lab1-lp2039952-2023-10-20-inazyqy.tar.xz https://bugs.launchpad.net/maas/+bug/2039952/+attachment/5711731/+files/sosreport-maas-lab1-lp2039952-2023-10-20-inazyqy.tar.xz
2023-10-20 11:45:45 Mauricio Faria de Oliveira bug added subscriber Mauricio Faria de Oliveira
2023-10-26 08:19:20 Jerzy Husakowski maas: importance Undecided Medium
2023-10-26 08:19:20 Jerzy Husakowski maas: status New Triaged
2023-10-26 08:19:20 Jerzy Husakowski maas: milestone 3.5.0
2023-10-26 08:19:40 Jerzy Husakowski nominated for series maas/3.3
2023-10-26 08:19:40 Jerzy Husakowski bug task added maas/3.3
2023-10-26 08:19:40 Jerzy Husakowski nominated for series maas/3.4
2023-10-26 08:19:40 Jerzy Husakowski bug task added maas/3.4
2023-10-26 08:19:47 Jerzy Husakowski maas/3.3: status New Triaged
2023-10-26 08:19:51 Jerzy Husakowski maas/3.4: status New Triaged
2023-10-26 08:19:54 Jerzy Husakowski maas/3.3: importance Undecided Medium
2023-10-26 08:19:56 Jerzy Husakowski maas/3.4: importance Undecided Medium
2023-10-26 08:20:04 Jerzy Husakowski maas/3.3: milestone 3.4.x
2023-10-26 08:20:14 Jerzy Husakowski maas/3.3: milestone 3.4.x 3.3.x
2023-10-26 08:20:19 Jerzy Husakowski maas/3.4: milestone 3.4.x
2023-10-31 09:20:26 Jeff Lane  bug added subscriber Rod Smith
2023-10-31 09:29:09 Jeff Lane  attachment added sosreport-weavile-LP2039952-2023-10-31-dawvpqs.tar.xz https://bugs.launchpad.net/maas/+bug/2039952/+attachment/5714600/+files/sosreport-weavile-LP2039952-2023-10-31-dawvpqs.tar.xz
2023-11-09 11:34:05 Adam Collard maas: assignee Christian Grabowski (cgrabowski)
2023-11-09 11:34:08 Adam Collard maas: importance Medium High