rack controller remained in degraded state

Hi Jason,

Is this still an issue? have you seen it again?

Also, even if degraded, MAAS should be able to continue to work. Are there any issues (e.g. machines failed to deploy, commission, etc).

We haven't seen it in a while, but reporting 'degraded' is a bug
either way, even if we're not seeing any failures. We have to be able
to rely on maas to tell us if it's healthy or not. If the flag doesn't
matter, then get rid of it.
On Thu, Aug 23, 2018 at 8:56 AM Andres Rodriguez
<email address hidden> wrote:
>
> ** Changed in: maas
> Milestone: 2.5.0alpha2 => 2.5.0beta1
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1781241
>
> Title:
> [2.4] rack controller remained in degraded state
>
> Status in MAAS:
> Incomplete
>
> Bug description:
> This is with 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1.
>
> After an HA deployment, one rack controller (qtmb3d) remained in a
> degraded state for over 20 minutes (the test timed out after that)
> (92% connected).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1781241/+subscriptions

Being in degraded state doesn't mean there's a *bug*. Being degraded means that there are a lot of things that could be wrong, bug or not. If there are no failures, that's good, because despite MAAS being in a degraded state, MAAS is working just fine.

e.g. a MAAS will be in degraded state if it is expecting 3 region controllers, but one of them is down. This doesn't mean there's a bug, this means there's a potential problem and MAAS is not working in its full potential because of a missing region.

Sure - it means there is something we should investigate. We should never
be in a degraded state on a fresh deploy in our test lab.

On Wed, Sep 5, 2018 at 4:21 PM Andres Rodriguez <email address hidden>
wrote:

> Being in degraded state doesn't mean there's a *bug*. Being degraded
> means that there are a lot of things that could be wrong, bug or not. If
> there are no failures, that's good, because despite MAAS being in a
> degraded state, MAAS is working just fine.
>
> e.g. a MAAS will be in degraded state if it is expecting 3 region
> controllers, but one of them is down. This doesn't mean there's a bug,
> this means there's a potential problem and MAAS is not working in its
> full potential because of a missing region.
>
> ** Changed in: maas
> Milestone: 2.5.0beta1 => 2.5.0beta2
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1781241
>
> Title:
> [2.4] rack controller remained in degraded state
>
> Status in MAAS:
> Incomplete
>
> Bug description:
> This is with 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1.
>
> After an HA deployment, one rack controller (qtmb3d) remained in a
> degraded state for over 20 minutes (the test timed out after that)
> (92% connected).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1781241/+subscriptions
>

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team

We saw this again on maas_2.7.0-8232-g.6e1dba4ab-0ubuntu1~18.04.1

Saw this again here: https://solutions.qa.canonical.com/#/qa/testRun/d044f76b-a0be-48f3-9bc0-f9502541323b

From the logs DNS on the first maas node went down and the node never recovered. Link to the artifacts from the deployment at the bottom.

Pretty sure we're seeing this again in this run: https://solutions.qa.canonical.com/#/qa/testRun/98caba03-f81c-4be3-8cc7-04e285059dd3

Saw this again again here with the same issues as Michael's run above: https://solutions.qa.canonical.com/#/qa/testRun/c2bd543c-afd2-4212-87d2-afd89cac8bd0

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)

We hit it again today. One of the controllers shows a lot of messages not able to reach postgres VIP and regiond is down. That controller is shown as down and two others as degraded

Running 3.2/stable snap (3.2.6-12016-g.19812b4da) on jammy

https://oil-jenkins.canonical.com/artifacts/72c89cde-5289-46eb-9600-ec5c338d30ca/generated/generated/maas/logs-2023-01-11-07.49.25.tgz

I hit this issue yesterday on 3.5. All the controllers were constantly changing their state from running to degraded or dead.

Attached is the log I managed to save.

Due to the lack of time, I went with 3.4. Then tried to deploy 3.5 again to collect an SOS report. After the third redeployment, everything went smoothly, and I didn't reproduce the issue.