MAAS CLI command fails with nonce already used error

Future Occurrences can be found in: https://solutions.qa.canonical.com/bugs/2033532

Bumping to high as we have an environment to debug it now.
[Jacopo] We can check the DB for when this Nounce has been used before.

Current status & what we see happening:
MAAS client issues a request to MAAS API with a unique nonce. This request appears to be stuck for 52 seconds, after which it drops with HTTP 419 (page expired). MAAS client appears to retry the connection using cached parameters, which leads to the "nonce already used" error.

There is nothing with a 52 second timeout in MAAS that we know of, so we suspect there is an intermediate party (caching proxy?) that is involved. We requested a tcpdump from the client side capturing the traffic when this issue is reproducible.

It appears that a restart of MAAS in TOR3 reduced the probability of occurrence of this issue.

The root cause has been tracked to the unintentional use of an internal proxy, that introduced a delay and a retry with an already-used nonce. The occurrence of this issue in the test lab has dropped off significantly (to zero, as far as we know) after the clients have been pointed to the appropriate MAAS endpoints instead of the internal proxy. Closing this for now based on this investigation. If the issue reoccurs for whatever reason, we will reopen the investigation.