MAAS sometimes get in a state where machine tags aren't updated from their definitions
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
High
|
Björn Tillenius | ||
2.3 |
Fix Committed
|
Medium
|
Björn Tillenius | ||
2.4 |
Fix Committed
|
High
|
Björn Tillenius | ||
2.6 |
Fix Released
|
High
|
Björn Tillenius |
Bug Description
Now and then we get a failure in CI where it adds a machine tag with defintion=true(), which should apply to all machines.
I stopped the CI when the tags test failed and debugged what's going in. regiond sends an RPC request, EvaluateTag, to rackd. rackd then uses the MAAS API to get the node details, but gets a 503 from the node GET request.
https:/
There are nothing logged in regiond .log about the 503, but trying to add more tags result in the same error. The API itself is working fine, though, since I can use it to add the tags.
Another odd thing is that restarting maas-rackd makes the problem go away. After restarting maas-rackd I couldn't reproduce it, so I couldn't add any extra debug logging in the code.
Related branches
- Björn Tillenius: Approve
-
Diff: 193 lines (+96/-11)4 files modifiedsrc/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+67/-8)
src/provisioningserver/rpc/tags.py (+7/-2)
src/provisioningserver/rpc/tests/test_tags.py (+11/-0)
- MAAS Lander: Needs Fixing
- Björn Tillenius: Approve
-
Diff: 193 lines (+96/-11)4 files modifiedsrc/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+67/-8)
src/provisioningserver/rpc/tags.py (+7/-2)
src/provisioningserver/rpc/tests/test_tags.py (+11/-0)
- Björn Tillenius: Approve
- MAAS Lander: Pending (unittests) requested
-
Diff: 243 lines (+112/-21)4 files modifiedsrc/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+74/-16)
src/provisioningserver/rpc/tags.py (+6/-2)
src/provisioningserver/rpc/tests/test_tags.py (+21/-2)
- MAAS Lander: Needs Fixing
- Alberto Donato (community): Approve
-
Diff: 231 lines (+98/-15)4 files modifiedsrc/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+67/-13)
src/provisioningserver/rpc/tags.py (+3/-1)
src/provisioningserver/rpc/tests/test_tags.py (+17/-0)
Changed in maas: | |
status: | New → Triaged |
importance: | Undecided → High |
assignee: | nobody → Björn Tillenius (bjornt) |
milestone: | none → 2.7.0alpha1 |
Changed in maas: | |
status: | Triaged → In Progress |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
status: | Fix Committed → Fix Released |
The 2.4 tests also fail with this failure now and then.