commit 1121a59edb48fe133df14bfd4384eef04ce687a7
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 7 17:27:16 2018 -0500
Remove allocations before setting vm_status to SHELVED_OFFLOADED
Tempest is intermittently failing a test which does the
following:
1. Create a server.
2. Shelve offload it.
3. Unshelve it.
Tempest waits for the server status to be SHELVED_OFFLOADED
before unshelving the server, which goes through the
scheduler to pick a compute node and claim resources on it.
When shelve offloading a server, the resource allocations
for the instance and compute node it was on are cleared, which
will also delete the internal consumer record in the placement
service.
The race is that the allocations are removed during shelve
offload *after* the server status changes to SHELVED_OFFLOADED.
This leaves a window where unshelve is going through the
scheduler and gets the existing allocations for the instance,
which are non-empty and have a consumer generation. The
claim_resources method in the scheduler then uses that
consumer generation when PUTing the allocations. That PUT
fails because in between the GET and PUT of the allocations,
placement has deleted the internal consumer record. When
PUTing the new allocations with a non-null consumer generation,
placement returns a 409 conflict error because for a new
consumer it expects the "consumer_generation" parameter to be
None.
This change handles the race by simply making sure the allocations
are deleted (along with the related consumer record in placement)
*before* the instance.vm_status is changed.
Change-Id: I2a6ccaff904c1f0759d55feeeef0ec1da32c65df
Closes-Bug: #1798688
(cherry picked from commit 6369f39244558b147f7b0796269d9a86ce9b12d8)
Reviewed: https:/ /review. opendev. org/c/openstack /nova/+ /771985 /opendev. org/openstack/ nova/commit/ 1121a59edb48fe1 33df14bfd4384ee f04ce687a7
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/rocky
commit 1121a59edb48fe1 33df14bfd4384ee f04ce687a7
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 7 17:27:16 2018 -0500
Remove allocations before setting vm_status to SHELVED_OFFLOADED
Tempest is intermittently failing a test which does the
following:
1. Create a server.
2. Shelve offload it.
3. Unshelve it.
Tempest waits for the server status to be SHELVED_OFFLOADED
before unshelving the server, which goes through the
scheduler to pick a compute node and claim resources on it.
When shelve offloading a server, the resource allocations
for the instance and compute node it was on are cleared, which
will also delete the internal consumer record in the placement
service.
The race is that the allocations are removed during shelve generation" parameter to be
offload *after* the server status changes to SHELVED_OFFLOADED.
This leaves a window where unshelve is going through the
scheduler and gets the existing allocations for the instance,
which are non-empty and have a consumer generation. The
claim_resources method in the scheduler then uses that
consumer generation when PUTing the allocations. That PUT
fails because in between the GET and PUT of the allocations,
placement has deleted the internal consumer record. When
PUTing the new allocations with a non-null consumer generation,
placement returns a 409 conflict error because for a new
consumer it expects the "consumer_
None.
This change handles the race by simply making sure the allocations
are deleted (along with the related consumer record in placement)
*before* the instance.vm_status is changed.
Change-Id: I2a6ccaff904c1f 0759d55feeeef0e c1da32c65df 47f7b0796269d9a 86ce9b12d8)
Closes-Bug: #1798688
(cherry picked from commit 6369f39244558b1