docker-puppet.py: rabbitmq -q list_users run during stack update
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| tripleo |
Incomplete
|
High
|
James Slagle | ||
Bug Description
If there's been a configuration change to rabbitmq that causes docker-puppet.py to reconfigure rabbitmq during a stack-update, it could fail because the puppet manifest uses "rabbitmqctl -q list_users". It will fail if rabbitmq is not running. This is triggered in the scenario where there is an initial failed deployment prior to rabbitmq starting, the user does a necessary change and attempts to redeploy, which is a stack-update now.
The update triggers this from puppet-
if $step >= 2 {
# In case of HA, starting of rabbitmq-server is managed by pacemaker, because of which, a dependency
# to Service[
if $stack_action == 'UPDATE' {
# Required for changing password on update scenario. Password will be changed only when
# called explicity, if the rabbitmq service is already running.
rabbitmq_user { $rabbitmq_user :
password => $rabbitmq_pass,
provider => 'rabbitmqctl',
admin => true,
}
}
Even though we have noop'd Rabbitmq_user, for some reason, the above resource still runs during a puppet "prefetch" step.
The debug output from docker-puppet.py ends up looking like:
Debug: Prefetching rabbitmqctl resources for rabbitmq_user
Debug: Executing: '/usr/sbin/
Debug: Command failed, retrying
which eventually times out and fails the docker-puppet.py run after 3 minutes.
| Changed in tripleo: | |
| status: | New → In Progress |
| importance: | Undecided → High |
| assignee: | nobody → James Slagle (james-slagle) |
| milestone: | none → queens-3 |
| James Slagle (james-slagle) wrote | #1 |
| James Slagle (james-slagle) wrote | #2 |
| Changed in tripleo: | |
| milestone: | queens-3 → queens-rc1 |
| Changed in tripleo: | |
| milestone: | queens-rc1 → rocky-1 |
| Changed in tripleo: | |
| milestone: | rocky-1 → rocky-2 |
| Changed in tripleo: | |
| milestone: | rocky-2 → rocky-3 |
| Changed in tripleo: | |
| milestone: | rocky-3 → rocky-rc1 |
| Changed in tripleo: | |
| milestone: | rocky-rc1 → stein-1 |
| Changed in tripleo: | |
| milestone: | stein-1 → stein-2 |
| Changed in tripleo: | |
| milestone: | stein-2 → stein-3 |
| Changed in tripleo: | |
| milestone: | stein-3 → stein-rc1 |
| Changed in tripleo: | |
| milestone: | stein-rc1 → train-1 |
| Changed in tripleo: | |
| milestone: | train-1 → train-2 |
| Changed in tripleo: | |
| milestone: | train-2 → train-3 |
| Changed in tripleo: | |
| milestone: | train-3 → ussuri-1 |
| Changed in tripleo: | |
| milestone: | ussuri-1 → ussuri-2 |
| Changed in tripleo: | |
| milestone: | ussuri-2 → ussuri-3 |
| Changed in tripleo: | |
| milestone: | ussuri-3 → ussuri-rc3 |
| Changed in tripleo: | |
| milestone: | ussuri-rc3 → victoria-1 |
| Changed in tripleo: | |
| milestone: | victoria-1 → victoria-3 |
| Changed in tripleo: | |
| milestone: | victoria-3 → wallaby-1 |
| Changed in tripleo: | |
| milestone: | wallaby-1 → wallaby-2 |
| Changed in tripleo: | |
| milestone: | wallaby-2 → wallaby-3 |
| Marios Andreou (marios-b) wrote | #3 |
| Changed in tripleo: | |
| milestone: | wallaby-3 → none |
| status: | In Progress → Incomplete |
I suspect that during any stack-update where docker-puppet.py reruns for rabbitmq we are running "rabbitmqctl -q list_users", the problem is just usually not seen since rabbitmq had been previously started.