Rerunning the overcloud deploy command with no changes restarts a
truckload of containers (first seen this via https://bugzilla.redhat.com/show_bug.cgi?id=1612960). So we really have
three separate issues here. Below is the list of all the containers that
may restart needlessly (at least what I have observed in my tests):
A) cron category:
ceilometer_agent_notification cinder_api cinder_api_cron cinder_scheduler
heat_api heat_api_cfn heat_api_cron heat_engine keystone keystone_cron
logrotate_crond nova_api nova_api_cron nova_conductor nova_consoleauth
nova_metadata nova_scheduler nova_vnc_proxy openstack-cinder-volume-docker-0
panko_api
These end up being restarted because in the config volume for the container there is
a cron file and cron files are generated with a timestamp inside:
$ cat /var/lib/config-data/puppet-generated/keystone/var/spool/cron/keystone
...
# HEADER: This file was autogenerated at 2018-08-07 11:44:57 +0000 by puppet.
...
We fix this by repiping tar into 'tar xO' and grepping away any line
that starts with # HEADER.
B) swift category:
swift_account_auditor swift_account_reaper swift_account_replicator
swift_account_server swift_container_auditor swift_container_replicator
swift_container_server swift_container_updater swift_object_auditor
swift_object_expirer swift_object_replicator swift_object_server
swift_object_updater swift_proxy swift_rsync
So the swift containers restart because when recalculating the md5 over the
/var/lib/config-data/puppet-generated/swift folder we also include:
B.1) /etc/swift/backups/... which is a folder which over time collects backup of the ringfiles
B.2) /etc/swift/*.gz it seems that the *.gz files seem to change over time
We just add a parameter to the tar command to exclude those files as
we do not need to trigger a restart if those files change.
--exclude='*/etc/swift/backups/*' --exclude='*/etc/swift/*.gz'
C) libvirt category:
nova_compute nova_libvirt nova_migration_target nova_virtlogd
This one seems to be due to the fact that the /etc/libvirt/passwd.db file contains a timestamp and
even when we disable a user and passwd.db does not exist, it gets
created:
[root@compute-1 nova_libvirt]# git diff cb2441bb1caf7572ccfd870561dcc29d7819ba04..0c7441f30926b111603ce4d4b60c6000fe49d290 .
passwd.db changes do not need to trigger a restart of the container se
we can safely exclude this file from any md5 calculation.
Part C) was: Co-Authored-By: Martin Schupper <email address hidden>
We only partial-bug this one because we want a cleaner fix where
exceptions to the files being checksummed will be specified in the tht
service files.
Partial-Bug: #1786065
Tested as follows:
./overcloud_deploy.sh
tripleo-ansible-inventory --static-yaml-inventory inv.yaml
ansible -f1 -i inv.yaml -m shell --become -a "docker ps --format=\"{{ '{{' }}.Names{{ '}}' }}: {{ '{{' }}.CreatedAt{{ '}}' }}\" | sort" overcloud > before
./overcloud_deploy.sh
ansible -f1 -i inv.yaml -m shell --become -a "docker ps --format=\"{{ '{{' }}.Names{{ '}}' }}: {{ '{{' }}.CreatedAt{{ '}}' }}\" | sort" overcloud > after
diff -u before after | wc -l
0
Change-Id: I10f5cacd9fee94d804ebcdffd0125676f5a209c4
(cherry picked from commit 42c3f180514da6aed5410a26f17db32d6ac1a62f)
(cherry picked from commit 2aa664e45878bba930b4c9eecb48532fb19ec7b7)
Reviewed: https:/ /review. openstack. org/599292 /git.openstack. org/cgit/ openstack/ tripleo- heat-templates/ commit/ ?id=be05b9bc375 aabd1936d238733 b7dad09f042e21
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit be05b9bc375aabd 1936d238733b7da d09f042e21
Author: Michele Baldessari <email address hidden>
Date: Wed Aug 8 21:04:53 2018 +0200
Make redeploy idempotent
Rerunning the overcloud deploy command with no changes restarts a /bugzilla. redhat. com/show_ bug.cgi? id=1612960). So we really have agent_notificat ion cinder_api cinder_api_cron cinder_scheduler cinder- volume- docker- 0
truckload of containers (first seen this via
https:/
three separate issues here. Below is the list of all the containers that
may restart needlessly (at least what I have observed in my tests):
A) cron category:
ceilometer_
heat_api heat_api_cfn heat_api_cron heat_engine keystone keystone_cron
logrotate_crond nova_api nova_api_cron nova_conductor nova_consoleauth
nova_metadata nova_scheduler nova_vnc_proxy openstack-
panko_api
These end up being restarted because in the config volume for the container there is config- data/puppet- generated/ keystone/ var/spool/ cron/keystone
a cron file and cron files are generated with a timestamp inside:
$ cat /var/lib/
...
# HEADER: This file was autogenerated at 2018-08-07 11:44:57 +0000 by puppet.
...
The timestamp is unfortunately hard coded into puppet in both the cron provider and the parsedfile /github. com/puppetlabs/ puppet/ blob/master/ lib/puppet/ provider/ cron/crontab. rb#L127 /github. com/puppetlabs/ puppet/ blob/master/ lib/puppet/ provider/ parsedfile. rb#L104
provider:
https:/
https:/
We fix this by repiping tar into 'tar xO' and grepping away any line
that starts with # HEADER.
B) swift category: account_ auditor swift_account_ reaper swift_account_ replicator account_ server swift_container _auditor swift_container _replicator container_ server swift_container _updater swift_object_ auditor object_ expirer swift_object_ replicator swift_object_server object_ updater swift_proxy swift_rsync
swift_
swift_
swift_
swift_
swift_
So the swift containers restart because when recalculating the md5 over the lib/config- data/puppet- generated/ swift folder we also include: backups/ ... which is a folder which over time collects backup of the ringfiles
/var/
B.1) /etc/swift/
B.2) /etc/swift/*.gz it seems that the *.gz files seem to change over time
We just add a parameter to the tar command to exclude those files as '*/etc/ swift/backups/ *' --exclude= '*/etc/ swift/* .gz'
we do not need to trigger a restart if those files change.
--exclude=
C) libvirt category: target nova_virtlogd
nova_compute nova_libvirt nova_migration_
This one seems to be due to the fact that the /etc/libvirt/ passwd. db file contains a timestamp and 2ccfd870561dcc2 9d7819ba04. .0c7441f30926b1 11603ce4d4b60c6 000fe49d290 .
even when we disable a user and passwd.db does not exist, it gets
created:
[root@compute-1 nova_libvirt]# git diff cb2441bb1caf757
passwd.db changes do not need to trigger a restart of the container se
we can safely exclude this file from any md5 calculation.
Part C) was: Co-Authored-By: Martin Schupper <email address hidden>
We only partial-bug this one because we want a cleaner fix where
exceptions to the files being checksummed will be specified in the tht
service files.
Partial-Bug: #1786065
Tested as follows: _deploy. sh ansible- inventory --static- yaml-inventory inv.yaml _deploy. sh
./overcloud
tripleo-
ansible -f1 -i inv.yaml -m shell --become -a "docker ps --format=\"{{ '{{' }}.Names{{ '}}' }}: {{ '{{' }}.CreatedAt{{ '}}' }}\" | sort" overcloud > before
./overcloud
ansible -f1 -i inv.yaml -m shell --become -a "docker ps --format=\"{{ '{{' }}.Names{{ '}}' }}: {{ '{{' }}.CreatedAt{{ '}}' }}\" | sort" overcloud > after
diff -u before after | wc -l
0
Change-Id: I10f5cacd9fee94 d804ebcdffd0125 676f5a209c4 ed5410a26f17db3 2d6ac1a62f) 930b4c9eecb4853 2fb19ec7b7)
(cherry picked from commit 42c3f180514da6a
(cherry picked from commit 2aa664e45878bba