[upgrade] Postgres and Cobbler containers remain non-stateless after upgrade
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Vladimir Kuklin | ||
6.1.x |
Fix Released
|
Critical
|
Fuel Library (Deprecated) | ||
7.0.x |
Fix Released
|
Critical
|
Vladimir Kuklin |
Bug Description
upgrade.sh prints the following message at the end of successful master node upgrade:
_______
Both updates and security repos were setuped. If you want to
retrieve latest updates make sure you have access to these repos
and run:
# yum update
# dockerctl destroy all
# dockerctl start all
_______
This procedure completely REMOVES ALL already existing environments from Nailgun DB on master node.
Root cause: upgrade script does not create the directories, used as external mount points for Cobbler and Postgres. So, the contents of these directories remains inside the Docker file system.
description: | updated |
Changed in fuel: | |
assignee: | nobody → Fuel for Openstack (fuel) |
tags: | removed: feature-upgrade |
description: | updated |
Changed in fuel: | |
assignee: | Fuel for Openstack (fuel) → Fuel Python Team (fuel-python) |
Changed in fuel: | |
status: | New → Confirmed |
Changed in fuel: | |
assignee: | Fuel Python Team (fuel-python) → Matthew Mosesohn (raytrac3r) |
status: | Confirmed → In Progress |
Changed in fuel: | |
assignee: | Fuel Library Team (fuel-library) → Evgeniy L (rustyrobot) |
status: | Triaged → In Progress |
Changed in fuel: | |
assignee: | Evgeniy L (rustyrobot) → Vladimir Kuklin (vkuklin) |
summary: |
- [upgrade] Remove "dockerctl destroy all" advice message from upgrade - script + [upgrade] Postgres and Cobbler containers remain non-stateless after + upgrade |
description: | updated |
Changed in fuel: | |
status: | In Progress → Fix Committed |
tags: | added: on-verification |
tags: | added: to-rca |
Found the root cause of incorrect 'dockerctl destroy all' results. fuel/container_ data/${ release} /
It appears, during upgrade puppet code does not creates end directories to store that data, which we moved outside postgres, cobbler and astute containers in order to make them stateless.
As a result we getting empty
/var/lib/
directory without ./cobbler, ./postgres and ./astute sub-directories.
During upgrade and after, Docker starts containers and complains nothing about the absent mount points. Instead, inside containers it thinks the appropriate directories are mount points, while actually these directories remain in the local internal container file system.
We need to fix puppet code to create the necessary directories (postgres, astute and cobbler) inside fuel/container_ data/${ release} /
/var/lib/
The message 'dockerctl destroy all' is actually correct for 6.1 and normally should not lead to data loss.