Docker upgrade script crashes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Evgeniy L |
Bug Description
Docker upgrade script (run_upgrade.sh) didn't perform upgrade and died with the following message. Moreover it left the node in non working condition rather than revert containers back to 5.0
2014-05-13 11:24:03 DEBUG 17839 (upgrade) Create container from image fuel/nailgun_5.1: {'command': 'manage.py syncdb', 'ports': None}
2014-05-13 11:24:03 DEBUG 17839 (upgrade) Start container "2b85f065420bf9
2014-05-13 11:25:12 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:26:17 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:27:23 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:28:28 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:29:34 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:30:39 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:31:45 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:32:50 WARNING 17839 (upgrade) Failed to run container "2b85f065420bf9
2014-05-13 11:32:52 ERROR 17839 (upgrade) Upgrade failed: Failed to execute migraion command "manage.py syncdb" exit code 1 container id 2b85f065420bf90
2014-05-13 11:32:52 ERROR 17839 (upgrade) Traceback (most recent call last):
File "/web/fuel-
self.upgrade()
File "/web/fuel-
self.
File "/web/fuel-
self.
File "/web/fuel-
retries_
File "/web/fuel-
params.
DockerExecutedE
[root@nailgun ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2b85f065420b fuel/nailgun_
2a60b4b141b9 fuel/postgres_
a6e59e48b321 busybox:latest true 15 minutes ago Exited (0) 15 minutes ago fuel-core-
d48d27ee8d83 fuel/mcollectiv
1bd8ba40e1d9 fuel/cobbler_
5c1acf9e2cff fuel/nginx_
1a11a276f85a fuel/ostf_
22a804bd08b5 fuel/nailgun_
0fb2e623cff0 fuel/rsyslog_
fa09e6be7b72 fuel/astute_
4e0cdd283a29 fuel/rsync_
50bd9bf1c5bf fuel/rabbitmq_
6307daabe3df fuel/postgres_
13ed3378e8a1 storage/
ab2a9212e8fe storage/repo:latest /bin/sh -c '/bin/ech 43 minutes ago Exited (0) 43 minutes ago fuel-storage-
773ea8a3e813 storage/dump:latest /bin/sh -c '/bin/ech 43 minutes ago Exited (0) 43 minutes ago fuel-storage-
Changed in fuel: | |
importance: | Undecided → Critical |
assignee: | nobody → Eugeny L. (meskaline) |
milestone: | none → 5.0 |
Changed in fuel: | |
assignee: | Eugeny L. (meskaline) → Evgeniy L (rustyrobot) |
Changed in fuel: | |
status: | New → Confirmed |
Changed in fuel: | |
status: | Confirmed → In Progress |
Regarding to migration error, I build new containers, checked on 198 iso, it works fine now.
We don't run rollback automatically right now, because in this case it's harder to debug why upgrade failed, you can check that rollback works manually, here is the instruction
https:/ /github. com/rustyrobot/ fuel-web/ blob/4bp/ fuel-upgarde/ fuel_upgrade_ system/ fuel_upgrade/ README. md#how- to-run- rollback- manually
If you will see this error again, provide additional information
* make sure that fuel-core- 5.1-postgresql container is up and running
* run "docker run -it fuel/nailgun_5.1 manage.py syncdb" to check that migration works