Fuel for OpenStack

[upgrade] Upgrade failed with /tmp/upgrade/pg_dump_all.sql: No such file or directory when run after rollback

Bug #1467051 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Python (Deprecated)
Fuel for OpenStack 7.0
6.1.x
Won't Fix
High
Fuel Python (Deprecated)
Fuel for OpenStack 6.1-updates
7.0.x
Invalid
High
Fuel Python (Deprecated)
Fuel for OpenStack 7.0

Bug Description

Upgrade failed if run it right after rollback

Steps:
1. Create and deploy any 6.0-58 cluster
2. Upload upgrade 6.1 script and run it with fake rollback upgrader
UPGRADERS='host-system docker openstack raise-error' ./upgrade.sh --password *****
3. After successful rollback run upgrade again
./upgrade.sh --no-rollback --password *****

Expected - upgrade is successful
Actual - it failed with
2015-06-20 05:22:26 ERROR 21873 (cli) Shell command executed with "1" exit code: dockerctl shell c5ea9370733a23ad06e9449fbb6b33f3d8f0a49a041351e136d4b4ab041f0e1e su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"
Traceback (most recent call last):
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 176, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/cli.py", line 168, in run_upgrade
    upgrade_manager.run()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
    upgrader.upgrade()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 79, in upgrade
    self.create_and_start_new_containers()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 273, in create_and_start_new_containers
    self.run_after_container_creation_command(container)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 288, in run_after_container_creation_command
    '', retries=30, interval=4)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 329, in exec_with_retries
    return func()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 284, in execute
    self.exec_cmd_in_container(container['container_name'], command)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/docker_engine.py", line 296, in exec_cmd_in_container
    utils.exec_cmd("dockerctl shell {0} {1}".format(db_container_id, cmd))
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
    _wait_and_check_exit_code(cmd, child)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
    'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: dockerctl shell c5ea9370733a23ad06e9449fbb6b33f3d8f0a49a041351e136d4b4ab041f0e1e su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"

Logs are attached

Nastya Urlapova (aurlapova)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Ok, looks like a bug in Docker itself. Here's what happened:

1. fuel_upgrade fails to import database dump due to "file not found"

2. we don't have this file inside container under /tmp/upgrade. moreover, we don't have any files inside this folder

3. /tmp/upgrade resource is mounted from host's /var/lib/fuel_upgrade/6.1/

4. /var/lib/fuel_upgrade/6.1/ contains all needed file

5. docker inspect shows, that this folder is indeed mounted resource

    "/tmp/upgrade": "/var/lib/docker/vfs/dir/df6b4cd6a4ecb5f1a244f7bf9dcd4de0c2c755a65d7da34324f574fad87ad564"

6. The virtual fs resource is empty "/var/lib/docker/vfs/dir/df6b4cd6a4ecb5f1a244f7bf9dcd4de0c2c755a65d7da34324f574fad87ad564".

Andrey Sledzinskiy (asledzinskiy)
tags: added: docs
tags: added: release-notes
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Andrii, how ofter this bug is reproduced? Perhaps we can set medium prio if it wasn't reproduced since then?

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Unfortunately, I can't say about reproduces because during system tests refactoring this test disappeared from CI.
It had 100% reproduces and we can try to reproduce it again

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Sorry, I found that test in SWARM
It is constantly reproducing

Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Doesn't reproduced for 7.0 on CI. Move it to Invalid.

Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

Won't fix for 6.x series as they are unsupported and there was no progress on the issue.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.