Fuel for OpenStack

dockerctl backup fails if containers are already paused

Bug #1603842 reported by John Petrini
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Sustaining
Fuel for OpenStack 10.0
7.0.x
Fix Released
High
Anton Chevychalov
Fuel for OpenStack 7.0-mu-6
8.0.x
Fix Released
High
Anton Chevychalov
Fuel for OpenStack 8.0-mu-4
Mitaka
Invalid
High
Fuel Sustaining
Fuel for OpenStack 9.1

Bug Description

dockerctl backup attempts to pause the containers, but if the containers are already paused (for example from a previous failed backup attempt) dockerctl backup will fail.

What I'd expect: Either dockerctl backup should recognize that the containers are already paused and continue with the backup resuming the contains when finished or if for some reason this behavior is necessary then it should check the container status before starting the backup procedure so that the failure in immediate rather than half way through the lengthy backup procedure.

[root@fuel ~]# /usr/bin/dockerctl backup /var/log/fueldocker
Reading container data...
Committing fuel-core-8.0-mcollective...
db3bc3312d4cc0fccb4c20728209fb788901ccd45ded63c218e2da37964e0c45
Committing fuel-core-8.0-cobbler...
d6533f91aa299309d996c1d8d8b723a6d19521f7bcc41e9380ed61fcade717a5
Committing fuel-core-8.0-nginx...
077302afd1758ae9475dfc0d90344418b6cf8214a576bc9b37986e29ab434698
Committing fuel-core-8.0-ostf...
4e357316cfd3b07432fc0bda2c0520550fc046be6a44ffcb9d068761979a229a
Committing fuel-core-8.0-nailgun...
db703ca5d2e85f411a2263793342391dc477265f1723851255329a5bfcf301c8
Committing fuel-core-8.0-rsyslog...
02a4171d6c3e32bc56f623fb9f0954601be35aa913088a48e6b1759620407ad6
Committing fuel-core-8.0-astute...
098fdf5f47d73522c821a332687fedb1f17e8232fb4cfe96d46b55581fbc4ee5
Committing fuel-core-8.0-rsync...
1a393b8220fe60e31768622a34f0bca800539c879f4c42c11426c1cd8b997442
Committing fuel-core-8.0-keystone...
10d17561f1f821089f77df332e8cde9b772bb6d551a96b88b1f90ce6f83c07ee
Committing fuel-core-8.0-rabbitmq...
28372785bfd3ebd270160cb61d77c8ed71802c8949402a6380fa0cdaf24e0e5a
Committing fuel-core-8.0-postgres...
575d3fa14ade66a579b2ad392d9c4b5fe16cc7e1f8a45ffa7d8ccb61bfdf37ab
Saving containers to combined archive...
time="2016-07-18T01:13:18Z" level=debug msg="[stream] End of stdout"
Cleaning up temporary images...
Untagged: fuel/postgres_8.0_2016-07-18_0107:latest
Deleted: 575d3fa14ade66a579b2ad392d9c4b5fe16cc7e1f8a45ffa7d8ccb61bfdf37ab
Untagged: fuel/rabbitmq_8.0_2016-07-18_0107:latest
Deleted: 28372785bfd3ebd270160cb61d77c8ed71802c8949402a6380fa0cdaf24e0e5a
Untagged: fuel/keystone_8.0_2016-07-18_0107:latest
Deleted: 10d17561f1f821089f77df332e8cde9b772bb6d551a96b88b1f90ce6f83c07ee
Untagged: fuel/rsync_8.0_2016-07-18_0107:latest
Deleted: 1a393b8220fe60e31768622a34f0bca800539c879f4c42c11426c1cd8b997442
Untagged: fuel/astute_8.0_2016-07-18_0107:latest
Deleted: 098fdf5f47d73522c821a332687fedb1f17e8232fb4cfe96d46b55581fbc4ee5
Untagged: fuel/rsyslog_8.0_2016-07-18_0107:latest
Deleted: 02a4171d6c3e32bc56f623fb9f0954601be35aa913088a48e6b1759620407ad6
Untagged: fuel/nailgun_8.0_2016-07-18_0107:latest
Deleted: db703ca5d2e85f411a2263793342391dc477265f1723851255329a5bfcf301c8
Untagged: fuel/ostf_8.0_2016-07-18_0107:latest
Deleted: 4e357316cfd3b07432fc0bda2c0520550fc046be6a44ffcb9d068761979a229a
Untagged: fuel/nginx_8.0_2016-07-18_0107:latest
Deleted: 077302afd1758ae9475dfc0d90344418b6cf8214a576bc9b37986e29ab434698
Untagged: fuel/cobbler_8.0_2016-07-18_0107:latest
Deleted: d6533f91aa299309d996c1d8d8b723a6d19521f7bcc41e9380ed61fcade717a5
Untagged: fuel/mcollective_8.0_2016-07-18_0107:latest
Deleted: db3bc3312d4cc0fccb4c20728209fb788901ccd45ded63c218e2da37964e0c45
Pausing containers...
Error response from daemon: Cannot pause container 8f4a682011e8: Container 8f4a682011e8a4ae699341ad6e041c1debda6e5e80e1c02545957932c0cb836c is already paused
Error: failed to pause containers: [8f4a682011e8]
Error response from daemon: Cannot pause container 7d9c5f6dee19: Container 7d9c5f6dee19bc87df27345e878a39a91ff1e0e195bdafe1d8119b0746ef1dfe is already paused
Error: failed to pause containers: [7d9c5f6dee19]
Error response from daemon: Cannot pause container 1ce7b238fb66: Container 1ce7b238fb6629f71e65fd69574a40b162debc58b8204f12f641000744e9b435 is already paused
Error: failed to pause containers: [1ce7b238fb66]
Error response from daemon: Cannot pause container dbede3985db4: Container dbede3985db41e42192aa623fb3914e68bca783c33fb62f2ed38151c4e967216 is already paused
Error: failed to pause containers: [dbede3985db4]
Error response from daemon: Cannot pause container 3f0453364c84: Container 3f0453364c845d57ad214be74cb78aa5ea73ef7e3f8fd1cbed2574e79ca32134 is already paused
Error: failed to pause containers: [3f0453364c84]
Error response from daemon: Cannot pause container 8b0b08ae4135: Container 8b0b08ae41355e3d42f5b0dd5463f10c3d24968bf4f312f8e795723c90c10370 is already paused
Error: failed to pause containers: [8b0b08ae4135]
Error response from daemon: Cannot pause container 880325d381e9: Container 880325d381e9d4395e841d7691c48decfa4f5267bd3350158de4c50c0fb0c75a is already paused
Error: failed to pause containers: [880325d381e9]
Error response from daemon: Cannot pause container bc481b553b4a: Container bc481b553b4a7d78e99049075a4a3b3e9f51e2f9c24c4c52032bb479d87f5e2f is already paused
Error: failed to pause containers: [bc481b553b4a]
Error response from daemon: Cannot pause container 24ec877bdbc1: Container 24ec877bdbc189d6c23de4071d82cf83dffbeb31af58d2a038ad0a8925e7dd31 is already paused
Error: failed to pause containers: [24ec877bdbc1]
Error response from daemon: Cannot pause container 9f398899266e: Container 9f398899266edab33a8171733eb9a9c68d555cb5b2030a10ca07e2bdffad6cf0 is already paused
Error: failed to pause containers: [9f398899266e]
Error response from daemon: Cannot pause container 11626b9707f9: Container 11626b9707f9f81fc15cdfea760584e30ec9c41ebd8cd9a13e18a202864530d4 is already paused
Error: failed to pause containers: [11626b9707f9]
Backup failed!

Tags: docker
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: none → 10.0
status: New → Invalid
importance: Undecided → High
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
Revision history for this message
Denis Puchkin (dpuchkin) wrote :

simple workaround - unpause all container:

docker ps -a | grep Paused | cut -d' ' -f1 | xargs -n1 --no-run-if-empty docker unpause

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/357188

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/377411

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/377411
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6a82a30f391810ffa1d95adacefeb5cd2c4abd69
Submitter: Jenkins
Branch: stable/7.0

commit 6a82a30f391810ffa1d95adacefeb5cd2c4abd69
Author: Anton Chevychalov <email address hidden>
Date: Thu Aug 18 15:46:49 2016 +0300

    Fix dockerctl backup logic to remove double pausing

    This commit changes logic of pausing containers to prevent trying
    to pause already paused containers.
    If container was running before backup it will be paused and than unpaused.
    If container was paused before backup it will stay paused after backup.

    Change-Id: I78f428660366932b34002cd9985be79df8162c38
    Closes-Bug: #1603842

Ekaterina Shutova (eshutova)
tags: added: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/357188
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=10fb76cd373eff03843ed77381bc4873d05fbd26
Submitter: Jenkins
Branch: stable/8.0

commit 10fb76cd373eff03843ed77381bc4873d05fbd26
Author: Anton Chevychalov <email address hidden>
Date: Thu Aug 18 15:46:49 2016 +0300

    Fix dockerctl backup logic to remove double pausing

    This commit changes logic of pausing containers to prevent trying
    to pause already paused containers.
    If container was running before backup it will be paused and than unpaused.
    If container was paused before backup it will stay paused after backup.

    Change-Id: I78f428660366932b34002cd9985be79df8162c38
    Closes-Bug: #1603842

Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on MOS7.0 + mu6 updates.

Before:
[root@nailgun ~]# /usr/bin/dockerctl backup /var/log/fueldocker
Pausing containers...
Error response from daemon: Cannot pause container 1aa4d276d822: Container 1aa4d276d822ea1d2094b36555cf7c10b6295a3057398931e345cdbc53c80c8d is already paused
FATA[0000] Error: failed to pause container named 1aa4d276d822
Error response from daemon: Cannot pause container 49e39a65aad3: Container 49e39a65aad3590d180574dd500a645079be70a8065378f44741ec9fae542389 is already paused
FATA[0000] Error: failed to pause container named 49e39a65aad3
Error response from daemon: Cannot pause container 7aeecf62b111: Container 7aeecf62b1116caa6da39cb515bc6841776c9e35b1ff155e5de86fb24d7425a4 is already paused
FATA[0000] Error: failed to pause container named 7aeecf62b111
Error response from daemon: Cannot pause container 31b55fb30dbd: Container 31b55fb30dbddc62e49fc30b4310416482b8080a55c207a68660ceebbe81e6d3 is already paused
FATA[0000] Error: failed to pause container named 31b55fb30dbd
Error response from daemon: Cannot pause container 924ad0f790ee: Container 924ad0f790eeee65ac006ec1948808dcc8818e076332f6409239c370e1390e2f is already paused
FATA[0000] Error: failed to pause container named 924ad0f790ee
82cc8d4ef38d
2a3f8d574537
525741b5f4cc
fb42a86da6a3
a26c37b9717f
778fcd21972a
Backup failed!
After:
[root@nailgun ~]# /usr/bin/dockerctl backup /var/log/fueldocker
.....
NOTE: Following containers was not unpaused because they were paused before backup
90b06316d41e 41323e3b84f3 97ba40c27061
Backing up PostgreSQL database to /var/log/fueldocker/postgres_backup.sql...
Compressing archives...
/var/log/fueldocker/fuel_backup_2016-10-16_1504.tar - Compression Ratio: 35.843. Average Compression Speed: 77.000MB/s.
Total time: 00:00:01.59
Cleaning up...
Backup complete. File is available at /var/log/fueldocker/fuel_backup_2016-10-16_1504.tar.lrz

tags: removed: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on MOS 8.0 + mu4 updates.
Before backup failed because of errors that "Container is already paused". With fix:
...
NOTE: Following containers was not unpaused because they were paused before backup
3c632f6a405b e5dc8102e244 576e564a50c3
Backing up PostgreSQL database to /var/log/fueldocker/postgres_backup.sql...
Compressing archives...
Compressing system-dirs.tar postgres_backup.sql
Compression Ratio: 31.563. Average Compression Speed: 25.333MB/s.
Total time: 00:00:02.45
Cleaning up...
Backup complete. File is available at /var/log/fueldocker/fuel_backup_2017-02-14_1034.tar.lrz

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.