volume backup process block
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Cinder |
Incomplete
|
Low
|
Unassigned | ||
Bug Description
When using a volume to create a backup, the whole process can't end due to child process is blocking.
Reproduce the problem through the following steps.
1)[root]# cinder create 7 --image b3b811d5-
2)[root]# cinder backup-create --name test_backup 61061b93-
+------
| Property | Value |
+------
| id | c33140e9-
| name | test_backup |
| volume_id | 61061b93-
+------
3)[root]# cinder list --all | grep test_backup
| 61061b93-
4)[root]# cinder backup-show c33140e9-
+------
| Property | Value |
+------
| availability_zone | nova |
| container | None |
| created_at | 2020-11-
| data_timestamp | 2020-11-
| description | None |
| fail_reason | None |
| has_dependent_
| id | c33140e9-
| is_incremental | False |
| name | test_backup |
| object_count | 0 |
| size | 7 |
| snapshot_id | None |
| status | creating |
| updated_at | 2020-11-
| volume_id | 61061b93-
The backup status will always in creating. here is Child process:
[root@node-8 ~]# ps -ef | grep rbd | grep diff
42407 25499 53947 0 Nov20 ? 00:03:17 rbd export-diff --id cinder --conf /tmp/tmpCg1B5n --pool volumes <email address hidden> -
42407 25515 53947 0 Nov20 ? 00:02:20 rbd import-diff --id cinderbackup --conf /etc/ceph/ceph.conf --pool backups - backups/
we check the child process , here is info:
[root@node-8 ~]# strace -p 25499
strace: Process 25499 attached
write(2, "Exporting image", 15
[root@node-8 ~]# ll /proc/25499/fd/2
l-wx------ 1 42407 64 Dec 4 14:18 /proc/25499/fd/2 -> pipe:[3241294247]
[root@node-8 ~]#
[root@node-8 ~]# cat /proc/25499/fd/2
2020-11-20 13:46:33.181939 7f42d250bd40 -1 Errors while parsing config file!
2020-11-20 13:46:33.181945 7f42d250bd40 -1 read_conf: ignoring line 3 because it doesn't end with a newline! Please end the config file witha newline.
2020-11-20 13:46:33.183935 7f42d250bd40 -1 Errors while parsing config file!
2020-11-20 13:46:33.183940 7f42d250bd40 -1 read_conf: ignoring line 3 because it doesn't end with a newline! Please end the config file witha newline.
Exporting image: 100% complete...done.
[root@node-8 ~]#
when we read from "/proc/25499/fd/2", exporting image will go on ，and child process will close in the end.
#############
The main code：
#############
def _piped_
"""Pipe output of cmd1 into cmd2."""
try:
p1 = subprocess.
except OSError as e:
raise
# NOTE(dosaboy): ensure that the pipe is blocking. This is to work
# around the case where evenlet.
# use a non-blocking pipe.
flags = fcntl.fcntl(
try:
p2 = subprocess.
except OSError as e:
raise
stdout, stderr = p2.communicate()
return p2.returncode, stderr
we do this test:
#######
test1:
######
import subprocess
import fcntl
import os
import eventlet
eventlet.
def piped_execute(cmd1, cmd2):
"""Pipe output of cmd1 into cmd2."""
try:
p1 = subprocess.
except OSError as e:
raise
# NOTE(dosaboy): ensure that the pipe is blocking. This is to work
# around the case where evenlet.
# use a non-blocking pipe.
flags = fcntl.fcntl(
fcntl.
try:
p2 = subprocess.
except OSError as e:
raise
p1.
p1.
stdout, stderr = p2.communicate()
return p2.returncode, stderr
if __name__ == '__main__':
cmd1 = ['rbd', 'export-diff', '--id', 'cinder', '--conf', '/tmp/tmpjLsT63', '--pool', 'volumes', '<email address hidden>', '-']
cmd2 = ['rbd', 'import-diff', '--id', 'cinderbackup', '--conf', '/etc/ceph/
piped_
########
test2:
########
import subprocess
import fcntl
import os
import eventlet
eventlet.
def piped_execute(cmd1, cmd2):
"""Pipe output of cmd1 into cmd2."""
try:
p1 = subprocess.
except OSError as e:
raise
# NOTE(dosaboy): ensure that the pipe is blocking. This is to work
# around the case where evenlet.
# use a non-blocking pipe.
flags = fcntl.fcntl(
fcntl.
try:
p2 = subprocess.
except OSError as e:
raise
p1.
stdout, stderr = p2.communicate()
return p2.returncode, stderr
if __name__ == '__main__':
cmd1 = ['rbd', 'export-diff', '--id', 'cinder', '--conf', '/tmp/tmpjLsT63', '--pool', 'volumes', '<email address hidden>', '-']
cmd2 = ['rbd', 'import-diff', '--id', 'cinderbackup', '--conf', '/etc/ceph/
piped_
in test1 we add p1.stderr.close() after p1.stdout.close()
in test2 we redirect p1.stderr to a file
The test results show that there is no problem.
so when use pipe like this:
p1 = subprocess.
p2 = subprocess.
data flow is：
p1 stdout ---> parent process ---> p2 stdin
but p1 stderr no place to read, If the data size exceeds the pipe size，child process p1 will block.
#######
test3:
######
import subprocess
import eventlet
eventlet.
def piped_execute(
cmd = 'dd if=/dev/urandom bs=1 count=%d 2>/dev/null' % size
try:
p = subprocess.
except OSError as e:
print("Pipe failed - %s " % e)
raise
p.wait()
if __name__ == '__main__':
piped_
the result is： data size exceeds 64k process is block.
so in cinder backup, use subprocess.Popen make a child process to handle “rbd export-diff”, Should stderr processing be optimized?
| Changed in cinder: | |
| status: | New → Incomplete |
| buliangjun (buliangjun) wrote | #2 |
| buliangjun (buliangjun) wrote | #3 |
| Changed in cinder: | |
| importance: | Undecided → Low |
| assignee: | nobody → Sofia Enriquez (lsofia-enriquez) |
| Changed in cinder: | |
| assignee: | Sofia Enriquez (lsofia-enriquez) → nobody |
Hi,
Can you please provide some additional information:
- Cinder version in use
- Cinder management driver in-use
- Does this behaviour persist when all commands are run outside of custom tests and in standalone mode with just CLI commands?
- Have you tried adding a new line to /etc/ceph/ceph.conf as indicated by the error message in the first section of your bug report?
2020-11-20 13:46:33.183935 7f42d250bd40 -1 Errors while parsing config file!
2020-11-20 13:46:33.183940 7f42d250bd40 -1 read_conf: ignoring line 3 because it doesn't end with a newline! Please end the config file witha newline.