/openstack directory creation race

Bug #1407121 reported by Hugh Saunders
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openstack-ansible
Fix Released
Medium
Hugh Saunders
Juno
Fix Released
Medium
Jesse Pretorius
Trunk
Fix Released
Medium
Hugh Saunders

Bug Description

"TASK: [container_setup | Create Required local monitoring directories]" is targeted at all containers but delegated to each container's host. It creates /openstack/{backup,log}/{{container_name}} which requires /openstack to be created if it doesn't exist. If a host has multiple containers, this task may be delegated to the host multiple times in parallel, this creates a race to create /openstack. Example below:

=================================================================================

**15:17:37** failed: [node17_nova_conductor_container-faa11b22 -> node17] => (item=/openstack/backup/node17_nova_conductor_container-faa11b22) => {"failed": true, "item": "/openstack/backup/node17_nova_conductor_container-faa11b22", "parsed": false}
**15:17:37** OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
**15:17:37** debug1: Reading configuration data /etc/ssh/ssh_config
**15:17:37** debug1: /etc/ssh/ssh_config line 19: Applying options for *
**15:17:37** debug1: auto-mux: Trying existing master
**15:17:37** debug2: fd 3 setting O_NONBLOCK
**15:17:37** debug2: mux_client_hello_exchange: master version 4
**15:17:37** debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
**15:17:37** debug3: mux_client_request_session: entering
**15:17:37** debug3: mux_client_request_alive: entering
**15:17:37** debug3: mux_client_request_alive: done pid = 18585
**15:17:37** debug3: mux_client_request_session: session request sent
**15:17:37** debug1: mux_client_request_session: master session id: 4
**15:17:37** Traceback (most recent call last):
**15:17:37** File "<stdin>", line 1928, in <module>
**15:17:37** File "<stdin>", line 238, in main
**15:17:37** OSError: [Errno 17] File exists: '/openstack'
**15:17:37** debug3: mux_client_read_packet: read header failed: Broken pipe
**15:17:37** debug2: Received exit status from master 1

**15:17:37** failed: [node17_cinder_api_container-a06914a1 -> node17] => (item=/openstack/backup/node17_cinder_api_container-a06914a1) => {"failed": true, "item": "/openstack/backup/node17_cinder_api_container-a06914a1", "parsed": false}
**15:17:37** OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
**15:17:37** debug1: Reading configuration data /etc/ssh/ssh_config
**15:17:37** debug1: /etc/ssh/ssh_config line 19: Applying options for *
**15:17:37** debug1: auto-mux: Trying existing master
**15:17:37** debug2: fd 3 setting O_NONBLOCK
**15:17:37** debug2: mux_client_hello_exchange: master version 4
**15:17:37** debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
**15:17:37** debug3: mux_client_request_session: entering
**15:17:37** debug3: mux_client_request_alive: entering
**15:17:37** debug3: mux_client_request_alive: done pid = 18585
**15:17:37** debug3: mux_client_request_session: session request sent
**15:17:37** debug1: mux_client_request_session: master session id: 6
**15:17:37** Traceback (most recent call last):
**15:17:37** File "<stdin>", line 1928, in <module>
**15:17:37** File "<stdin>", line 238, in main
**15:17:37** OSError: [Errno 17] File exists: '/openstack'
**15:17:37** debug3: mux_client_read_packet: read header failed: Broken pipe
**15:17:37** debug2: Received exit status from master 1
**15:17:37**

=================================================================================

Changed in openstack-ansible:
assignee: nobody → Hugh Saunders (hughsaunders)
status: New → In Progress
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (master)

Reviewed: https://review.openstack.org/144776
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=ac7c30c6ad645353c3af84d89d061ea187afa96b
Submitter: Jenkins
Branch: master

commit ac7c30c6ad645353c3af84d89d061ea187afa96b
Author: Hugh Saunders <email address hidden>
Date: Fri Jan 2 16:52:13 2015 +0000

    Restructure creation, perms for bind mounted dirs

    Creating /openstack in a task targetted at the host, prevents a race
    when it is created in a task targetted at containers but delegated to
    the host. Same problem also exists with common subdirs.

    /var/log/nova has a similar problem, for metal 'containers' it should be
    a symlink to /openstack/log/container_name, but for lxc containers
    /var/log/nova should be an empty dir ready for logs.

    Change-Id: I6366535a76562e9f95cc1e0352daf76d9ae39e3b
    Closes-Bug: #1407121

Changed in openstack-ansible:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-ansible-deployment (juno)

Fix proposed to branch: juno
Review: https://review.openstack.org/150360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (juno)

Reviewed: https://review.openstack.org/150360
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=1948c15a11990dab609f7886990ece086020ad23
Submitter: Jenkins
Branch: juno

commit 1948c15a11990dab609f7886990ece086020ad23
Author: Hugh Saunders <email address hidden>
Date: Fri Jan 2 16:52:13 2015 +0000

    Restructure creation, perms for bind mounted dirs

    Creating /openstack in a task targetted at the host, prevents a race
    when it is created in a task targetted at containers but delegated to
    the host. Same problem also exists with common subdirs.

    /var/log/nova has a similar problem, for metal 'containers' it should be
    a symlink to /openstack/log/container_name, but for lxc containers
    /var/log/nova should be an empty dir ready for logs.

    Change-Id: I6366535a76562e9f95cc1e0352daf76d9ae39e3b
    Closes-Bug: #1407121
    (cherry picked from commit ac7c30c6ad645353c3af84d89d061ea187afa96b)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.