deployment can fail because : /var/log/containers/nova/nova-manage.log is owned by root:root

Bug #1820590 reported by Martin Schuppert
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Martin Schuppert

Bug Description

All nova-manage command via tht are run as nova user, which is correct. If for some reason a manual nova-manage command was triggered on a controller as root user where no previous nova-manage command was run results in creating the nova-manage.log with root:root as owner. As a result subsequent overcloud deploy runs can fail like:

        "Error running ['docker', 'run', '--name', 'nova_api_ensure_default_cell', '--label', 'config_id=tripleo_step3', '--label', 'container_name=nova_api_ensure_default_cell', '--label', 'managed_by=paunch', '-
-label', 'config_data={\"start_order\": 2, \"image\": \"192.168.24.1:8787/rhosp14/openstack-nova-api:2019-02-26.1\", \"environment\": [\"TRIPLEO_CONFIG_HASH=fb9151fb3fb692b17545c33eff5f974d\"], \"command\": \"/usr
/bin/bootstrap_host_exec nova_api /nova_api_ensure_default_cell.sh\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca
-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/
etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/log/containers/httpd/nova-api:/var/log/httpd\", \"/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro\", \"/var/lib/docker-config-scripts/nova_api_ensure_default_cell.sh:/nova_api_ensure_default_cell.sh:ro\"], \"net\": \"host\", \"detach\": false}', '--env=TRIPLEO_CONFIG_HASH=fb9151fb3fb692b17545c33eff5f974d', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/log/containers/httpd/nova-api:/var/log/httpd', '--volume=/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro', '--volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro', '--volume=/var/lib/docker-config-scripts/nova_api_ensure_default_cell.sh:/nova_api_ensure_default_cell.sh:ro', '192.168.24.1:8787/rhosp14/openstack-nova-api:2019-02-26.1', '/usr/bin/bootstrap_host_exec', 'nova_api', '/nova_api_ensure_default_cell.sh']. [1]",
        "stdout: (cellv2) Updating default cell_v2 cell bad87fae-4511-4aa2-9a6e-1460184f2192",
        "stderr: Traceback (most recent call last):",
        " File \"/usr/bin/nova-manage\", line 10, in <module>",
        " sys.exit(main())",
        " File \"/usr/lib/python2.7/site-packages/nova/cmd/manage.py\", line 2323, in main",
        " logging.setup(CONF, \"nova\")",
        " File \"/usr/lib/python2.7/site-packages/oslo_log/log.py\", line 264, in setup",
        " _setup_logging_from_conf(conf, product_name, version)",
        " File \"/usr/lib/python2.7/site-packages/oslo_log/log.py\", line 353, in _setup_logging_from_conf",
        " filelog = file_handler(logpath)",
        " File \"/usr/lib64/python2.7/logging/handlers.py\", line 392, in __init__",
        " logging.FileHandler.__init__(self, filename, mode, encoding, delay)",
        " File \"/usr/lib64/python2.7/logging/__init__.py\", line 902, in __init__",
        " StreamHandler.__init__(self, self._open())",
        " File \"/usr/lib64/python2.7/logging/__init__.py\", line 925, in _open",
        " stream = open(self.baseFilename, self.mode)",
        "IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'",
        "stdout: 50613161c2722e144158895957891d58584d4fc69331f776a96b25f8a79d3852",
        "stdout: 4394e139dabf8576b002cf2b0f931630a06d88b8175668109135f36586850e1a"

Changed in tripleo:
assignee: nobody → Martin Schuppert (mschuppert)
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → stein-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/643936

Changed in tripleo:
status: Triaged → In Progress
tags: added: queens-backport-potential rocky-backport-potential
tags: added: idempotency
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/643936
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c62247fa8b9ed0ebbe01b8751d2ca45d66220084
Submitter: Zuul
Branch: master

commit c62247fa8b9ed0ebbe01b8751d2ca45d66220084
Author: Martin Schuppert <email address hidden>
Date: Mon Mar 18 10:32:56 2019 +0100

    Run chown for nova log files on every run to fix wrong permissions

    If nova-manage command was triggered on a host for the first time as root
    (usually manual runs) the nova-manage.log gets created as root user. On
    overcloud deploy runs the nova-manage command is run as nova user. In such
    situation the overcloud deploy fails as the nova user can not write to the
    nova-manage.log. With this change we run the chown of the logs files on
    every overcloud deploy to fix the nova-manage.log file permissions.

    Closes-Bug: #1820590

    Change-Id: Iaa8db09712da6c0d9553fab39d7d5b50fa7cf287

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/644547

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/644548

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/644547
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=85885cbcfa9e83dc22cadc7785479aa4fecddd44
Submitter: Zuul
Branch: stable/rocky

commit 85885cbcfa9e83dc22cadc7785479aa4fecddd44
Author: Martin Schuppert <email address hidden>
Date: Mon Mar 18 10:32:56 2019 +0100

    Run chown for nova log files on every run to fix wrong permissions

    If nova-manage command was triggered on a host for the first time as root
    (usually manual runs) the nova-manage.log gets created as root user. On
    overcloud deploy runs the nova-manage command is run as nova user. In such
    situation the overcloud deploy fails as the nova user can not write to the
    nova-manage.log. With this change we run the chown of the logs files on
    every overcloud deploy to fix the nova-manage.log file permissions.

    Closes-Bug: #1820590

    Change-Id: Iaa8db09712da6c0d9553fab39d7d5b50fa7cf287
    (cherry picked from commit c62247fa8b9ed0ebbe01b8751d2ca45d66220084)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/644548
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f036df558a3d200c74154e5bcba5e3b9a477f1a5
Submitter: Zuul
Branch: stable/queens

commit f036df558a3d200c74154e5bcba5e3b9a477f1a5
Author: Martin Schuppert <email address hidden>
Date: Mon Mar 18 10:32:56 2019 +0100

    Run chown for nova log files on every run to fix wrong permissions

    If nova-manage command was triggered on a host for the first time as root
    (usually manual runs) the nova-manage.log gets created as root user. On
    overcloud deploy runs the nova-manage command is run as nova user. In such
    situation the overcloud deploy fails as the nova user can not write to the
    nova-manage.log. With this change we run the chown of the logs files on
    every overcloud deploy to fix the nova-manage.log file permissions.

    Closes-Bug: #1820590

    Change-Id: Iaa8db09712da6c0d9553fab39d7d5b50fa7cf287
    (cherry picked from commit c62247fa8b9ed0ebbe01b8751d2ca45d66220084)
    (cherry picked from commit 85885cbcfa9e83dc22cadc7785479aa4fecddd44)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.5.0

This issue was fixed in the openstack/tripleo-heat-templates 10.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/655909
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=08015d6f9166e9e62bb7ff3f38c5d7e08ba646e5
Submitter: Zuul
Branch: master

commit 08015d6f9166e9e62bb7ff3f38c5d7e08ba646e5
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 26 15:35:33 2019 +0200

    Run nova-manage as root to prevent wrong nova-manage.log permissions

    To verify if the default cell is created we run a nova-manage command.
    Right now this is triggered as root and not as nova like the otheres.
    In case it is the first time on the node it creates the nova-manage.log
    with root user and follow up call as nova user will fail.

    Related-bug: 1820590

    Change-Id: I56a961615c0afa748a42c25d4614637275b7c33b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/656038

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/656075

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/656076

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/656038
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4a6ba2bbaa5a813a4dad61b7c20c04ee77381395
Submitter: Zuul
Branch: stable/stein

commit 4a6ba2bbaa5a813a4dad61b7c20c04ee77381395
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 26 15:35:33 2019 +0200

    Run nova-manage as root to prevent wrong nova-manage.log permissions

    To verify if the default cell is created we run a nova-manage command.
    Right now this is triggered as root and not as nova like the otheres.
    In case it is the first time on the node it creates the nova-manage.log
    with root user and follow up call as nova user will fail.

    Related-bug: 1820590

    Change-Id: I56a961615c0afa748a42c25d4614637275b7c33b
    (cherry picked from commit 08015d6f9166e9e62bb7ff3f38c5d7e08ba646e5)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/656076
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cce6c7e34b327c92436b3c7116403f1651e5e10d
Submitter: Zuul
Branch: stable/queens

commit cce6c7e34b327c92436b3c7116403f1651e5e10d
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 26 15:35:33 2019 +0200

    Run nova-manage as root to prevent wrong nova-manage.log permissions

    To verify if the default cell is created we run a nova-manage command.
    Right now this is triggered as root and not as nova like the otheres.
    In case it is the first time on the node it creates the nova-manage.log
    with root user and follow up call as nova user will fail.

    Related-bug: 1820590

    Change-Id: I56a961615c0afa748a42c25d4614637275b7c33b
    (cherry picked from commit 08015d6f9166e9e62bb7ff3f38c5d7e08ba646e5)
    (cherry picked from commit 4a6ba2bbaa5a813a4dad61b7c20c04ee77381395)
    (cherry picked from commit a968838990d485479bbe4ff15c80dafa51175082)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/656075
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a968838990d485479bbe4ff15c80dafa51175082
Submitter: Zuul
Branch: stable/rocky

commit a968838990d485479bbe4ff15c80dafa51175082
Author: Martin Schuppert <email address hidden>
Date: Fri Apr 26 15:35:33 2019 +0200

    Run nova-manage as root to prevent wrong nova-manage.log permissions

    To verify if the default cell is created we run a nova-manage command.
    Right now this is triggered as root and not as nova like the otheres.
    In case it is the first time on the node it creates the nova-manage.log
    with root user and follow up call as nova user will fail.

    Related-bug: 1820590

    Change-Id: I56a961615c0afa748a42c25d4614637275b7c33b
    (cherry picked from commit 08015d6f9166e9e62bb7ff3f38c5d7e08ba646e5)
    (cherry picked from commit 4a6ba2bbaa5a813a4dad61b7c20c04ee77381395)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.4.0

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.4.0

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.