Bug #1820577 “overcloud deployment fails at Run container-puppet...” : Bugs : tripleo

Revision history for this message

Juan Antonio Osorio Robles (juan-osorio-robles) wrote on 2019-03-18:

#1

How was this deployed?

Revision history for this message

Michele Baldessari (michele) wrote on 2019-03-18:

#2

Download full text (6.8 KiB)

So this likely works on docker because docker will simply create a non-existing bind-mount (like /etc/pki/tls/private/haproxy in this case).

The command that failed for us is the following (normal deploy via infrared but with the additional 'overcloud-ssl true' param):
openstack overcloud deploy \
  --timeout 100 \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --libvirt-type kvm \
  --stack overcloud \
  -r /home/stack/composable_roles/roles/roles_data.yaml \
  -e /home/stack/composable_roles/roles/nodes.yaml \
-e /home/stack/composable_roles/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/composable_roles/network/network-environment.yaml \
-e /home/stack/composable_roles/enable-tls.yaml \
-e /home/stack/composable_roles/inject-trust-anchor.yaml \
-e /home/stack/composable_roles/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e ~/fencing.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \
-e /home/stack/composable_roles/debug.yaml \
-e /home/stack/composable_roles/config_heat.yaml \
--log-file overcloud_deployment_67.log

Here are the custom ones that might be relevant here:
### /home/stack/composable_roles/enable-tls.yaml ###
parameter_defaults:
  # Set CSRF_COOKIE_SECURE / SESSION_COOKIE_SECURE in Horizon
  # Type: boolean
  HorizonSecureCookies: True
  # The content of the SSL certificate (without Key) in PEM format.
  # Type: string
  SSLCertificate: |
    -----BEGIN CERTIFICATE-----
    MIIDaDCCAlCgAwIBAgIBATANBgkqhkiG9w0BAQsFADBiMQswCQYDVQQGEwJVUzEL
    MAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIQXQx
    CzAJBgNVBAsMAlFFMRUwEwYDVQQDDAwxOTIuMTY4LjI0LjIwHhcNMTkwMzE4MTcy
    MzEwWhcNMjAwMzE3MTcyMzEwWjBgMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTkMx
    EDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIQXQxCzAJBgNVBAsMAlFF
    MRMwEQYDVQQDDAoxMC4wLjAuMTAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
    CgKCAQEA3eues7Hlhtzo7IO8dohSeQzQx9K9gu/UNBIypq5cDKng99td8OGL//dy
    ijAiOw3LkE+z9QjoEXRGONmu+m1f429G8JDtW47WfzAEZ5Fr1GojRUs/ZkmBwI6g
    TfxZKqX8XX4Ws3PDb70giwcANi/9B7kN6ltbL/U6seyEOSmFfEF1juS/otARqQe4
    lZioxJKwaQn0pSmPHMIwyQj0zOctiXY9GqEf3dQgFyfi7InC5wnU/RUAcmOUJpoy
    GalyXf6RI8oaIjPdCt9Ox+ZEogyAgdPgpGjKyGLlj4uG+sqj2/Xap0VnmTWYioQG
    wkD9VRrKYq6k80JuGmyEZ2+VHSXEQQIDAQABoyswKTAJBgNVHRMEAjAAMAsGA1Ud
    DwQEAwIF4DAPBgNVHREECDAGhwQKAABlMA0GCSqGSIb3DQEBCwUAA4IBAQBWcQkj
    3I2/mbOnYZt7nUTNGqPZctTLqFfN2UvMjHla2ZFBb3BfW2LUkj0Z91kmztRCNRbh
    Z8dKq583CCbLnWpjbKpWJfP0nWD/zvrB76tzJXZXQdM81t92C2AOelhaivfuBw9b
    ZoBiN/CfHOroZgN7oOZzmhjxm+y5/mr3sBxbuMsdk7UdCgzNmFv06hnG1V3HnacP
    uApJ55FDelvNDFrypbCdLm5aw00CDD+CFZy9I10nM3vFF1MOJXU/bvTa1Dc94uY6
    nsxYiUaH+E7X6ONxyP0Ttz2XvQe5n9B/sHCpSmySlV7Xnwlu4QhFFh9qT0QxGeq8
    wlW8xDcD+88LP815
    -----END CERTIFICATE-----
  # The content of an SSL intermediate CA certificate in PEM format.
  # Type: string
  SSLIntermediateCertificate: ''
  # The content of the SSL Key in PEM format.
  # Type: string
  SSLKey: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEowIBAAKCAQEA3eues7Hlhtzo7IO8dohSeQzQx9K9gu/UNBIypq5cDKng99...

So this likely works on docker because docker will simply create a non-existing bind-mount (like /etc/pki/tls/private/haproxy in this case).

The command that failed for us is the following (normal deploy via infrared but with the additional 'overcloud-ssl true' param):
openstack overcloud deploy \
  --timeout 100 \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --libvirt-type kvm \
  --stack overcloud \
  -r /home/stack/composable_roles/roles/roles_data.yaml \
  -e /home/stack/composable_roles/roles/nodes.yaml \
-e /home/stack/composable_roles/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/composable_roles/network/network-environment.yaml \
-e /home/stack/composable_roles/enable-tls.yaml \
-e /home/stack/composable_roles/inject-trust-anchor.yaml \
-e /home/stack/composable_roles/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e ~/fencing.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \
-e /home/stack/composable_roles/debug.yaml \
-e /home/stack/composable_roles/config_heat.yaml \
--log-file overcloud_deployment_67.log

Here are the custom ones that might be relevant here:
### /home/stack/composable_roles/enable-tls.yaml ###
parameter_defaults:
  # Set CSRF_COOKIE_SECURE / SESSION_COOKIE_SECURE in Horizon
  # Type: boolean
  HorizonSecureCookies: True
  # The content of the SSL certificate (without Key) in PEM format.
  # Type: string
  SSLCertificate: |
    -----BEGIN CERTIFICATE-----
    MIIDaDCCAlCgAwIBAgIBATANBgkqhkiG9w0BAQsFADBiMQswCQYDVQQGEwJVUzEL
    MAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIQXQx
    CzAJBgNVBAsMAlFFMRUwEwYDVQQDDAwxOTIuMTY4LjI0LjIwHhcNMTkwMzE4MTcy
    MzEwWhcNMjAwMzE3MTcyMzEwWjBgMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTkMx
    EDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIQXQxCzAJBgNVBAsMAlFF
    MRMwEQYDVQQDDAoxMC4wLjAuMTAxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
    CgKCAQEA3eues7Hlhtzo7IO8dohSeQzQx9K9gu/UNBIypq5cDKng99td8OGL//dy
    ijAiOw3LkE+z9QjoEXRGONmu+m1f429G8JDtW47WfzAEZ5Fr1GojRUs/ZkmBwI6g
    TfxZKqX8XX4Ws3PDb70giwcANi/9B7kN6ltbL/U6seyEOSmFfEF1juS/otARqQe4
    lZioxJKwaQn0pSmPHMIwyQj0zOctiXY9GqEf3dQgFyfi7InC5wnU/RUAcmOUJpoy
    GalyXf6RI8oaIjPdCt9Ox+ZEogyAgdPgpGjKyGLlj4uG+sqj2/Xap0VnmTWYioQG
    wkD9VRrKYq6k80JuGmyEZ2+VHSXEQQIDAQABoyswKTAJBgNVHRMEAjAAMAsGA1Ud
    DwQEAwIF4DAPBgNVHREECDAGhwQKAABlMA0GCSqGSIb3DQEBCwUAA4IBAQBWcQkj
    3I2/mbOnYZt7nUTNGqPZctTLqFfN2UvMjHla2ZFBb3BfW2LUkj0Z91kmztRCNRbh
    Z8dKq583CCbLnWpjbKpWJfP0nWD/zvrB76tzJXZXQdM81t92C2AOelhaivfuBw9b
    ZoBiN/CfHOroZgN7oOZzmhjxm+y5/mr3sBxbuMsdk7UdCgzNmFv06hnG1V3HnacP
    uApJ55FDelvNDFrypbCdLm5aw00CDD+CFZy9I10nM3vFF1MOJXU/bvTa1Dc94uY6
    nsxYiUaH+E7X6ONxyP0Ttz2XvQe5n9B/sHCpSmySlV7Xnwlu4QhFFh9qT0QxGeq8
    wlW8xDcD+88LP815
    -----END CERTIFICATE-----
  # The content of an SSL intermediate CA certificate in PEM format.
  # Type: string
  SSLIntermediateCertificate: ''
  # The content of the SSL Key in PEM format.
  # Type: string
  SSLKey: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEowIBAAKCAQEA3eues7Hlhtzo7IO8dohSeQzQx9K9gu/UNBIypq5cDKng99td
    8OGL//dyijAiOw3LkE+z9QjoEXRGONmu+m1f429G8JDtW47WfzAEZ5Fr1GojRUs/
    ZkmBwI6gTfxZKqX8XX4Ws3PDb70giwcANi/9B7kN6ltbL/U6seyEOSmFfEF1juS/
    otARqQe4lZioxJKwaQn0pSmPHMIwyQj0zOctiXY9GqEf3dQgFyfi7InC5wnU/RUA
    cmOUJpoyGalyXf6RI8oaIjPdCt9Ox+ZEogyAgdPgpGjKyGLlj4uG+sqj2/Xap0Vn
    mTWYioQGwkD9VRrKYq6k80JuGmyEZ2+VHSXEQQIDAQABAoIBAGtTO3JuU7IQfnl8
    EJZGwZQXdZVePxbnA/qW3nYsFwps0gcsyVbozbtiIbvhIXzr5AoL8d1MjGd8k0WF
    SZGFef1VYLqRbKl+ABCM2WR5OOwG1L37cVL3s8+7ap2ssDbnBZD1nOb20orkratv
    HFQYi0fD4I351rTv5Y4M87ltckgvvaQhM4I6xuSq+fU3CMXex8SodJaRp8ryMXtg
    TCc3F7HvZHkWGJAAGSyJ7BGIK59Re4bVVGoLtQAbWHQlCvK4qK1FuD/anpMrbJCJ
    z1h82bq7afPRk6S7RrGq+XgEYXQQvzGb5gNynk0i2EziMN8SLWKJI8IJHw9ENDGN
    zlBtrO0CgYEA8r1Vz/6sQa+148ljhUBgjDH1YxsMRszaHyI8EHKRMirj5S2WnSuh
    5GFZ35k8zoV6eEpkDpo9pykIbf5xWF4x/W0sayHB8owRg9hR/JCEu2dKS0tXk2Kz
    Yd2jw9gEkwTgjhIRj/y0N+DrIgyQdH7efCjVCdk/HX0t6AwNKnev/pMCgYEA6gsh
    4jKVUTfcXNQltjgruzPnfziAqKQ21Q2tvi1J3f5C0G5Ahi8/L/P49iPc0AGpQAWs
    3zUrd1e4zj8/HqU4C8+BigdcJSLMiOxH4dwj0MUrrtQ0ZWCCR6auVXNRjz/bMPTn
    6/A1y5JJL6qUa6cQyoqI7CkqlZze84q5QNiWYlsCgYAQaWq24GInIskqeIBJDxw3
    /ly38ak31DvfJQJonkZg3POBmD55q+yLM2XPL14kHHQ2U9lF/3mxpp2SSkTBk8TP
    fKUnLSYezweUIXkRmXfP7+DfDF4EcgTs1f5vjFhq1EaWdHRJhu1sZcGgELdnmPhT
    7rLQsqaIyODksoPlXihBDwKBgBaiUPqRLcOXBWas1aDTudb25BJ3ommsx+i+0+iq
    dJRVfJyvrOnNM+0tSQx012v+XSHl6pwDhlqaTzMTBsWt8Mejn8F7iLvGq8UqJKGB
    Kq2MA/d6aj7LoI3RXtbAukiCQm5voOY+atVvJkjH1Ga813DfCPLd4FJrw1wGNHhC
    +fg1AoGBALjJjxtI8hRU2jtHHqv/TStT8zVeXQCkQEXq1HrbKG37RRk2qpnQWuSC
    Vb8ehWIvJu44htJ78W9o7z3GcGccnd6ihWs87lOOs4SJbj1sg2tO3hGcfqnuA0mU
    bSepd0aZd5IXVPRrxG0IT2GC4v2GwPe8qryeaIUqEx1jxhKohmec
    -----END RSA PRIVATE KEY-----
  # Static parameters - these are values that must be
  # included in the environment but should not be changed.
  # The filepath of the certificate as it will be stored in the controller.
  # Type: string
  DeployedSSLCertificatePath: /etc/pki/tls/private/overcloud_endpoint.pem
  # End static parameters
### /home/stack/composable_roles/inject-trust-anchor.yaml ###
parameter_defaults:
  # The content of a CA's SSL certificate file in PEM format. This is evaluated on the client side.
  # Mandatory. This parameter must be set by the user.
  # Type: string
  SSLRootCertificate: |
    -----BEGIN CERTIFICATE-----
    MIIDpTCCAo2gAwIBAgIUItY3H2WSwy+lZDPBTDSuQnrGes8wDQYJKoZIhvcNAQEL
    BQAwYjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo
    MRAwDgYDVQQKDAdSZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2
    OC4yNC4yMB4XDTE5MDMxODE2Mzc1M1oXDTIwMDMxNzE2Mzc1M1owYjELMAkGA1UE
    BhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdS
    ZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2OC4yNC4yMIIBIjAN
    BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxl21+ya9LcASoCL18ntty751ezFV
    olT4RZeV9KA+jdMKgd8ErIFaZar+i/W47ZZZWBVaFQo3sdLad/gQEAGbrTKYaWir
    ZlJIAmLJ/dQaPo/IPR1MItQtXMKAyQsPnTHUkihiXrLNgbtpSgXDHXgvpq+DeaUq
    b7EzJ/S5A3gTEZtwCHBjQ5svhEooMm5ZbWqmqNuJcC3hEYSy7eH72qM/KYXKB5pD
    Ofulx6CNQKy886gS99aOb7P8gnVL5j9krz5mG6UlN1Y5PAAT3RiZpNSG8bPjwGsc
    QHyoG7reIK+/K3W2Mit00qJSDKOugatqpCOYkINcPAIq6KIGPdI7Iq7BSQIDAQAB
    o1MwUTAdBgNVHQ4EFgQUVBRXM9WAqOYhW42KgRD+uY0x+L8wHwYDVR0jBBgwFoAU
    VBRXM9WAqOYhW42KgRD+uY0x+L8wDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
    AQsFAAOCAQEAvj1HwjZhNOTDu+X9Ou3DAz1FAo14xBytP6uBDjoJGsP+mq5CkBwN
    32SGBiPqlwQSj/uvdA7cc/gz5i//uYWl9TU85Lu0sMYKb36UQWRo/59rsB0lu7sf
    MWJIaUSqkJerSX9Mw1xJ7HTSY7j1Ur4jxs4JZgHIPdcPJN31TwodTYF/mFHxE8ro
    Ty21BQT9B7K9XLJAq/cHXnn7uh4q4DxFSe7glUJUCz3LXpHZoR1rSzAMYbLKrolD
    P8bwJu94UG3pera4EPYLLx+v+JoLTlsPl9fO5r6mvEcFIIs0PR5FFVXt2X7rRtm9
    ALYhwaOhOkCXxjDhyV0nA2HnCJSUrkndww==
    -----END CERTIFICATE-----
resource_registry:
  OS::TripleO::NodeTLSCAData: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/tls/ca-inject.yaml

Emilien Macchi (emilienm) on 2019-03-18

Changed in tripleo:
status:	New → Triaged
importance:	Undecided → High
milestone:	none → stein-rc1

Revision history for this message

Juan Antonio Osorio Robles (juan-osorio-robles) wrote on 2019-03-19:

#3

This is supposed to work on podman too, since the directory is created in puppet https://github.com/openstack/puppet-tripleo/blob/master/manifests/certmonger/haproxy_dirs.pp#L37 https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-public-tls-certmonger.yaml#L53

Isn't step 1 too early to try to start the haproxy container? Thought it was supposed to be in step 2

Revision history for this message

Michele Baldessari (michele) wrote on 2019-03-19:

#4

Download full text (7.3 KiB)

Ah thanks for pointing out where the folders should get created. So this failure happens with the 'container-puppet-haproxy' container which happens on step1:
2019-03-18 17:47:57,152 p=486 u=mistral | TASK [Run container-puppet tasks (generate config) during step 1] **************
2019-03-18 17:47:57,152 p=486 u=mistral | Monday 18 March 2019 17:47:57 +0000 (0:00:00.929) 0:08:23.294 **********
2019-03-18 17:50:35,097 p=486 u=mistral | ok: [compute-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:11,404 p=486 u=mistral | ok: [controller-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:27,130 p=486 u=mistral | ok: [controller-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:30,301 p=486 u=mistral | ok: [controller-2] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:30,352 p=486 u=mistral | TASK [Debug output for task: Run container-puppet tasks (generate config) during step 1] ***
2019-03-18 17:52:30,352 p=486 u=mistral | Monday 18 March 2019 17:52:30 +0000 (0:04:33.199) 0:12:56.493 **********
2019-03-18 17:52:30,476 p=486 u=mistral | fatal: [controller-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "2019-03-18 17:47:57,755 INFO: 27003 -- Running container-puppet",
        "2019-03-18 17:47:57,756 INFO: 27003 -- Service compilation completed.",
        "2019-03-18 17:47:57,756 INFO: 27003 -- Starting multiprocess configuration steps. Using 6 processes.",
        "2019-03-18 17:47:57,769 INFO: 27004 -- Starting configuration of aodh using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-aodh-api:latest",
        "2019-03-18 17:47:57,770 INFO: 27006 -- Starting configuration of glance_api using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:latest",
        "2019-03-18 17:47:57,769 INFO: 27005 -- Starting configuration of cinder using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cinder-api:latest",
        "2019-03-18 17:47:57,771 INFO: 27007 -- Starting configuration of haproxy using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-haproxy:latest",
        "2019-03-18 17:47:57,772 INFO: 27008 -- Starting configuration of heat_api_cfn using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-heat-api-cfn:latest",
        "2019-03-18 17:47:57,772 INFO: 27009 -- Starting configuration of horizon using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-horizon:latest",
        "2019-03-18 17:47:58,108 INFO: 27008 -- Removing container: container-puppet-heat_api_cfn",
        "2019-03-18 17:47:58,169 INFO: 27005 -- Removing container: container-puppet-cinder",
...

Ah thanks for pointing out where the folders should get created. So this failure happens with the 'container-puppet-haproxy' container which happens on step1:
2019-03-18 17:47:57,152 p=486 u=mistral |  TASK [Run container-puppet tasks (generate config) during step 1] **************
2019-03-18 17:47:57,152 p=486 u=mistral |  Monday 18 March 2019  17:47:57 +0000 (0:00:00.929)       0:08:23.294 **********
2019-03-18 17:50:35,097 p=486 u=mistral |  ok: [compute-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:11,404 p=486 u=mistral |  ok: [controller-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:27,130 p=486 u=mistral |  ok: [controller-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:30,301 p=486 u=mistral |  ok: [controller-2] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2019-03-18 17:52:30,352 p=486 u=mistral |  TASK [Debug output for task: Run container-puppet tasks (generate config) during step 1] ***
2019-03-18 17:52:30,352 p=486 u=mistral |  Monday 18 March 2019  17:52:30 +0000 (0:04:33.199)       0:12:56.493 **********
2019-03-18 17:52:30,476 p=486 u=mistral |  fatal: [controller-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "2019-03-18 17:47:57,755 INFO: 27003 -- Running container-puppet",
        "2019-03-18 17:47:57,756 INFO: 27003 -- Service compilation completed.",
        "2019-03-18 17:47:57,756 INFO: 27003 -- Starting multiprocess configuration steps.  Using 6 processes.",
        "2019-03-18 17:47:57,769 INFO: 27004 -- Starting configuration of aodh using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-aodh-api:latest",
        "2019-03-18 17:47:57,770 INFO: 27006 -- Starting configuration of glance_api using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:latest",
        "2019-03-18 17:47:57,769 INFO: 27005 -- Starting configuration of cinder using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cinder-api:latest",
        "2019-03-18 17:47:57,771 INFO: 27007 -- Starting configuration of haproxy using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-haproxy:latest",
        "2019-03-18 17:47:57,772 INFO: 27008 -- Starting configuration of heat_api_cfn using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-heat-api-cfn:latest",
        "2019-03-18 17:47:57,772 INFO: 27009 -- Starting configuration of horizon using image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-horizon:latest",
        "2019-03-18 17:47:58,108 INFO: 27008 -- Removing container: container-puppet-heat_api_cfn",
        "2019-03-18 17:47:58,169 INFO: 27005 -- Removing container: container-puppet-cinder",
        "2019-03-18 17:47:58,231 INFO: 27009 -- Removing container: container-puppet-horizon",
        "2019-03-18 17:47:58,355 INFO: 27006 -- Removing container: container-puppet-glance_api",
        "2019-03-18 17:47:58,469 INFO: 27004 -- Removing container: container-puppet-aodh",
        "2019-03-18 17:47:58,877 INFO: 27007 -- Removing container: container-puppet-haproxy",
        "2019-03-18 17:47:59,039 INFO: 27008 -- Pulling image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-heat-api-cfn:latest",
        "2019-03-18 17:47:59,096 INFO: 27004 -- Pulling image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-aodh-api:latest",
        "2019-03-18 17:47:59,532 INFO: 27009 -- Pulling image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-horizon:latest",
        "2019-03-18 17:47:59,588 INFO: 27005 -- Pulling image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cinder-api:latest",
        "2019-03-18 17:47:59,836 INFO: 27007 -- Image already exists: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-haproxy:latest",
        "2019-03-18 17:47:59,966 INFO: 27006 -- Pulling image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:latest",
        "2019-03-18 17:48:03,153 WARNING: 27007 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', 'container-puppet-haproxy', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,haproxy_config', '--env', 'NAME=haproxy', '--env', 'HOSTNAME=controller-0', '--env', 'NO_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--log-driver', 'json-file', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpvtrdil17:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/dev/log:/dev/log:rw', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-haproxy.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--volume', '/etc/pki/tls/private/haproxy:/etc/pki/tls/private/haproxy:ro', '--volume', '/etc/pki/tls/certs/haproxy:/etc/pki/tls/certs/haproxy:ro', '--volume', '/etc/pki/tls/private/overcloud_endpoint.pem:/etc/pki/tls/private/overcloud_endpoint.pem:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-haproxy:latest'] run failed after error checking path \"/etc/pki/tls/private/haproxy\": stat /etc/pki/tls/private/haproxy: no such file or directory",
        " attempt(s): 1",
        "2019-03-18 17:48:03,153 WARNING: 27007 -- Retrying running container: haproxy",
        "2019-03-18 17:48:06,286 WARNING: 27007 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-haproxy'] run failed after unable to find container container-puppet-haproxy: no container with name or ID container-puppet-haproxy found: no such container",
        " attempt(s): 2",
        "2019-03-18 17:48:06,286 WARNING: 27007 -- Retrying running container: haproxy",
        "2019-03-18 17:48:09,430 WARNING: 27007 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-haproxy'] run failed after unable to find container container-puppet-haproxy: no container with name or ID container-puppet-haproxy found: no such container",
        " attempt(s): 3",
        "2019-03-18 17:48:09,430 WARNING: 27007 -- Retrying running container: haproxy",
        "2019-03-18 17:48:09,430 ERROR: 27007 -- Failed running container for haproxy",

In my host puppet manifest I do have certmonger:
[root@controller-0 tripleo-config]# grep certmonger /var/lib/tripleo-config/puppet_step_config.pp 
include ::tripleo::profile::base::certmonger_user

Revision history for this message

Michele Baldessari (michele) wrote on 2019-03-19:

#5

So just recapping for my own good (please correct any mistakes on my part):
a) The /etc/pki/tls/private/haproxy dir should be created by the following puppet profile:
include ::tripleo::profile::base::certmonger_user which includes tripleo::certmonger::haproxy and it runs at step1 on *the host* and it should run during an ansible task called 'Run puppet host configuration for step {{ step }}' (see tht/common/deploy-steps-tasks.yaml)

b) The failing container is 'container-puppet-haproxy' and it gets called during the ansible task called 'Run container-puppet tasks (generate config) during step 1'

Unless I am seeing things 'Run puppet host configuration for step {{ step }}' comes before 'Run container-puppet tasks (generate config) during step 1' in tht/common/deploy-steps-tasks.yaml so am not sure yet as to why this is not working as expected

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-19: Fix proposed to tripleo-heat-templates (master)

#6

Fix proposed to branch: master
Review: https://review.openstack.org/644504

Changed in tripleo:
assignee:	nobody → Juan Antonio Osorio Robles (juan-osorio-robles)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-19: Fix merged to tripleo-heat-templates (master)

#7

Reviewed: https://review.openstack.org/644504
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b848cef629dfbf83b638281e57361a383aa0a1f2
Submitter: Zuul
Branch: master

commit b848cef629dfbf83b638281e57361a383aa0a1f2
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Mar 19 10:15:25 2019 +0200

Only bind-mount internal TLS haproxy dirs if enabled

We were bind-mounting those directories when public TLS was enabled...
it needed to be in the internal TLS conditional.

Change-Id: I7487c0f3b495dce2f5ce3028e8516cc3c215f896
Closes-Bug: #1820577

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-18: Fix included in openstack/tripleo-heat-templates 10.5.0

#8

This issue was fixed in the openstack/tripleo-heat-templates 10.5.0 release.

tripleo

overcloud deployment fails at Run container-puppet tasks (generate config) during step 1 - /etc/pki/tls/private/haproxy: no such file or directory

Bug Description

Other bug subscribers

Remote bug watches