tripleo

openshift CI job (scenario009) fails to run overcloud-prep-images.sh

Bug #1816441 reported by Martin André on 2019-02-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Martin André	tripleo stein-3

Bug Description

Looks like "sudo podman ps | grep executor" doesn't return any results.

2019-02-18 12:35:43.588657 | primary | fatal: [undercloud]: FAILED! => {
2019-02-18 12:35:43.588788 | primary | "changed": true,
2019-02-18 12:35:43.589223 | primary | "cmd": "set -o pipefail && /home/zuul/overcloud-prep-images.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/zuul/overcloud_prep_images.log",
2019-02-18 12:35:43.589369 | primary | "delta": "0:00:33.609533",
2019-02-18 12:35:43.589528 | primary | "end": "2019-02-18 12:35:43.566037",
2019-02-18 12:35:43.589602 | primary | "rc": 125,
2019-02-18 12:35:43.589725 | primary | "start": "2019-02-18 12:35:09.956504"
2019-02-18 12:35:43.589785 | primary | }
2019-02-18 12:35:43.589830 | primary |
2019-02-18 12:35:43.589882 | primary | MSG:
2019-02-18 12:35:43.589926 | primary |
2019-02-18 12:35:43.590024 | primary | non-zero return code
2019-02-18 12:35:43.615264 | primary |
2019-02-18 12:35:43.615653 | primary | PLAY RECAP *********************************************************************
2019-02-18 12:35:43.615793 | primary | undercloud : ok=1 changed=1 unreachable=0 failed=1

http://logs.openstack.org/62/628962/6/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/ecfd391/job-output.txt.gz#_2019-02-18_12_35_43_588657

2019-02-18 12:41:43 | + container_cli=podman
2019-02-18 12:41:43 | ++ sudo podman ps
2019-02-18 12:41:43 | ++ grep executor
2019-02-18 12:41:43 | ++ awk '{print $1}'
2019-02-18 12:41:43 | + cont=
2019-02-18 12:41:43 | + sudo podman exec --user root bash -c 'easy_install pip && pip install ara'
2019-02-18 12:41:43 | unable to exec into bash: no container with name or ID bash found: no such container

http://logs.openstack.org/60/628960/7/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/0afd5e3/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-02-18_12_41_43

Tags:

Revision history for this message

Martin André (mandre) wrote on 2019-02-18:

mistral-executor fails to start:

+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 412, in main
    config = load_config()
  File "/usr/local/bin/kolla_set_configs", line 297, in load_config
    validate_config(config)
  File "/usr/local/bin/kolla_set_configs", line 238, in validate_config
    if not set(data.keys()) >= required_keys:
AttributeError: 'unicode' object has no attribute 'keys'

http://logs.openstack.org/62/628962/6/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/ecfd391/logs/undercloud/var/log/extra/podman/containers/mistral_executor/stdout.log.txt.gz

Revision history for this message

Martin André (mandre) wrote on 2019-02-18:

Possibly caused by https://review.openstack.org/#/c/634425/ where it produces a malformed /var/lib/kolla/config_files/mistral_executor.json in the event UndercloudConfigFilePath is empty.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-18: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/637591

Changed in tripleo:
status:	Triaged → In Progress

Bogdan Dobrelya (bogdando) on 2019-02-19

tags:

added: rocky-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-19: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/637591
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a08daffcc790607d7d0fdf0cbad0755f9037aa8d
Submitter: Zuul
Branch: master

commit a08daffcc790607d7d0fdf0cbad0755f9037aa8d
Author: Martin André <email address hidden>
Date: Mon Feb 18 18:24:38 2019 +0100

Fix malformed mistal_executor kolla config file

    In the event UndercloudConfigFilePath is empty, the produced
    /var/lib/kolla/config_files/mistral_executor.json file was malformed
    and kolla would complain with:

    + sudo -E kolla_set_configs
    INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
    INFO:__main__:Validating config file
    ERROR:__main__:Unexpected error:
    Traceback (most recent call last):
      File "/usr/local/bin/kolla_set_configs", line 412, in main
        config = load_config()
      File "/usr/local/bin/kolla_set_configs", line 297, in load_config
        validate_config(config)
      File "/usr/local/bin/kolla_set_configs", line 238, in validate_config
        if not set(data.keys()) >= required_keys:
    AttributeError: 'unicode' object has no attribute 'keys'