Ironic-conductor container time out

Bug #2089027 reported by Sylvère K
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kayobe
New
Undecided
Unassigned

Bug Description

When deploying services on the overcloud (`kayobe overcloud service deploy`), the ironic-conductor container time-out when restarting.

Running the kayobe command twice seems to fix this issue*.

RUNNING HANDLER [ironic : Restart ironic-conductor container] *************************************************************************************************************
fatal: [controller0]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1732193014, "HostConfig": {"NetworkMode": "host"}, "Id": "cb8495e89cd88b101cf32191eb76dfa9b09bf26820978bf03a841832c7af6acf", "Image": "quay.io/openstack.kolla/ironic-conductor:2024.1-rocky-9", "ImageID": "sha256:102cecade69a3635235d935c9ce54fa1e6b3ef421948dcef1584ef370837c81e", "Labels": {"build-date": "20241023", "io.buildah.version": "1.33.7", "kolla_version": "18.2.1", "license": "BSD-3-Clause", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "ironic-conductor", "org.opencontainers.image.authors": "Magauer Lukas, Neil Hanlon, Louis Abel", "org.opencontainers.image.license": "BSD-3-Clause", "org.opencontainers.image.name": "rockylinux", "org.opencontainers.image.url": "https://github.com/rocky-linux/rocky-toolbox-images", "org.opencontainers.image.vendor": "Rocky Enterprise Software Foundation", "org.opencontainers.image.version": "9", "vendor": "Rocky Enterprise Software Foundation", "version": "9"}, "Mounts": [{"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/ironic-conductor", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/lib/modules", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/lib/modules", "Type": "bind"}, {"Destination": "/var/lib/ironic", "Driver": "local", "Mode": "rw", "Name": "ironic", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/ironic/_data", "Type": "volume"}, {"Destination": "/var/lib/ironic/metrics", "Driver": "local", "Mode": "rw", "Name": "ironic_prometheus_exporter_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/ironic_prometheus_exporter_data/_data", "Type": "volume"}, {"Destination": "/dev", "Mode": "rw", "Propagation": "rprivate", "RW": true, "Source": "/dev", "Type": "bind"}, {"Destination": "/run", "Mode": "shared", "Propagation": "shared", "RW": true, "Source": "/run", "Type": "bind"}, {"Destination": "/sys", "Mode": "rw", "Propagation": "rprivate", "RW": true, "Source": "/sys", "Type": "bind"}], "Names": ["/ironic_conductor"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DNSNames": null, "DriverOpts": null, "EndpointID": "4499fc017c315973d3e66a65bbe4948684444a47d068583d91cae4b1719957f1", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": "488673867c539b0ce1ed467a7e4b9a77f7332a06a43838f112ad2e10598c28b7"}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"}

*The Ironic component doesn't seem to work properly even if the deployment process return no error.

Could be linked to this ironic bug : https://bugs.launchpad.net/ironic/+bug/2089029

Sylvère K (sylvr)
description: updated
description: updated
Revision history for this message
Pierre Riteau (priteau) wrote :

Can you share which version of Kayobe is being used?

Sylvère K (sylvr)
Changed in kayobe:
status: New → Incomplete
status: Incomplete → New
Revision history for this message
Sylvère K (sylvr) wrote :

kayobe 16.0.1.dev2

Upgrading the Ironic component fixed the issue.

What's the best way to keep track of component's version to verify in which version a bug has been fixed ?

Thanks.

Revision history for this message
Riccardo Pittau (rpittau) wrote :

@Sylvere we as ironic community we do our best to keep track of fixed bugs in our projects release notes, for example for ironic https://docs.openstack.org/releasenotes/ironic/

other openstack projects do the same, if a bug has been fixed usually has a link to a patch which contains a release note assigned to a specific release/version

that should help tracking the bug/fix

Revision history for this message
Sylvère K (sylvr) wrote (last edit ):

Well, the issue might have been fixed in 24.1.4.dev1, which isn't yet in the release notes.

I think I should go to stable release instead of what I have now.

And I meant keeping track of the component's version deployed/upgraded by kayobe (as opposed to have to check manually components' version before and after each deploy/upgrade).

Thanks a lot, and I think this bug can be "closed" (I'm not sure which status to put it in).

Sylvère K (sylvr)
description: updated
Revision history for this message
Sylvère K (sylvr) wrote :
Download full text (26.6 KiB)

Well, I encountered this error again, so I updated the bug description with the log, and this is the log with -vvv

```
fatal: [controller0]: FAILED! => {
    "Command": "dumb-init --single-child -- kolla_start",
    "Created": 1732198835,
    "HostConfig": {
        "NetworkMode": "host"
    },
    "Id": "38bd11f6dc03d1e12c1d7b1477aee7a9190523c2f4028b99c5beab54c67bad70",
    "Image": "quay.io/openstack.kolla/ironic-conductor:2024.1-rocky-9",
    "ImageID": "sha256:102cecade69a3635235d935c9ce54fa1e6b3ef421948dcef1584ef370837c81e",
    "Labels": {
        "build-date": "20241023",
        "io.buildah.version": "1.33.7",
        "kolla_version": "18.2.1",
        "license": "BSD-3-Clause",
        "maintainer": "Kolla Project (https://launchpad.net/kolla)",
        "name": "ironic-conductor",
        "org.opencontainers.image.authors": "Magauer Lukas, Neil Hanlon, Louis Abel",
        "org.opencontainers.image.license": "BSD-3-Clause", ...

Revision history for this message
Will Szumski (willjs) wrote :

Its an odd one as it looks like an issue with your docker runtime to me. If there was a code issue with Ironic, I would still expect the container to start. Not sure of the exact conditions that cause 'Container timeout' though.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.