nova-compute needs to set migration hostname to resolved fqdn

Bug #2089781 reported by Edward Hope-Morley
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Fix Committed
High
Edward Hope-Morley

Bug Description

The cloud-compute relation is used to communicate address and hostname used by the compute host for live-migration. This is consumed by the nova-cloud-controller charm to populate and propagate ssh known_hosts to all hypervisors. If the address is from a non-OAM network (MAAS), the nova-cloud-controller will reverse resolve the address and get a different fqdn to that obtained when doing socket.gethostname() on the nova-compute host.

The tl;dr is that MAAS will prepend the fqdn with the interface name on which the address is configured for non-OAM (or "MAAS Provided") networks. This is done so as when you reverse dns an address you get a unique fqdn and vice-versa. Where this becomes a problem is when Nova performs a live-migration for a vm where the image originally used to create that vm no longer exists in Glance. When this happens Nova reverts to doing an scp of the vm backing image from src to dst and for this it uses the nova.conf:host config which with the charms will always be an fqdn from the OAM network. This results in a "Host key verification failed." error because when a non-oam network is used for migration only the fqdn for the address from that network will have been added to the known_hosts file.

The cloud-compute 'hostname' setting is set [1] by the nova-compute charm to provide the oam fqdn to the nova-cloud-controller which then adds it to the known_hosts but since the fqdn is not obtained in the same way that nova.conf:host is they can get different values which when different cause the above scp to fail.

This is not a problem that surfaces often because many deployments use the OAM network for the migrate and cloud-compute bindings but it is also not uncommon for the migrate binding to be set to something different in which case this needs to work.

Proposal here is to modify [1] to be more like [2] so that they both return the same fqdn and that way both nova-cloud-controller and compute are guaranteed to match.

[1] https://github.com/openstack/charm-nova-compute/blob/7c60a9241468dc67c0efcd5f757fc552970a5e81/hooks/nova_compute_hooks.py#L441
[2] https://github.com/openstack/charm-nova-compute/blob/7c60a9241468dc67c0efcd5f757fc552970a5e81/hooks/charmhelpers/contrib/openstack/context.py#L2383
[3] https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1969971

summary: - nova-compute needs to migration hostname to resolved fqdn
+ nova-compute needs to set migration hostname to resolved fqdn
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (master)
Changed in charm-nova-compute:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/936452

Changed in charm-nova-compute:
assignee: nobody → Edward Hope-Morley (hopem)
Changed in charm-nova-compute:
importance: Undecided → High
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/937006

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/937007

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (stable/zed)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/937009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (master)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/936451
Committed: https://opendev.org/openstack/charm-nova-compute/commit/25a0aba59d0b760b1c12b2c99c269aad80d2e087
Submitter: "Zuul (22348)"
Branch: master

commit 25a0aba59d0b760b1c12b2c99c269aad80d2e087
Author: Edward Hope-Morley <email address hidden>
Date: Thu Nov 28 13:11:51 2024 +0000

    Make cloud-compute hostname same as nova.conf host

    To avoid inconsistency between the fqdn added to the nova.conf
    'host' config and the hostname set of the cloud-compute relation
    we need to ensure that both are using the same means of determining
    the host fqdn. This is so that the when nova-cloud-controller
    charm populates ssh known_hosts and shares with all computes it
    definitely contains an entry for the nova.conf host config since
    this is sometimes used as part of a live-migration.

    Change-Id: Iddcf3fdacf7e7374b57b628edf1e6d0df4da082c
    Closes-Bug: #2089781

Changed in charm-nova-compute:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/936452
Committed: https://opendev.org/openstack/charm-nova-compute/commit/c7189347c9df5465c300ebcf74b6a02a86ec1f4a
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit c7189347c9df5465c300ebcf74b6a02a86ec1f4a
Author: Edward Hope-Morley <email address hidden>
Date: Thu Nov 28 13:11:51 2024 +0000

    Make cloud-compute hostname same as nova.conf host

    To avoid inconsistency between the fqdn added to the nova.conf
    'host' config and the hostname set of the cloud-compute relation
    we need to ensure that both are using the same means of determining
    the host fqdn. This is so that the when nova-cloud-controller
    charm populates ssh known_hosts and shares with all computes it
    definitely contains an entry for the nova.conf host config since
    this is sometimes used as part of a live-migration.

    Change-Id: Iddcf3fdacf7e7374b57b628edf1e6d0df4da082c
    Closes-Bug: #2089781

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/937006
Committed: https://opendev.org/openstack/charm-nova-compute/commit/fde0780f73fb3e80545ed0c895a4cf3a4320b7e6
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit fde0780f73fb3e80545ed0c895a4cf3a4320b7e6
Author: Edward Hope-Morley <email address hidden>
Date: Thu Nov 28 13:11:51 2024 +0000

    Make cloud-compute hostname same as nova.conf host

    To avoid inconsistency between the fqdn added to the nova.conf
    'host' config and the hostname set of the cloud-compute relation
    we need to ensure that both are using the same means of determining
    the host fqdn. This is so that the when nova-cloud-controller
    charm populates ssh known_hosts and shares with all computes it
    definitely contains an entry for the nova.conf host config since
    this is sometimes used as part of a live-migration.

    Also updates dependency requirements (using pip-compile) to
    fix broken dependency versions for tempest.

    Change-Id: Iddcf3fdacf7e7374b57b628edf1e6d0df4da082c
    Closes-Bug: #2089781

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/937007
Committed: https://opendev.org/openstack/charm-nova-compute/commit/5b3154a701d8c5b7f2d6cd3d25bfc61f44383c1e
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 5b3154a701d8c5b7f2d6cd3d25bfc61f44383c1e
Author: Edward Hope-Morley <email address hidden>
Date: Thu Nov 28 13:11:51 2024 +0000

    Make cloud-compute hostname same as nova.conf host

    To avoid inconsistency between the fqdn added to the nova.conf
    'host' config and the hostname set of the cloud-compute relation
    we need to ensure that both are using the same means of determining
    the host fqdn. This is so that the when nova-cloud-controller
    charm populates ssh known_hosts and shares with all computes it
    definitely contains an entry for the nova.conf host config since
    this is sometimes used as part of a live-migration.

    Change-Id: Iddcf3fdacf7e7374b57b628edf1e6d0df4da082c
    Closes-Bug: #2089781

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/937008
Committed: https://opendev.org/openstack/charm-nova-compute/commit/3325fd5334233ce2bddae25d4b9b08b2471814ab
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 3325fd5334233ce2bddae25d4b9b08b2471814ab
Author: Edward Hope-Morley <email address hidden>
Date: Thu Nov 28 13:11:51 2024 +0000

    Make cloud-compute hostname same as nova.conf host

    To avoid inconsistency between the fqdn added to the nova.conf
    'host' config and the hostname set of the cloud-compute relation
    we need to ensure that both are using the same means of determining
    the host fqdn. This is so that the when nova-cloud-controller
    charm populates ssh known_hosts and shares with all computes it
    definitely contains an entry for the nova.conf host config since
    this is sometimes used as part of a live-migration.

    Also removes Kinetic-Zed test.

    Change-Id: Iddcf3fdacf7e7374b57b628edf1e6d0df4da082c
    Closes-Bug: #2089781

tags: added: in-stable-zed
tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/937009
Committed: https://opendev.org/openstack/charm-nova-compute/commit/f54a86911a1d4afd578d5cce0bb6f949feedd226
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f54a86911a1d4afd578d5cce0bb6f949feedd226
Author: Edward Hope-Morley <email address hidden>
Date: Thu Nov 28 13:11:51 2024 +0000

    Make cloud-compute hostname same as nova.conf host

    To avoid inconsistency between the fqdn added to the nova.conf
    'host' config and the hostname set of the cloud-compute relation
    we need to ensure that both are using the same means of determining
    the host fqdn. This is so that the when nova-cloud-controller
    charm populates ssh known_hosts and shares with all computes it
    definitely contains an entry for the nova.conf host config since
    this is sometimes used as part of a live-migration.

    Change-Id: Iddcf3fdacf7e7374b57b628edf1e6d0df4da082c
    Closes-Bug: #2089781

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.