[SRU] Failed to create port on network <UUID>, because fixed_ips included invalid subnet <UUID>

Bug #2038109 reported by Felipe Reyes
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Magnum
Fix Released
Undecided
Felipe Reyes
Ubuntu Cloud Archive
Invalid
Undecided
Unassigned
Antelope
In Progress
Undecided
Unassigned
Bobcat
Won't Fix
Undecided
Unassigned
Ussuri
New
Undecided
Unassigned
Yoga
New
Undecided
Unassigned
Zed
Won't Fix
Undecided
Unassigned
magnum (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
In Progress
Undecided
Unassigned
Jammy
In Progress
Undecided
Unassigned

Bug Description

[Impact]

When creating a new "cluster"that overrides the fixed network defined in the cluster template, but not the subnet.

It would be expected that the cluster create request fails with a 400 error since the client is submitting an invalid request.

[Environment]

Focal Ussuri

[Test Case]

1. Create a new cluster template WITHOUT a fixed network/subnet set.
openstack coe cluster template create k8s-cluster-template \
    --image fedora-coreos-32 \
    --keypair testkey \
    --external-network ext_net \
    --flavor m1.small \
    --network-driver flannel \
    --coe kubernetes \
    --fixed-network admin_net \
    --fixed-subnet admin_subnet

2. Create a new cluster using the template previously created and select an existing network

openstack coe cluster create \
    --cluster-template k8s-cluster-template \
    --timeout 120 \
    --fixed-network private \
    k8scluster

Expected result

The cluster gets created

Actual result:

The cluster creation fails with the following error:

$ openstack coe cluster show k8scluster -f json -c faults | jq -r '.faults'
{
  "default-master": "Resource CREATE failed: BadRequest: resources.kube_masters.resources[0].resources.kube_master_eth0: Invalid input for operation: Failed to create port on network 525df7a4-1aeb-4eae-a37f-432a809a8161, because fixed_ips included invalid subnet 30e1b4ed-811f-4226-a19d-0a56cc72fc10.\nNeutron server returns request_ids: ['req-7a55a40a-3aa3-4a67-8ecf-b2e47ae16a84']",
  "default-worker": "Resource CREATE failed: BadRequest: resources.kube_masters.resources[0].resources.kube_master_eth0: Invalid input for operation: Failed to create port on network 525df7a4-1aeb-4eae-a37f-432a809a8161, because fixed_ips included invalid subnet 30e1b4ed-811f-4226-a19d-0a56cc72fc10.\nNeutron server returns request_ids: ['req-7a55a40a-3aa3-4a67-8ecf-b2e47ae16a84']"
}

[ Where problems could occur ]

- This change introduces validation of the configuration passed by the user during the creation of a new cluster, issues can be raised during the creation of new clusters, but not for already created clusters.

[ Other Info ]

- The patches associated to this SRU were merged during the OpenStack 2024.1 (Caracal) devel cycle
- Patches
  + https://opendev.org/openstack/magnum/commit/753baadbb8b5b4c3032d4618166b1c899a50fb07
  + https://opendev.org/openstack/magnum/commit/a8bce0bfee81218cd1c0ddcf3e2b86b96659933e

Tags: patch

Related branches

Felipe Reyes (freyes)
Changed in magnum (Ubuntu):
status: New → Confirmed
Revision history for this message
Felipe Reyes (freyes) wrote :
Download full text (16.0 KiB)

the issue can be reproduced using master (with devstack)

$ openstack coe cluster template create k8s-cluster-template --image fedora-coreos-35.20220116.3.0-openstack.x86_64 --keypair testkey --external-network public --flavor m1.small --network-driver flannel --coe kubernetes
Request to create cluster template k8s-cluster-template accepted
+-----------------------+------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------+
| insecure_registry | - |
| labels | {} |
| updated_at | - |
| floating_ip_enabled | True |
| fixed_subnet | - |
| master_flavor_id | - |
| uuid | 9c7edc11-a710-4f24-8850-f9911e7ff0d5 |
| no_proxy | - |
| https_proxy | - |
| tls_disabled | False |
| keypair_id | testkey |
| public | False |
| http_proxy | - |
| docker_volume_size | - |
| server_type | vm |
| external_network_id | public |
| cluster_distro | fedora-coreos |
| image_id | fedora-coreos-35.20220116.3.0-openstack.x86_64 |
| volume_driver | - |
| registry_enabled | False |
| docker_storage_driver | overlay2 |
| apiserver_port | - |
| name | k8s-cluster-template |
| created_at | 2023-10-03T19:45:21.945595+00:00 |
| network_driver | flannel |
| fixed_network | - |
| coe | kubernetes |
| flavor_id | m1.small |
| master_lb_enabled | False |
| dns_nameserver | 8.8.8.8 |
| hidden | False |
| tags | - |
+-----------------------+------------------------------------------------+
$ openstack coe cluster create \
    --cluster-template k8s-cluster-template \ ...

Revision history for this message
Felipe Reyes (freyes) wrote :

The approach I'm taking to solve this problem is to implement the validate_fixed_network and validate_fixed_subnet - https://opendev.org/openstack/magnum/src/commit/a3276fd53b3a018522e3159137664d9808603514/magnum/api/attr_validator.py#L99

Changed in magnum:
assignee: nobody → Felipe Reyes (freyes)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to magnum (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum/+/897362

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum/+/897506

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum/+/897507

Revision history for this message
Felipe Reyes (freyes) wrote : Re: Failed to create port on network <UUID>, because fixed_ips included invalid subnet <UUID>

There is an issue with magnum-ui preventing users from selecting a subnet when overriding the network, see https://bugs.launchpad.net/magnum-ui/+bug/2038663

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to magnum-ui (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/magnum-ui/+/898007

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to magnum (master)

Reviewed: https://review.opendev.org/c/openstack/magnum/+/897362
Committed: https://opendev.org/openstack/magnum/commit/753baadbb8b5b4c3032d4618166b1c899a50fb07
Submitter: "Zuul (22348)"
Branch: master

commit 753baadbb8b5b4c3032d4618166b1c899a50fb07
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 5 18:39:39 2023 -0300

    Add validator for fixed_network.

    The validate_fixed_network() validator looks for an available network
    matching the id or name, if no networks match a FixedNetworkNotFound
    exception is raised, if more than one networks match then the Conflict
    exception is raised.

    Partial-Bug: #2038109
    Change-Id: I7fb0122889feb4f8fa039be5752e8ac3fbc23b94

Revision history for this message
Jake Yip (waipengyip) wrote : Re: Failed to create port on network <UUID>, because fixed_ips included invalid subnet <UUID>

Hi Felipe,

Thanks for finding this. Looks like fixed_network and fixed_subnet must both be specified, else things break apart at the heat driver layer.

- fixed-network without fixed-subnet = heat can't find subnet with error "Multiple subnet matches found for name '', use an ID to be more specific."
- fixed-subnet without fixed-network = heat template doesn't use the subnet due to the detection at [1]

I think it is fair that there are some validation to make sure network / subnet exist. But I'm not sure if we should validate that both of them exist together in the api/attr_validation.py layer. Future driver implementation may only need one and not the other. If Heat needs both, Heat should validate it.

Can we move forward by simply validating subnet exists like how you did with network?

[1] https://opendev.org/openstack/magnum/src/branch/stable/2023.2/magnum/drivers/heat/template_def.py#L591

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to magnum-ui (master)

Reviewed: https://review.opendev.org/c/openstack/magnum-ui/+/898007
Committed: https://opendev.org/openstack/magnum-ui/commit/6f6c3db282fe2f0e08ad69c557eb153858b0164a
Submitter: "Zuul (22348)"
Branch: master

commit 6f6c3db282fe2f0e08ad69c557eb153858b0164a
Author: Felipe Reyes <email address hidden>
Date: Tue Oct 10 18:33:27 2023 -0300

    Add dropdown list to select fixed subnet

    When creating a cluster and the user decides to reuse an existing
    network, it must also choose a subnet, otherwise the resulting
    configuration may be invalid when trying to allocate a port.

    Closes-Bug: #2038663
    Related-Bug: #2038109
    Change-Id: I08943f10418385a39eecc7b4117d162854d2d010

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to magnum-ui (stable/2023.2)

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/magnum-ui/+/905083

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to magnum (master)

Reviewed: https://review.opendev.org/c/openstack/magnum/+/897506
Committed: https://opendev.org/openstack/magnum/commit/a8bce0bfee81218cd1c0ddcf3e2b86b96659933e
Submitter: "Zuul (22348)"
Branch: master

commit a8bce0bfee81218cd1c0ddcf3e2b86b96659933e
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 5 18:45:55 2023 -0300

    Add validator for fixed_subnet

    Validate the existance of the subnet referenced by fixed_subnet. It's
    not checked if the subnet is associated to the fixed_network.

    Closes-Bug: #2038109
    Change-Id: Ia75f0ae525b768ad5b965d22b522cca6f80dcab2

Changed in magnum:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/magnum 18.0.0.0rc1

This issue was fixed in the openstack/magnum 18.0.0.0rc1 release candidate.

Revision history for this message
Felipe Reyes (freyes) wrote : Re: Failed to create port on network <UUID>, because fixed_ips included invalid subnet <UUID>

Marking Caracal as invalid since this patch is already included in the version shipped with OpenStack Caracal

Changed in cloud-archive:
status: New → Invalid
Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching debdiff for Antelope

Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching debdiff for Bobcat

Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching debdiff for Zed

Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching debdiff for Jammy (Yoga)

Changed in magnum (Ubuntu Jammy):
status: New → In Progress
Revision history for this message
Felipe Reyes (freyes) wrote :

Attaching debdiff for Focal (Ussuri)

Changed in magnum (Ubuntu Focal):
status: New → In Progress
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp2038109_antelope.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Felipe Reyes (freyes) wrote :

marking Zed as won't fix, because is EOL

Revision history for this message
Felipe Reyes (freyes) wrote :

marking bobcat as won't fix since it's EOL

summary: - Failed to create port on network <UUID>, because fixed_ips included
- invalid subnet <UUID>
+ [SRU] Failed to create port on network <UUID>, because fixed_ips
+ included invalid subnet <UUID>
Felipe Reyes (freyes)
description: updated
Revision history for this message
James Page (james-page) wrote :

Uploaded to Ubuntu Jammy and Focal for SRU team review and to the Antelope UCA.

Revision history for this message
James Page (james-page) wrote :

Fix accepted upstream last year to 24.04 onwards do not have this bug.

Changed in magnum (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.