Upgrade scripts have different versions merged in upgrade.d folder

Bug #2091944 reported by Heitor Matsui
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Heitor Matsui

Bug Description

Brief Description
-------
During upgrade activation, in /etc/upgrade.d/, two different versions of scripts are mixed together:

$ ls
01-remove-armada-if-unused.py
02-delete-pod-eviction-timeout-service-parameter.py
02-remove-psp-service-parameter.py
03-delete-encrypted-fs-service-parameter.sh
04-add-tls-version-cipher-parameters.sh
05-oidc-service-parameters-migration.py
06-security-kubeapi-service-parameters-apply.py
07-remove-vault.sh
08-activate-keystone.py
09-cleanup-dnsmasq-hosts-file.py
100-update_dcmanager_subclouds_table.py
101-configure-ipsec-on-hosts.py
102-create-required-platform-certs.py
103-update-psql-roles.py
104-remove-admin-role.py
10-renew-pxeboot-dhcp-leases.py
117-ptp-db-upgrade.py
11-update-static-hieradata.py
12-create-sysinv-upgrade-entries.py
13-add-mgmt-fqdn-to-dnsmasq.py
13-drbd-update-secure-config.sh
14-populate-ihost-sw-version-field.py
15-update-controllerfs-state.py
16-update-filesystems-capabilities.py
17-ldap-users-backup.sh
18-assert-docker-health.sh
18-resize-systemcontroller-filesystems.sh
196-k8s-update-system-node-rbac.sh
197-reset-config-target.py
198-update-isystem-data.py
199-update-sysinv-motd.py
19-enable-fluxcd-controllers.py
19-recreate-vim-systemcontroller-endpoint.sh
20-barbican-data-migration.sh
20-k8s-app-upgrade.sh
21-keystone-data-migration.sh
21-watch-apps-upgrade.sh
22-add-service-parameter.py
22-resize-systemcontroller-filesystems.sh
23-recreate-vim-systemcontroller-endpoint.sh
24-barbican-data-migration.sh
25-keystone-data-migration.sh
26-add-service-parameter.py
29-mgmt-network-fqdn-reconfig.py
30-update_network_addrpool_table.py
35-sync-system-local-ca-in-subclouds.py
40-assert-docker-health.sh
50-remove-calico-ipip.py
62-enable-fluxcd-controllers.py
65-k8s-app-upgrade.sh
66-k8s-wrcp-dm-app-upgrade.sh
67-update-openldap-users.sh
80-remove-psp-autogenerated-policies.py
82-watch-apps-upgrade.sh
83-restart-vault-pods.sh
86-upgrade-dm.sh
88-remove-subcloud-endpoints.py
89-sync-dcorch-subcloud-mgmt-ip.py
92-reset-k8s-pki-folder-permissions.sh
93-update-mgmt-ipsec-flag.py
94-capabilities-column-cleaning.py

There's, among others, cases like:

20-k8s-app-upgrade.sh and 65-k8s-app-upgrade.sh;
18-resize-systemcontroller-filesystems.sh and 22-resize-systemcontroller-filesystems.sh
We can see the wrong execution, in software.log (this script is now in position #22):

2024-12-13T17:30:42: 18-resize-systemcontroller-filesystems.sh[81955]: INFO: Starting filesystems resize on DC System Controller for increased parallel subcloud deployment for from release 22.12 to 24.09 with action activate
2024-12-13T17:30:42: 18-resize-systemcontroller-filesystems.sh[81955]: INFO: Not a DC System Controller deployment. No filesystem resize needed.
2024-12-13T17:30:42: 18-resize-systemcontroller-filesystems.sh[81955]: INFO: Filesystems resizing for DC System Controller finished successfully for from release 22.12 to 24.09 with action activate

Severity
-------
<Major: System/Feature is usable but degraded>

Steps to Reproduce
-------
1 - Install stx-8
2 - Upgrade system controller
3 - Upgrade subclouds <- hit issue here

Expected Behavior
-------
Subcloud is upgraded

Actual Behavior
-------
Activate phase failed due to cert-manager error

Reproducibility
-------
Reproduced in two different subclouds

System Configuration
-------
Any

Last Pass
-------
N/A

Timestamp/Logs
-------
In description above.

Alarms
-------
NA

Test Activity
-------
Upgrade Testing

Workaround
-------
NA

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/937885

Changed in starlingx:
status: New → In Progress
Changed in starlingx:
assignee: nobody → Heitor Matsui (heitormatsui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to update (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/update/+/937888

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/c/starlingx/config/+/937885
Committed: https://opendev.org/starlingx/config/commit/306c4b20476150ee24e472a6cef1ccdbe8064e65
Submitter: "Zuul (22348)"
Branch: master

commit 306c4b20476150ee24e472a6cef1ccdbe8064e65
Author: Heitor Matsui <email address hidden>
Date: Tue Dec 17 11:51:13 2024 -0300

    Move upgrade scripts to /usr/local/share

    In the current directory /etc/upgrade.d, when upgrade scripts
    are removed or renamed in a patch, the patching system using
    apt-ostree does not recognize these changes because the files
    are considered config files, and ends up with duplicated files
    in the directory after the patch is applied.

    This commit moves the upgrade scripts from /etc to /usr/local/share
    so that scripts are changed correctly in the scenario described
    previously.

    TODO (another commit): change the patch scripts directory from
                           /etc/update.d to /usr/local/share/

    Test Plan
    PASS: AIO-SX: apply a patch with deleted, changed and renamed scripts
          in stx-10, verify the changes are applied accordingly
    PASS: AIO-SX: build a prepatched iso with the changes from the
          previous TC and run a major release deployment stx-8 -> stx-10

    Closes-bug: 2091944

    Change-Id: Ia36dcda9630a9b3d7e37fe0ce82e3e1fcb09daa1
    Signed-off-by: Heitor Matsui <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to update (master)

Reviewed: https://review.opendev.org/c/starlingx/update/+/937888
Committed: https://opendev.org/starlingx/update/commit/9da2ee318676011812ee1ee44b5c5dc155c8e7cd
Submitter: "Zuul (22348)"
Branch: master

commit 9da2ee318676011812ee1ee44b5c5dc155c8e7cd
Author: Heitor Matsui <email address hidden>
Date: Tue Dec 17 12:13:48 2024 -0300

    Change migrate/activate default script directory

    This commit changes the default location of upgrade scripts,
    which is currently used by major release deployment.

    Test Plan
    PASS: AIO-SX: run stx-8 -> stx-10 major release deployment e2e,
          verify the new script location is used in migrate/activate

    Depends-on: https://review.opendev.org/c/starlingx/config/+/937885
    Partial-bug: 2091944

    Change-Id: I3791030a2c7827308146bc6cbca91b6bd2014e46
    Signed-off-by: Heitor Matsui <email address hidden>

Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.10.0 stx.update
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.