Ubuntu
cloud-init package

Time To SSH Regression

  1. Lunar (23.04)
  2. Bug #2039505
Bug #2039505 reported by Brett Holman
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
Critical
Brett Holman
Focal
Fix Committed
Undecided
Unassigned
Jammy
Fix Committed
Undecided
Unassigned
Lunar
Fix Committed
Undecided
Unassigned
Mantic
Fix Committed
Undecided
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.

Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.

The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.

[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:

  - time to SSH: number of retries required to successfully SSH into the VM
  - validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
  - systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
  - systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot

[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.

[Other info]
LP: #2013403
LP: #2039441

[Original Description]

Affected version: 23.3

Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).

This was discovered while investigating LP: #2039441.

[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930

See original description
Chad Smith (chad.smith)
Changed in cloud-init (Ubuntu):
status: New → Fix Committed
Revision history for this message
Chad Smith (chad.smith) wrote :

Downstream commit[1] for Ubuntu merged which reverts this change. It will be released as cloud-init version 23.3.2-0ubuntu0~23.10.1.

[1] https://github.com/canonical/cloud-init/commit/052d898023fbd6f7d87338e31f6cca6535cccef7

When cloud-init merges the ability to avoid snap.seeded.service costs in cloud-config.service this change will be re-applied.

Changed in cloud-init (Ubuntu):
importance: Undecided → High
importance: High → Critical
assignee: nobody → Brett Holman (holmanb)
Chad Smith (chad.smith)
description: updated
Chad Smith (chad.smith)
description: updated
description: updated
Chad Smith (chad.smith)
Changed in cloud-init (Ubuntu Focal):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Jammy):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Lunar):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Mantic):
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 23.4~3g0cb0b80f-0ubuntu1

---------------
cloud-init (23.4~3g0cb0b80f-0ubuntu1) noble; urgency=medium

  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * drop the following cherry-picks now included:
    + d/p/cpick-0d9f149a-Pytestify-apt-config-test-modules-4424
    + d/p/cpick-5023e9f9-Refactor-test_apt_source_v1.py-to-use-pytest-4427
    + d/p/cpick-e9cdd7e3-Install-gnupg-if-gpg-not-found-4431
    + d/p/cpick-015543d3-apt-install-software-properties-common-when-absent-but
  * Upstream snapshot based on upstream/main at 0cb0b80f.
    - Bugs fixed in this snapshot: (LP: #2034273, #2030729)
      (LP: #2038945, #2039453)

 -- Chad Smith <email address hidden> Wed, 08 Nov 2023 22:24:09 -0700

Changed in cloud-init (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Alan Sparks (alsparks) wrote :

Is there information when this updated package will land in the Focal and Jammy updates? Seems only on trunk for now. Thanks.

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Please test proposed package

Hello Brett, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-mantic
tags: added: verification-needed-lunar
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Brett, or anyone else affected,

Accepted cloud-init into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed-jammy
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Brett, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed-focal
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Brett, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.3.3-0ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.