| 2023-10-16 23:42:10 |
Brett Holman |
bug |
|
|
added bug |
| 2023-10-17 01:06:38 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2023-10-17 09:19:57 |
Alberto Contreras |
bug |
|
|
added subscriber Alberto Contreras |
| 2023-10-17 09:54:53 |
Andrew Cloke |
bug |
|
|
added subscriber Andrew Cloke |
| 2023-10-17 22:02:44 |
Chad Smith |
cloud-init (Ubuntu): status |
New |
Fix Committed |
|
| 2023-10-17 23:28:02 |
Chad Smith |
cloud-init (Ubuntu): importance |
Undecided |
High |
|
| 2023-10-17 23:28:05 |
Chad Smith |
cloud-init (Ubuntu): importance |
High |
Critical |
|
| 2023-10-17 23:28:21 |
Chad Smith |
cloud-init (Ubuntu): assignee |
|
Brett Holman (holmanb) |
|
| 2023-10-19 18:32:31 |
Chad Smith |
description |
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
The symptom we can see on long-delayed systemd-users-sessions.service is that ssh and a non-root user will get this console error message until cloud-config.service has completed in early boot:
`System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).`
We really want to reduce that time to ssh for non-root users, so expected short-term fix in downstream Ubuntu is to revert the commit b3c9b6a7.
Long-term will be a resolution in cloud-init upstream to avoid strict systemd unit ordering After=snapd.seeded.service for cloud-config.service by only blocking on snap seeding if the user-data/vendor-data provided to the instance depends on snaps being present.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-10-19 18:53:28 |
Chad Smith |
description |
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
The symptom we can see on long-delayed systemd-users-sessions.service is that ssh and a non-root user will get this console error message until cloud-config.service has completed in early boot:
`System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).`
We really want to reduce that time to ssh for non-root users, so expected short-term fix in downstream Ubuntu is to revert the commit b3c9b6a7.
Long-term will be a resolution in cloud-init upstream to avoid strict systemd unit ordering After=snapd.seeded.service for cloud-config.service by only blocking on snap seeding if the user-data/vendor-data provided to the instance depends on snaps being present.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-10-19 18:53:33 |
Chad Smith |
description |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
=== Begin SRU Template ===
[Impact]
In 23.3.1, system unit ordering changes moved the configuration Before=systemd-user-sessions.service out of the earlier cloud-init.service boot stage and into the subsequent cloud-config.service boot stage to ensure all users have been configured before providing a login prompt on the console. This was originally intended to fix LP: #2013403 for emulated riscv environments, but the general time to SSH/login cost is too great at this stage to leave the impact active in most images where snap seeding is being perfomed on first boot.
Leaving this current extended delay to login unresolved also breaks tooling which uses `uvt-kvm wait` which is present in some continuous integration testing.
The fix is to revert https://github.com/canonical/cloud-init/commit/b3c9b6a7.
[Test Case]
Launch a daily image and a daily-image with proposed cloud-init and compare the following data points:
- time to SSH: number of retries required to successfully SSH into the VM
- validate time to ssh by sampling: systemd-analyze critical-chain systemd-user-sessions.service to see total time until login was unblocked from systemd perspective
- systemctl show -p Before,After cloud-config.service cloud-init.service --no-pager
- systemd-analyze blame: # validate systemd-user-sessions is not one of the longest blocks to boot
[Regression Potential]
This is a revert to functionality that was working for years. It will regress emulated riscV users per LP: #2013403 as they may be able to see a login prompt that will show up before cloud-config completes and could reject their correct configured password as invalid until the cloud-config.service completes setup on first boot.
[Other info]
LP: #2013403
LP: #2039441
[Original Description]
Affected version: 23.3
Commit b3c9b6a7 introduced a dependency on snapd.seeded.service for systemd-user-sessions.service. This adds a 9.65s delay to user login (ssh or local tty) which was missed during performance testing[2] due to using an image that had already seeded snapd (testing on a dirty image re-run via `cloud-init clean`).
This was discovered while investigating LP: #2039441.
[1] https://github.com/canonical/cloud-init/commit/b3c9b6a79c85ebc8c87908383c34b0314c2205b6
[2] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930 |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Lunar |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Lunar) |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Focal |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Focal) |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Jammy |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Jammy) |
|
| 2023-10-24 14:50:51 |
Chad Smith |
nominated for series |
|
Ubuntu Mantic |
|
| 2023-10-24 14:50:51 |
Chad Smith |
bug task added |
|
cloud-init (Ubuntu Mantic) |
|
| 2023-10-24 21:44:25 |
Ubuntu Archive Robot |
bug |
|
|
added subscriber James Falcon |
| 2023-10-30 19:15:41 |
Chad Smith |
cloud-init (Ubuntu Focal): status |
New |
Fix Committed |
|
| 2023-10-30 19:15:45 |
Chad Smith |
cloud-init (Ubuntu Jammy): status |
New |
Fix Committed |
|
| 2023-10-30 19:15:47 |
Chad Smith |
cloud-init (Ubuntu Lunar): status |
New |
Fix Committed |
|
| 2023-10-30 19:15:51 |
Chad Smith |
cloud-init (Ubuntu Mantic): status |
New |
Fix Committed |
|
| 2023-11-11 11:40:30 |
Launchpad Janitor |
cloud-init (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2023-11-16 20:02:04 |
Andreas Hasenack |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2023-11-16 20:02:14 |
Andreas Hasenack |
bug |
|
|
added subscriber SRU Verification |
| 2023-11-16 20:02:19 |
Andreas Hasenack |
tags |
|
verification-needed verification-needed-mantic |
|
| 2023-11-16 20:04:30 |
Andreas Hasenack |
tags |
verification-needed verification-needed-mantic |
verification-needed verification-needed-lunar verification-needed-mantic |
|
| 2023-11-16 20:07:10 |
Andreas Hasenack |
tags |
verification-needed verification-needed-lunar verification-needed-mantic |
verification-needed verification-needed-jammy verification-needed-lunar verification-needed-mantic |
|
| 2023-11-16 20:08:01 |
Andreas Hasenack |
tags |
verification-needed verification-needed-jammy verification-needed-lunar verification-needed-mantic |
verification-needed verification-needed-focal verification-needed-jammy verification-needed-lunar verification-needed-mantic |
|
