Ubuntu
python3-defaults package

[SRU] py3clean fails when using alternate character set

Bug #2075337 reported by Robie Basak
26
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
New
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Noble
Fix Committed
Undecided
Unassigned
dh-python (Ubuntu)
Invalid
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Noble
Invalid
Undecided
Unassigned
python3-defaults (Ubuntu)
Confirmed
Critical
Matthias Klose
Jammy
Fix Committed
Critical
Unassigned
Noble
Fix Committed
Critical
Unassigned
ubuntu-advantage-tools (Ubuntu)
New
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
Noble
New
Undecided
Unassigned

Bug Description

[ Impact ]

With the current version of python3-defaults in jammy and noble, packages which utilize py3compile and py3clean in their maintscripts (which is more or less any package which includes Python modules), and which also are affected by diversions installed by dpkg-divert, will fail to install/upgrade/remove on systems with non-UTF-8 locales, which include non-ASCII characters in their localized dpkg -L messages (and which therefore produce output that is not valid UTF-8).

As this has the potential to break apt upgrade, the impact is quite severe (especially if this occurs during a dist-upgrade). However, the scope is probably *fairly* limited given this only affects systems with non-UTF-8 locales (something that has not been the default for many years), and which produce non-ASCII output (e.g. French characters with diacriticals) in their localized messages for dpkg -L.

[ Rectification Plan ]

Firstly, python3-defaults must be patched in oracular, noble, and jammy. doko has uploaded the fix upstream and synced this to oracular. Benjamin has sponsored the noble and jammy SRUs.

Next, affected packages must be patched to include a "Breaks" line on earlier versions of python3-minimal (the python3-defaults binary package containing py3clean and py3compile).

Packages which install Python modules are easily selected, but those affected by diversions are (computationally) impossible to select (see comment 18 for details). Thus, determining the set of affected packages was non-trivial, but Benjamin has kindly produced such a list by empirical means (see comment 24). These will be added to the ticket as they are uploaded.

[ Known affected packages ]

```
$ for section in main restricted universe multiverse; do echo "noble $section:"; sqlite3 affected.sqlite3 "SELECT package, source FROM package_affected WHERE section = '$section' and install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY package;"; echo; done
noble main:
cloud-init|cloud-init
nvme-stas|nvme-stas
python3-rtslib-fb|python-rtslib-fb
targetcli-fb|targetcli-fb
ubiquity|ubiquity
ubiquity-frontend-debconf|ubiquity
ubiquity-frontend-gtk|ubiquity
ubuntu-drivers-common|ubuntu-drivers-common
ubuntu-pro-client|ubuntu-advantage-tools
walinuxagent|walinuxagent

noble restricted:

noble universe:
a2d|a2d
apertium-apy|apertium-apy
blueman|blueman
borgmatic|borgmatic
condor|condor
cpupower-gui|cpupower-gui
debomatic|debomatic
dell-recovery|dell-recovery
deluge-web|deluge
depthcharge-tools|depthcharge-tools
dhcpy6d|dhcpy6d
dkimpy-milter|dkimpy-milter
fenrir|fenrir
galileo|galileo
glances|glances
graphite-carbon|graphite-carbon
grokmirror|grokmirror
guerillabackup|guerillabackup
ionit|ionit
jupyterhub|jupyterhub
liquidctl|liquidctl
localslackirc|localslackirc
logdata-anomaly-miner|logdata-anomaly-miner
lomiri|lomiri
mailman3|mailman3
mailman3-full|mailman3
matrix-sydent|matrix-sydent
mom|mom
mopidy|mopidy
mpd-sima|mpd-sima
nordugrid-arc-arex|nordugrid-arc
notus-scanner|notus-scanner
nrpe-ng|nrpe-ng
onionprobe|onionprobe
openscap-daemon|openscap-daemon
ospd-openvas|ospd-openvas
pagekite|pagekite
pagure|pagure
pagure-ev-server|pagure
pagure-milters|pagure
pglistener|pglistener
policyd-rate-limit|policyd-rate-limit
postfix-mta-sts-resolver|postfix-mta-sts-resolver
powa-collector|powa-collector
prelude-correlator|prelude-correlator
prometheus-pgbouncer-exporter|prometheus-pgbouncer-exporter
prometheus-xmpp-alerts|prometheus-xmpp-alerts
pyroman|pyroman
python3-autopilot|autopilot
python3-btchip|btchip-python
python3-pyro5|pyro5
qweborf|weborf
razercfg|razercfg
sgt-launcher|sgt-launcher
sms4you|sms4you
supervisor|supervisor
syncplay-server|syncplay
timekpr-next|timekpr-next
twms|twms
ubiquity-frontend-kde|ubiquity
vanguards|vanguards
vaultlocker|vaultlocker
voctomix-outcasts|voctomix-outcasts
xdiagnose|xdiagnose

noble multiverse:
r4d|r4d
virtualbox|virtualbox
```

On Ubuntu 22.04 (jammy) only packages in universe are affected:

```
$ sqlite3 affected-jammy.sqlite3 "SELECT package, source FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY package;"
ganeti-3.0|ganeti
sgt-launcher|sgt-launcher
```

Note: Due to time constraints, 3861/59351 (6 %) universe package haven't been analyzed.

Code and resulting database can be found on https://github.com/bdrung/bdrung-scripts/tree/lp-2075337-find-affected

[ Test Plan ]

Where $series is jammy or noble as appropriate, and $package is the name of the affected package being SRU'd:

$ lxc launch ubuntu:$series py3clean-test
$ lxc shell py3clean-test
# locale-gen fr_FR
Generating locales (this might take a while)...
  fr_FR.ISO-8859-1... done
Generation complete.
# update-locale LANG=fr_FR
# exit
# adjust terminal to ISO-8859-1
$ lxc shell py3clean-test
# apt update && apt install --reinstall $package

With the current version of python3-defaults in jammy and noble this should fail at the final apt install --reinstall step. With the proposed version, it should succeed.

[ What Could Go Wrong? ]

The potential for regression is low since the python3-defaults changes are minimal and have been reasonably tested, and the only change to affected packages is the addition of a Breaks clause (i.e. the content of the packages are unaffected, just the meta-data).

The major cause for concern is that we are not 100% certain that the Breaks addition is sufficient to fix this. The Debian policy manual is ambiguous as to whether Breaks implies that the fixed package must be unpacked *before* the prerm maintscript of the package it breaks is executed; the policy manual mentions this must happen before the "unpack" phase, but prerm scripts run *before* the unpack phase. However, in all experiments run so far, this did indeed resolve the problem.

Still, as an extra hedge against failure here, we intend that the python3-defaults SRU is released ASAP, and that we then delay a week to allow it to propagate before releasing SRUs for the affected packages.

[ Original Description ]

We noticed this in bug 2060769 but think it is a separate issue.

dh_python3 is putting a py3clean invocation into package prerm maintainer scripts. When that runs, it explodes if the system character set is for example ISO-8859-1.

This appears to affect every package for which dh_python3 has inserted a call to py3clean into the package prerm.

python3-minimal 3.12.3-0ubuntu1

In this reproducer, cloud-init is at 24.1.3-0ubuntu3.3.

Steps to reproduce:

$ lxc launch ubuntu:noble rbasak-pro-l8n
$ lxc shell rbasak-pro-l8n
# locale-gen fr_FR
Generating locales (this might take a while)...
  fr_FR.ISO-8859-1... done
Generation complete.
# update-locale LANG=fr_FR
# exit
# adjust terminal to ISO-8859-1
$ lxc shell rbasak-pro-l8n
# apt update && apt install --reinstall cloud-init

Expected results: success

Actual results:

Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances... Fait
Lecture des informations d'état... Fait
0 mis à jour, 0 nouvellement installés, 1 réinstallés, 0 à enlever et 5 non mis à jour.
Il est nécessaire de prendre 598 ko dans les archives.
Après cette opération, 0 o d'espace disque supplémentaires seront utilisés.
Réception de :1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 cloud-init all 24.1.3-0ubuntu3.3 [598 kB]
598 ko réceptionnés en 1s (926 ko/s)
Préconfiguration des paquets...
(Lecture de la base de données... 34406 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de .../cloud-init_24.1.3-0ubuntu3.3_all.deb ...
Traceback (most recent call last):
  File "/usr/bin/py3clean", line 210, in <module>
    main()
  File "/usr/bin/py3clean", line 196, in main
    pfiles = set(dpf.from_package(options.package))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/python3/debpython/files.py", line 54, in from_package
    stdout = str(stdout, 'utf-8')
             ^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte
dpkg: avertissement: le sous-processus ancien paquet cloud-init script pre-removal a renvoyé un état de sortie d'erreur 1
dpkg: tentative d'exécution du script du nouveau paquet à la place...
Traceback (most recent call last):
  File "/usr/bin/py3clean", line 210, in <module>
    main()
  File "/usr/bin/py3clean", line 196, in main
    pfiles = set(dpf.from_package(options.package))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/python3/debpython/files.py", line 54, in from_package
    stdout = str(stdout, 'utf-8')
             ^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte
dpkg: erreur de traitement de l'archive /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb (--unpack) :
 le sous-processus nouveau cloud-init paquet pre-removal script a renvoyé un état de sortie d'erreur 1
dmesg: read kernel buffer failed: Operation not permitted
                                                         Traceback (most recent call last):
  File "/usr/bin/py3compile", line 323, in <module>
    main()
  File "/usr/bin/py3compile", line 302, in main
    compile(files, versions,
  File "/usr/bin/py3compile", line 185, in compile
    for fn, versions_to_compile in filter_files(files, e_patterns, versions):
  File "/usr/bin/py3compile", line 128, in filter_files
    for fpath in files:
  File "/usr/share/python3/debpython/files.py", line 71, in filter_public
    for fn in files:
  File "/usr/share/python3/debpython/files.py", line 54, in from_package
    stdout = str(stdout, 'utf-8')
             ^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte
dpkg: erreur lors du nettoyage:
 le sous-processus paquet cloud-init script post-installation installé a renvoyé un état de sortie d'erreur 1
Des erreurs ont été rencontrées pendant l'exécution :
 /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

See original description
Revision history for this message
Robie Basak (racb) wrote :

Investigating this a bit further, /usr/share/python3/debpython/files.py is calling "dpkg -L <package>" and when that fails to decode as UTF-8 we get the error.

I tried that for cloud-init and this in stdout for "dpkg -L cloud-init":

/etc/systemd/system/sshd-keygen@.service.d/disable-sshd-keygen-if-cloud-init-active.conf
/lib
détourné par base-files vers : /lib.usr-is-merged
/lib/systemd
/lib/systemd/system

So it seems that the issue only occurs if it lists a diversion (such as the base-files usrmerge diversion) in the output and that is being translated out of ASCII 7-bit.

Revision history for this message
Robie Basak (racb) wrote :

On the scope of the issue, I think bug affects users upgrading any package if the package uses dh_python3, ships a file that has been diverted, the system is using a non-UTF8 character set and the translation for "file is diverted" uses non-ASCII characters. cloud-init and ubuntu-advantage-tools are affected. I don't know what else.

I also saw py3compile fail for the same reason, which happens in postinst rather than prerm, so would suggest that affected systems installing such a package would also be affected, rather than just upgrading. However cloud-init and ubuntu-pro-client are of course already installed in the places where they are used. I haven't done any analysis to find further packages affected.

Revision history for this message
Robie Basak (racb) wrote :

I discussed this with Didier, and there's a concern that any SRUs that trigger this issue may break WSL users due to them possibly using ISO-8859-1 by default under certain conditions. He will check and get back to us.

In the meantime, he has asked that we do not release SRUs that may be affected. So far we know of cloud-init and ubuntu-advantage-tools. This bug therefore blocks SRUs currently in-flight. cloud-init: bug 2071762; ubuntu-advantage-tools: bug 2060769.

Revision history for this message
Robie Basak (racb) wrote :

Here's the buggy code that assumes UTF-8:

https://git.launchpad.net/ubuntu/+source/python3-defaults/tree/debpython/files.py#n54

Revision history for this message
Benjamin Drung (bdrung) wrote :

The dpkg call should set the language to enforce dpkg to output in UTF-8. I see no reason why the user's locale should be used internally.

Revision history for this message
Robie Basak (racb) wrote :

Setting C.UTF-8 would work internally I agree. But what does this mean for SRUs? Does every affected SRU now have to declare Breaks: python3-minimal (<< whatever)?

Revision history for this message
Benjamin Drung (bdrung) wrote :

That would probably the safest solution (as long as the Breaks is not forgotten in the SRU).

Revision history for this message
Robie Basak (racb) wrote :

Any suggestions on how to identify SRUs that are affected so that we can ensure that the necessary Breaks are in there?

(also, do we know that Breaks will work as needed for an issue in the prerm?)

Revision history for this message
Robie Basak (racb) wrote :

I think it's fair to call this Critical since SRU releases generally are currently blocked by this. Looking at the definition, "Severely affects applications beyond the package responsible for the root cause" also applies.

Changed in python3-defaults (Ubuntu):
importance: High → Critical
Revision history for this message
Benjamin Drung (bdrung) wrote :

Run something like `grep -E 'py3clean|dpkg -L' $prerm` on all prerm files in the existing package.

On my system:
```
$ grep -l -E 'py3clean|dpkg -L' /var/lib/dpkg/info/*.prerm | wc -l
411
```

Revision history for this message
Benjamin Drung (bdrung) wrote :

Good suggestion from the discussion with Ravi: Require all noble SRUs to check if the package is affected by this bug. Add an entry to the SRU template for noble with a test that the submitter has to check.

Revision history for this message
Benjamin Drung (bdrung) wrote :

Proposed upstream fix: https://salsa.debian.org/cpython-team/python3-defaults/-/merge_requests/15

Revision history for this message
Benjamin Drung (bdrung) wrote :

Nice that such a fundamental package has zero tests.

Revision history for this message
Benjamin Drung (bdrung) wrote :

I wrote a small Python script that tries to install a package in a prepared chroot and looks for UnicodeDecodeError in the apt output. I let this script running. Let's see how many affected package can be found.

Revision history for this message
Benjamin Drung (bdrung) wrote :

Current analysis status:

```
$ sqlite3 affected.sqlite3 "SELECT count(package) FROM package_affected where affected is not NULL;"
1322
$ sqlite3 affected.sqlite3 "SELECT package FROM package_affected where affected = 1;"
cinder-api
cinder-backup
cinder-scheduler
cinder-volume
cloud-init
fglrx-pxpress
python3-cinder
python3-rtslib-fb
```

The script will run probably the whole weekend:

```
$ sqlite3 affected.sqlite3 "SELECT avg(processing_duration) FROM package_affected;"
5.08064879165574
$ sqlite3 affected.sqlite3 "SELECT count(package) FROM package_affected where affected is NULL;"
71144
```

Revision history for this message
Matthias Klose (doko) wrote :

the fallback dpkg -L call in dh-python's autoscripts/prerm-py3clean should also set the env var.

Revision history for this message
Benjamin Drung (bdrung) wrote :

First I thought that the fallback dpkg -L call needs to be fixed as well, but then I noticed that the pipe consumers will use the user's locale as well. So that change will probably not be needed.

Revision history for this message
Dave Jones (waveform) wrote :

Having spent some time digging into this (at the request of the SRU team), I'll summarise my findings:

Firstly, the analysis is correct: py3clean is ultimately the issue; packages that both use py3clean and have diversions are affected (diversions are the only circumstances that produce localised output in dpkg-query -L).

Secondly, the proposed fix upstream is also fine, though personally I'd also like to see the stdout decode use errors='replace' (there's no good reason to fail here in the event of dodgy UTF-8 in a translation, for instance). I'll attach debdiffs for noble and oracular to illustrate my intent.

Thirdly, the proposed workaround for affected packages: using Breaks on python3-minimal with a version strictly less than the fixed version also appears to be sufficient (in all the cases I've tested this causes python3-minimal to be at least unpacked prior to the prerm script of the affected package being executed).

So, if the fix for python3-minimal can be uploaded, it simply remains to determine which packages are affected. This is where things get tricky.

Benjamin's efforts are much appreciated here, and will likely be decisive in light of the following: determining which packages use py3clean is relatively simple (pretty much anything that installs a python module). Determining which packages have diversions turns out to be extremely difficult.

It's not enough to detect if a package *itself* uses dpkg-divert. Take the cloud-init case where this was first detected: the postinst calls dpkg-divert but only to *remove* an old diversion. The diversion that actually affects this package comes from usr-merge (because cloud-init still has files under the unmerged paths for various reasons). In essence, any package (foo) can cause a diversion in another package (bar) without the affected package (bar) showing any sign of this in either its source or its binary artefacts. Further, the diversion may or may not affect the package (bar) as the source of the diversion (foo) may be optional to install.

Hence, there are two approaches. The thorough, but likely impractical, approach would be to apply the "Breaks" fix to all packages using py3clean: i.e. all packages that install python modules. Then there's (Benjamin's) empirical approach: attempt to install everything from the former set and see what fails. As noted above, this cannot guarantee correctness as we cannot be certain that all packages that may divert files in our target package are installed, but in practice it's probably (hopefully!) "good enough" given the rarity of diversions and avoids updates to (presumably) several thousand packages.

Revision history for this message
Dave Jones (waveform) wrote :

Attaching debdiff for proposed changes to python3-minimal in oracular

Revision history for this message
Dave Jones (waveform) wrote :

Attaching debdiff for proposed changes to python3-minimal in noble

Ubuntu Foundations Team Bug Bot (crichton)
tags: added: patch
Revision history for this message
Matthias Klose (doko) wrote :

ta, I'll merge that for oracular together with the python 3.12.5 updates

Matthias Klose (doko)
Changed in python3-defaults (Ubuntu):
assignee: nobody → Matthias Klose (doko)
Revision history for this message
Benjamin Drung (bdrung) wrote :

Dave, I included your additional change from oracular.debdiff in https://salsa.debian.org/cpython-team/python3-defaults/-/merge_requests/15

My analysis script is nearing its end. I'll post the results in some hours.

Revision history for this message
Benjamin Drung (bdrung) wrote :
Download full text (3.8 KiB)

Analysis update: There are 72 affected packages so far.

```
$ sqlite3 affected.sqlite3 "SELECT section, COUNT(*) FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) GROUP BY section;"
main|10
multiverse|2
universe|60
```

```
$ sqlite3 affected.sqlite3 "SELECT package, source, section, install_version FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY section, package;"
cloud-init|cloud-init|main|24.1.3-0ubuntu3.3
nvme-stas|nvme-stas|main|2.3.1-1
python3-rtslib-fb|python-rtslib-fb|main|2.1.74-0ubuntu5
targetcli-fb|targetcli-fb|main|1:2.1.53-1ubuntu3
ubiquity|ubiquity|main|24.04.5
ubiquity-frontend-debconf|ubiquity|main|24.04.5
ubiquity-frontend-gtk|ubiquity|main|24.04.5
ubuntu-drivers-common|ubuntu-drivers-common|main|1:0.9.7.6ubuntu3
ubuntu-pro-client|ubuntu-advantage-tools|main|32.3.1~24.04
walinuxagent|walinuxagent|main|2.9.1.1-0ubuntu3
r4d|r4d|multiverse|1.7-4
virtualbox|virtualbox|multiverse|7.0.16-dfsg-2ubuntu1
a2d|a2d|universe|2.0.3-2
apertium-apy|apertium-apy|universe|0.11.7-2.1
blueman|blueman|universe|2.3.5-3build1
borgmatic|borgmatic|universe|1.8.3-1
condor|condor|universe|23.4.0+dfsg-1ubuntu4
cpupower-gui|cpupower-gui|universe|0.7.2-2.1ubuntu2
debomatic|debomatic|universe|0.26-3
dell-recovery|dell-recovery|universe|1.66
deluge-web|deluge|universe|2.1.2~dev0+20240121-1
depthcharge-tools|depthcharge-tools|universe|0.6.2-1
dhcpy6d|dhcpy6d|universe|1.2.3-1
dkimpy-milter|dkimpy-milter|universe|1.2.3-1
fenrir|fenrir|universe|1.9.8-1
galileo|galileo|universe|0.5.1-9.1
glances|glances|universe|3.4.0.3+dfsg-1
graphite-carbon|graphite-carbon|universe|1.1.7-1.1
grokmirror|grokmirror|universe|2.0.11-2
guerillabackup|guerillabackup|universe|0.5.0-1
ionit|ionit|universe|0.5.0-1.1
jupyterhub|jupyterhub|universe|3.0.0+ds1-1
liquidctl|liquidctl|universe|1.13.0-2
localslackirc|localslackirc|universe|1.27-1
logdata-anomaly-miner|logdata-anomaly-miner|universe|2.6.1-1
lomiri|lomiri|universe|0.2.1-10build1
mailman3|mailman3|universe|3.3.8-3.1
mailman3-full|mailman3|universe|
matrix-sydent|matrix-sydent|universe|2.5.1-1.1
mom|mom|universe|0.6.0-2.1
mopidy|mopidy|universe|3.4.2-1
mpd-sima|mpd-sima|universe|0.18.2-1
nordugrid-arc-arex|nordugrid-arc|universe|6.18.0-2.2build3
notus-scanner|notus-scanner|universe|22.6.2-1
nrpe-ng|nrpe-ng|universe|0.2.0-1.1
onionprobe|onionprobe|universe|1.1.2+ds-1
openscap-daemon|openscap-daemon|universe|0.1.10-3.1ubuntu3
ospd-openvas|ospd-openvas|universe|22.6.2-1
pagekite|pagekite|universe|1.5.2.201011-2
pagure|pagure|universe|5.11.3+dfsg-2.1
pagure-ev-server|pagure|universe|5.11.3+dfsg-2.1
pagure-milters|pagure|universe|5.11.3+dfsg-2.1
pglistener|pglistener|universe|5.0-2
policyd-rate-limit|policyd-rate-limit|universe|1.0.1.1-2.1
postfix-mta-sts-resolver|postfix-mta-sts-resolver|universe|1.1.2-1.1
powa-collector|powa-collector|universe|1.2.0-2
prelude-correlator|prelude-correlator|universe|5.2.0-1.1
prometheus-pgbouncer-exporter|prometheus-pgbouncer-exporter|universe|1.7-3
prometheus-xmpp-alerts|prometheus-xmpp-alerts|universe|0.5.3-1.1
pyroman|pyroman|universe|0.6.2-1.1
python3-autopilot|autopilot|universe|1.6.1+...

Read more...

Revision history for this message
Benjamin Drung (bdrung) wrote :
Download full text (5.4 KiB)

Analysis result
===============

Code and resulting database can be found on https://github.com/bdrung/bdrung-scripts/tree/lp-2075337-find-affected

122 packages failed to analyze:

```
$ sqlite3 affected.sqlite3 "SELECT section, count(*) from package_affected WHERE install_affected IS NULL or (remove_affected IS NULL and remove_duration IS NOT NULL) GROUP by section;"
main|6
multiverse|16
restricted|32
universe|68
```

70 source packages are affected:

```
$ sqlite3 affected.sqlite3 "SELECT section, COUNT(DISTINCT source) FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) GROUP BY section;"
main|8
multiverse|2
universe|61
```

```
$ sqlite3 affected.sqlite3 "SELECT source FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) GROUP BY source ORDER BY source;"
a2d
apertium-apy
autopilot
blueman
borgmatic
btchip-python
cloud-init
condor
cpupower-gui
debomatic
dell-recovery
deluge
depthcharge-tools
dhcpy6d
dkimpy-milter
fenrir
galileo
glances
graphite-carbon
grokmirror
guerillabackup
ionit
jupyterhub
liquidctl
localslackirc
logdata-anomaly-miner
lomiri
mailman3
matrix-sydent
mom
mopidy
mpd-sima
nordugrid-arc
notus-scanner
nrpe-ng
nvme-stas
onionprobe
openscap-daemon
ospd-openvas
pagekite
pagure
pglistener
policyd-rate-limit
postfix-mta-sts-resolver
powa-collector
prelude-correlator
prometheus-pgbouncer-exporter
prometheus-xmpp-alerts
pyro5
pyroman
python-rtslib-fb
r4d
razercfg
sgt-launcher
sms4you
supervisor
syncplay
targetcli-fb
timekpr-next
twms
ubiquity
ubuntu-advantage-tools
ubuntu-drivers-common
vanguards
vaultlocker
virtualbox
voctomix-outcasts
walinuxagent
weborf
xdiagnose
```

or 76 binary packages:

```
$ sqlite3 affected.sqlite3 "SELECT section, COUNT(*) FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) GROUP BY section;"
main|10
multiverse|2
universe|64
```

```
$ sqlite3 affected.sqlite3 "SELECT package, source, section, install_version FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY section, package;"
cloud-init|cloud-init|main|24.1.3-0ubuntu3.3
nvme-stas|nvme-stas|main|2.3.1-1
python3-rtslib-fb|python-rtslib-fb|main|2.1.74-0ubuntu5
targetcli-fb|targetcli-fb|main|1:2.1.53-1ubuntu3
ubiquity|ubiquity|main|24.04.5
ubiquity-frontend-debconf|ubiquity|main|24.04.5
ubiquity-frontend-gtk|ubiquity|main|24.04.5
ubuntu-drivers-common|ubuntu-drivers-common|main|1:0.9.7.6ubuntu3
ubuntu-pro-client|ubuntu-advantage-tools|main|32.3.1~24.04
walinuxagent|walinuxagent|main|2.9.1.1-0ubuntu3
r4d|r4d|multiverse|1.7-4
virtualbox|virtualbox|multiverse|7.0.16-dfsg-2ubuntu1
a2d|a2d|universe|2.0.3-2
apertium-apy|apertium-apy|universe|0.11.7-2.1
blueman|blueman|universe|2.3.5-3build1
borgmatic|borgmatic|universe|1.8.3-1
condor|condor|universe|23.4.0+dfsg-1ubuntu4
cpupower-gui|cpupower-gui|universe|0.7.2-2.1ubuntu2
debomatic|debomatic|universe|0.26-3
dell-recovery|dell-recovery|universe|1.66
deluge-web|deluge|universe|2.1.2~dev0+20240121-1
depthcharge-tools|depthcharge-tools|universe|0.6.2-1
dhcpy6d|dhcpy6d|universe|1.2.3-1
dkimpy-milter|dkimpy-milter|uni...

Read more...

Revision history for this message
Robie Basak (racb) wrote :

Thanks all for working on this!

Do we know if releases prior to Noble are affected?

Revision history for this message
Benjamin Drung (bdrung) wrote :

I tested the affected packages on jammy. Result: Only sgt-launcher is affected on jammy.

I'm running a full test on jammy (starting with main).

Revision history for this message
Dave Jones (waveform) wrote :

Attaching debdiffs for sponsorship in noble and jammy.

For the purposes of review, the jammy diff does include one additional change to avoid using shell=True on the invocation of dpkg -L. This can be excised if necessary, but I really didn't want to leave something that ultimately runs as root doing string interpolation and then calling something via the shell, when I don't have to!

Revision history for this message
Dave Jones (waveform) wrote :
Revision history for this message
Dave Jones (waveform) wrote :
Revision history for this message
Benjamin Drung (bdrung) wrote :

Sponsored python3-defaults 3.12.3-0ubuntu2 for noble (after adding the bug reference to the changelog) and uploaded python3-defaults 3.10.6-1~22.04.1 for jammy.

Benjamin Drung (bdrung)
Changed in cloud-init (Ubuntu Jammy):
status: New → Invalid
Changed in ubuntu-advantage-tools (Ubuntu Jammy):
status: New → Invalid
Revision history for this message
Benjamin Drung (bdrung) wrote :
Revision history for this message
Benjamin Drung (bdrung) wrote :

Attached the debdiff for ubuntu-advantage-tools. I did not upload it, because the test TestExportGPGKey.test_export_single_key_from_keyring_dir failed during build.

Revision history for this message
Brian Murray (brian-murray) wrote :

What is the plan for fixing this in Oracular?

Revision history for this message
Benjamin Drung (bdrung) wrote :

For oracular: Matthias uploaded 3.12.5-1:
https://tracker.debian.org/news/1553054/accepted-python3-defaults-3125-1-source-into-unstable/

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

There's a lot of history to follow here. Can someone SRUify the bug description to properly explain the situation? From what I see many packages are affected, but I hope no no-change rebuilds will be required for those as this is python? What is the test case, regression potential? How was this issue introduced? Is it a regression or was this already present in noble GA?

I haven't been following the discussion but I think it's fair to request such information to be visible on the SRU bug.

Dave Jones (waveform)
summary: - py3clean fails when using alternate character set
+ [SRU] py3clean fails when using alternate character set
Revision history for this message
Dave Jones (waveform) wrote :

Marking dh_python as invalid; it was added originally on the theory that we may have needed to patch the prerm fallback but this has thankfully turned out to be unnecessary.

description: updated
Changed in python3-defaults (Ubuntu):
status: New → Confirmed
Changed in python3-defaults (Ubuntu Jammy):
importance: Undecided → Critical
status: New → Confirmed
Changed in python3-defaults (Ubuntu Noble):
importance: Undecided → Critical
status: New → Confirmed
Changed in dh-python (Ubuntu):
status: New → Invalid
Changed in dh-python (Ubuntu Jammy):
status: New → Invalid
Changed in dh-python (Ubuntu Noble):
status: New → Invalid
Benjamin Drung (bdrung)
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Please test proposed package

Hello Robie, or anyone else affected,

Accepted cloud-init into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/24.2-0ubuntu1~24.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-init (Ubuntu Noble):
status: New → Fix Committed
tags: added: verification-needed verification-needed-noble
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Robie, or anyone else affected,

Accepted python3-defaults into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3-defaults/3.12.3-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python3-defaults (Ubuntu Noble):
status: Confirmed → Fix Committed
Changed in python3-defaults (Ubuntu Jammy):
status: Confirmed → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Robie, or anyone else affected,

Accepted python3-defaults into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3-defaults/3.10.6-1~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Benjamin Drung (bdrung) wrote :

On Ubuntu 22.04 (jammy) only packages in universe are affected:

```
$ sqlite3 affected-jammy.sqlite3 "SELECT package, source FROM package_affected WHERE install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY package;"
ganeti-3.0|ganeti
sgt-launcher|sgt-launcher
```

Note: Due to time constraints, 3861/59351 (6 %) universe package haven't been analyzed.

Code and resulting database can be found on https://github.com/bdrung/bdrung-scripts/tree/lp-2075337-find-affected

description: updated
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (python3-defaults/3.10.6-1~22.04.1)
Download full text (8.4 KiB)

All autopkgtests for the newly accepted python3-defaults (3.10.6-1~22.04.1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

adsys/0.14.1~22.04 (arm64)
android-platform-system-core/1:10.0.0+r36-9 (arm64)
apbs/3.0.0+dfsg1-3build2 (amd64, arm64, armhf, s390x)
apparmor/3.0.4-2ubuntu2.3 (arm64)
astropy/5.0.2-1 (amd64)
automake-1.16/1:1.16.5-1.3 (armhf, s390x)
bambam/1.1.2+dfsg-3 (amd64, armhf, s390x)
bdebstrap/0.4.0-1 (s390x)
billiard/3.6.4.0-2 (amd64, ppc64el)
binoculars/0.0.10-1.1 (armhf)
binutils/2.38-4ubuntu2.6 (i386)
boost1.74/1.74.0-14ubuntu3 (arm64, armhf)
booth/1.0-237-gdd88847-4ubuntu2.2 (armhf)
ccdproc/2.3.0-2 (armhf)
ceilometer/2:18.1.0-0ubuntu1 (armhf)
chardet/4.0.0-1 (armhf)
cherrypy3/18.6.1-4 (arm64, armhf)
click-man/0.4.1-2 (armhf)
cloudpickle/2.0.0-1 (amd64, arm64, armhf, ppc64el, s390x)
cluster-glue/1.0.12-20ubuntu3 (amd64, arm64)
cmake-extras/1.6-1 (armhf)
colorclass/2.2.0-2.2 (armhf)
construct/2.10.67+dfsg1-1 (armhf)
crmsh/4.3.1-1ubuntu3 (armhf)
cryptominisat/5.8.0+dfsg1-2 (amd64, arm64, ppc64el, s390x)
dart/6.12.1+dfsg4-11build2 (armhf)
dasbus/unknown (s390x)
deap/1.3.1-3build1 (armhf)
diffoscope/205 (amd64, arm64, armhf, ppc64el, s390x)
dj-database-url/unknown (armhf)
dj-static/0.0.6-6 (armhf)
django-axes/5.24.0-1 (ppc64el)
django-q/1.3.9-3 (armhf)
django-tables/2.4.1-2 (armhf)
djoser/2.1.0-1 (armhf)
dnstwist/0~20220131-1 (amd64, armhf, ppc64el, s390x)
dolfin/2019.2.0~git20210928.3eacdb4-3build1 (amd64, arm64, ppc64el)
dothost/0.2-6 (armhf)
dovecot/1:2.3.16+dfsg1-3ubuntu2.2 (armhf)
dulwich/0.20.31-1.1build1 (s390x)
dulwich/unknown (armhf)
elementpath/2.5.0-1 (armhf)
evolution-data-server/3.44.4-0ubuntu1.1 (armhf)
faiss/1.7.2-5 (armhf)
fakesleep/0.1-2 (armhf)
fast5/0.6.5-6build1 (armhf)
fenics-dolfinx/1:0.3.0-13build1 (armhf)
fetchmail/6.4.27-1 (amd64)
firewalld/1.1.1-1ubuntu1 (arm64)
firewalld/unknown (armhf)
flask-migrate/unknown (s390x)
foo2zjs/20200505dfsg0-2ubuntu2.22.04.1 (armhf)
freeradius/3.0.26~dfsg~git20220223.1.00ed0241fa-0ubuntu3.2 (armhf)
galpy/1.6.0.post0-2build2 (armhf)
gau2grid/unknown (armhf)
gdspy/unknown (armhf)
geneagrapher/1.0c2+git20200719-1 (s390x)
gnome-photos/unknown (s390x)
heartbeat/1:3.0.6-12build1 (armhf)
heat/1:18.0.1-0ubuntu1.1 (armhf)
ikiwiki-hosting/0.20180719-2 (armhf)
ipywidgets/6.0.0-9 (armhf)
jaraco.text/unknown (armhf)
lammps/20220106.git7586adbb6a+ds1-2 (armhf)
ledger-autosync/1.0.3.7.g69f30ea-2 (armhf, s390x)
liblinear/2.3.0+dfsg-5 (armhf)
libproxy/unknown (armhf)
libreoffice/1:7.3.7-0ubuntu0.22.04.5 (arm64, armhf)
libsoup3/3.0.7-0ubuntu1 (s390x)
libvirt-python/unknown (armhf)
linux-aws-6.2/blacklisted (amd64, arm64)
linux-aws-6.5/unknown (arm64)
linux-azure-6.2/blacklisted (amd64, arm64)
linux-gcp-6.2/blacklisted (amd64, arm64)
linux-hwe-5.19/blacklisted (amd64, arm64, armhf, ppc64el, s390x)
linux-hwe-6.2/blacklisted (amd64, arm64, armhf, ppc64el, s390x)
linux-lowlatency-hwe-5.19/blacklisted (amd64, arm64)
linux-lowlatency-hwe-6.2/blacklisted (amd64, arm64)
linux-oem-6.0/blacklisted (amd64)
linux-oracle-6.2/blacklisted (arm64)
llvm-toolchain-12/1:12.0.1-19ubuntu3 (arm64)
mutatormath/3.0.1-1 (armhf)
mypy/0.942-1ubuntu1 (arm...

Read more...

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (python3-defaults/3.12.3-0ubuntu2)
Download full text (11.4 KiB)

All autopkgtests for the newly accepted python3-defaults (3.12.3-0ubuntu2) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

backblaze-b2/1.3.8-6 (arm64)
bitstruct/8.15.1-1build3 (armhf)
boost1.74/1.74.0+ds1-23.1ubuntu3 (arm64)
boost1.83/1.83.0-2.1ubuntu3 (amd64)
cachy/0.3.0-4 (armhf)
check-patroni/unknown (s390x)
cmake/unknown (s390x)
colorspacious/1.1.2-5 (i386)
cov-core/unknown (s390x)
coz-profiler/unknown (s390x)
cppimport/22.08.02-2 (arm64)
crowdsec/1.4.6-6ubuntu0.24.04.1 (armhf)
csound/unknown (s390x)
cython-legacy/0.29.37-2ubuntu1 (s390x)
dh-cargo/unknown (s390x)
dh-ros/0.13.4 (amd64, arm64, armhf, i386, ppc64el, s390x)
django-auth-ldap/4.6.0-1 (armhf)
dolfin/2019.2.0~legacy20240219.1c52e83-1ubuntu4 (s390x)
dulwich/unknown (s390x)
gcc-13/13.2.0-23ubuntu4 (armhf, s390x)
gcc-14/14-20240412-0ubuntu1 (armhf)
gcc-9/9.5.0-6ubuntu2 (armhf)
git-delete-merged-branches/unknown (armhf)
git-filter-repo/2.38.0-2 (s390x)
gitless/0.8.8-4 (armhf)
glueviz/1.17.1+dfsg-1 (armhf)
gtk+2.0/2.24.33-4ubuntu1.1 (armhf)
hatop/0.8.2-2 (armhf)
hplip/3.23.12+dfsg0-0ubuntu5 (armhf)
ikiwiki-hosting/0.20220716-2 (arm64, armhf)
isc-kea/2.4.1-3build3 (armhf)
jc/1.25.1-1 (amd64, arm64, armhf, ppc64el, s390x)
jinja2/3.1.2-1ubuntu1.1 (s390x)
libmediainfo/24.01+dfsg-1build2 (s390x)
libreoffice/4:24.2.5-0ubuntu0.24.04.1 (arm64)
libreoffice/unknown (armhf, s390x)
lilypond/unknown (armhf)
m2crypto/0.40.1-2build2 (arm64, armhf)
mdtraj/1.9.9-1ubuntu2 (arm64)
netplan.io/1.0-2ubuntu1.2 (s390x)
neutron/unknown (s390x)
nibabel/5.2.1-1 (s390x)
npm2deb/0.3.0-12 (arm64)
numpy/1:1.26.4+ds-6ubuntu1 (armhf)
offpunk/2.2-1 (arm64, armhf, ppc64el, s390x)
onioncircuits/0.8.1-1 (armhf)
oscrypto/1.3.0-5 (armhf)
pacemaker/2.1.6-5ubuntu2 (armhf)
palettable/unknown (s390x)
passenger/6.0.17+ds-1ubuntu5 (arm64, s390x)
paste/unknown (armhf)
pastedeploy/3.1-1 (armhf)
pastedeploy/unknown (s390x)
patroni/3.2.2-2 (arm64)
pdb2pqr/3.6.1+dfsg-1 (armhf)
peptidebuilder/unknown (armhf)
petsc/3.19.6+dfsg1-2.2ubuntu2 (armhf)
pgq/unknown (armhf)
pgxnclient/unknown (armhf)
pipenv/2023.12.1+ds-1 (s390x)
podman-compose/1.0.6-1 (s390x)
postfix/3.8.6-1build2 (armhf)
prometheus-xmpp-alerts/unknown (armhf)
pwdsphinx/1.0.18-3 (armhf)
pyacidobasic/unknown (armhf)
pybluez/unknown (armhf)
pydantic/1.10.14-1 (armhf)
pydbus/unknown (armhf)
pydocstyle/unknown (s390x)
pyethash/unknown (armhf)
pyfuse3/3.3.0-0.1 (armhf)
pykeepass/unknown (s390x)
pylabels/unknown (armhf)
pylama/unknown (armhf)
pyninjotiff/0.4.0-3 (s390x)
pynormaliz/unknown (armhf)
pyproject-metadata/unknown (armhf)
pyprojroot/unknown (armhf)
pyrr/0.10.3-6 (i386)
pyspread/unknown (armhf)
pyswarms/unknown (armhf)
pytest-mpi/0.6-5 (armhf)
pytest-testinfra/unknown (s390x)
pytest-tornado/unknown (armhf)
pytest-twisted/1.14.0-3 (armhf)
python-aalib/0.4-3 (s390x)
python-airspeed/unknown (armhf)
python-ansible-compat/4.1.11-1 (s390x)
python-argon2/unknown (armhf)
python-authlib/unknown (armhf)
python-av/11.0.0-4build1 (armhf, s390x)
python-axolotl-curve25519/unknown (armhf)
python-bids-validator/1.14.1-1 (arm64, i386, ppc64el, s390x)
python-blessed/unknown (armhf)
python-bumps/unknown (armhf)
python-bx/unkno...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.