gzip -9n sometimes generates a different output file on different architectures

Status changed to 'Confirmed' because the bug affects multiple users.

Confirmed here. For some reason the compressed manpages are different on amd64 vs. i386, even though the contents are identical. We should get this fixed in SRU so that libpam-modules is usable for multiarch.

The manpages are being compressed with the correct options to be idempotent (-9nf), the uncompressed content is identical,
and the compressed files are the same size; yet the .gz files don't match between amd64 and i386. I don't know what's going on here.

None of the other manpages in the package are affected.

Now if I compare with the *armel* package, /usr/share/man/man8/pam_mkhomedir.8.gz is different from *both* amd64 and i386, and pam_shells.8.gz matches the i386 one.

Recompressing the amd64 pam_shells.8.gz with gzip -9nf on amd64 gives the same file as on i386. Recompressing pam_mkhomedir with the amd64 gzip gives the original file. Recompressing pam_mkhomedir with the armel gzip gives the version of the file found on amd64 and i386.

And gzip hasn't changed in the archive since August 2010, and zlib hasn't changed since Mar 2011.

The broken builds date from 2011-08-18 on allspice (amd64) and aizoaceae (armel). Same version of debhelper was used in all cases. No version information for gzip, zlib is visible in the log (since they're pre-installed), but there should be no differences there anyway.

Hello NightRider, or anyone else affected,

Accepted pam into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

$ diff -ur i386/ amd64/
<snip>
Binary files i386/usr/share/man/man8/pam_shells.8.gz and amd64/usr/share/man/man8/pam_shells.8.gz differ
$

Still broken. Still no idea why.

Extra data points here:

The amd64 misbuild produced the same output both times, and was on the same host (allspice).
The armel misbuild only occurred the first time (on a babbage), but was fine the second time (on a panda).

Given that this seems to be reproducible (and potentially hardware-specific), this might be debuggable, with access to the hardware that exhibits the issue.

To double-check that this wasn't buildd chroot-specific, I grabbed the current oneiric-amd64 chroot from launchpad, and replicated an identical lp-buildd build on my local laptop. In my local tests, the manpages were compressed correctly (where "correctly" means "the same as i386"), so this really does look like it might be hardware specific.

I got this dialog ->
   ROOT:
   Could not install '/var/cache/apt/archives/libpam-modules_1.1.3-2ubuntu1_i386.deb'
   The upgrade will continue but the '/var/cache/apt/archives/libpam-modules_1.1.3-2ubuntu1_i386.deb' package may not be in a working state. Please consider submitting a bug report about it.
   './usr/share/man/man8/pam_shells.8.gz' is different from the same file on the system

and now I have this dialog on screen ->
   ROOT:
   Could not install the upgrades
   The upgrade has aborted. Your system could be in an unusable state. A recovery will run now (dpkg --configure -a).

This doesn't look good ... :-(

Should I let this thing run the recovery? Or is that going to leave my system unusable ??

On Sat, Oct 22, 2011 at 03:09:33PM -0000, SwaJime wrote:
> I got this dialog ->
> ROOT:
> Could not install '/var/cache/apt/archives/libpam-modules_1.1.3-2ubuntu1_i386.deb'
> The upgrade will continue but the '/var/cache/apt/archives/libpam-modules_1.1.3-2ubuntu1_i386.deb' package may not be in a working state. Please consider submitting a bug report about it.
> './usr/share/man/man8/pam_shells.8.gz' is different from the same file on the system

> and now I have this dialog on screen ->
> ROOT:
> Could not install the upgrades
> The upgrade has aborted. Your system could be in an unusable state. A recovery will run now (dpkg --configure -a).

> This doesn't look good ... :-(

You'll need to remove the i386 version of the package with 'dpkg --remove
libpam-modules:i386', then run 'dpkg --configure -a'. You can reinstall
libpam-modules:i386 once this bug has been fixed.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

I tried that and this is what came up:

root@desktop:~# dpkg --remove libpam-modules:i386
dpkg: warning: there's no installed package matching libpam-modules:i386

> I tried that and this is what came up:

> root@desktop:~# dpkg --remove libpam-modules:i386
> dpkg: warning: there's no installed package matching libpam-modules:i386

Did the upgrade complete successfully?

It said it couldn't ... and after I hit OK to "recover" nothing happened.
I shut down and restarted my computer later and it seems to be ok.

Would the "recovery" have undone any of the upgrades?

One thing I see is that ie6 is no longer working ... and I tried to run aptitude upgrade:

root@desktop:~# aptitude upgrade
Resolving dependencies...
Unable to resolve dependencies for the upgrade: no solution found.
Unable to safely resolve dependencies, try running with --full-resolver.

root@desktop:~# aptitude upgrade --full-resolver
The following NEW packages will be installed:
  flashplugin-downloader{ab} libasound2-plugins{a} libjack-jackd2-0{ab} libmtp-common{ab} libmtp-runtime{a}
  libmtp9{a} wine1.3{ab}
The following packages will be upgraded:
  audacious-plugins banshee banshee-extension-soundmenu flashplugin-installer wine
5 packages upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 17.8 MB/18.0 MB of archives. After unpacking 113 MB will be used.
The following packages have unmet dependencies:
  wine1.3: Conflicts: wine1.2 but 1.2.3-0ubuntu1 is installed.
  libjack0: Conflicts: libjack-0.116 which is a virtual package.
  libjack-jackd2-0: Conflicts: libjack-0.116 which is a virtual package.
                    Conflicts: libjack0 but 1:0.121.0+svn4469-2ubuntu2 is installed.
  flashplugin-downloader: Conflicts: flashplugin-nonfree (< 11.0.1.152ubuntu1) but 11.0.1.152ubuntu0.11.04.1 is installed.
  libmtp-common: Breaks: libmtp8 (<= 1.0.6-6) but 1.0.6-2 is installed.
  flashplugin-nonfree: Conflicts: flashplugin-nonfree which is a virtual package.
open: 36; closed: 1085; defer: 8; conflict: 18 .The following actions will resolve these dependencies:

       Remove the following packages:
1) appmenu-qt
2) fbreader
3) ffado-mixer-qt4
4) fglrx
5) fglrx-amdcccle
6) flashplugin-installer
7) flashplugin-nonfree
8) genpo
9) hydrogen
10) ia32-libs-multiarch
11) libacl1
12) libasound2
13) libasyncns0
14) libatk1.0-0
15) libattr1
16) libaudio2
17) libavahi-client3
18) libavahi-common3
19) libc6
20) libcairo-gobject2
21) libcairo2
22) libcomerr2
23) libcups2
24) libcupsimage2
25) libcurl3
26) libdatrie1
27) libdb5.1
28) libdbus-1-3 ...

Also my menu is gone, so I can't find my applications. It seems that the menu for firefox has replaced it.

I accepted that "solution" and now I have no wine.

I was afraid of that. Now my computer no longer boots up to a login
screen. The word ubuntu is displayed with some running dots, and then
instead of a login screen the monitor complains that it has "no input
signal".

Now what?

:-(

On 10/23/11, SwaJime <email address hidden> wrote:
> I accepted that "solution" and now I have no wine.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/871083
>
> Title:
> package libpam-modules 1.1.3-2ubuntu1 failed to install/upgrade:
> ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is different from
> the same file on the system
>
> Status in “pam” package in Ubuntu:
> Fix Committed
> Status in “pam” source package in Oneiric:
> Fix Committed
>
> Bug description:
> SRU justification:
> libpam-modules is not multiarch-coinstallable because of unexplained
> misgeneration of compressed manpages on the buildds in a previous run. This
> makes it impossible to use pam authentication for foreign-arch software due
> to the uninstallable modules, and also causes unpleasant upgrade issues.
>
> Test case:
> 1) on amd64, enable armel in addition to i386 as a foreign architecture by
> running:
> echo 'foreign-architecture armel' >>
> /etc/dpkg/dpkg.cfg.d/multiarch-armel
> echo 'deb [arch=armel] http://ports.ubuntu.com/ubuntu-ports/ oneiric
> main' >> /etc/apt/sources.list
> apt-get update
> 2) try to install libpam-modules for the other two architectures by
> running 'apt-get install libpam-modules:i386 libpam-modules:armel'
> 3) confirm that both packages fail to install.
> 4) enable oneiric-proposed (including for armel above).
> 5) try to install libpam-modules for all three architectures by running
> 'apt-get install libpam-modules libpam-modules:i386 libpam-modules:armel'
> 6) confirm that the packages install successfully.
>
> error while updating
>
> ProblemType: Package
> DistroRelease: Ubuntu 11.10
> Package: libpam-modules 1.1.3-2ubuntu1
> ProcVersionSignature: Ubuntu 2.6.38-11.50-generic 2.6.38.8
> Uname: Linux 2.6.38-11-generic x86_64
> NonfreeKernelModules: wl
> ApportVersion: 1.23-0ubuntu2
> Architecture: amd64
> Date: Sat Oct 8 18:57:52 2011
> ErrorMessage: ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is
> different from the same file on the system
> InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64
> (20110427.1)
> SourcePackage: pam
> Title: package libpam-modules 1.1.3-2ubuntu1 failed to install/upgrade:
> ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is different from the
> same file on the system
> UpgradeStatus: Upgraded to oneiric on 2011-10-09 (0 days ago)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/pam/+bug/871083/+subscriptions
>

--
John Wesley Simpson
SwaJime's Cove℠
www.swajime.com

I reinstalled fglrx ... now I get a login screen.
But when I log in I get 'Failed to load session "ubuntu"' and the only
button says 'Log Out"

On 10/23/11, John Hibbs <email address hidden> wrote:
> I was afraid of that. Now my computer no longer boots up to a login
> screen. The word ubuntu is displayed with some running dots, and then
> instead of a login screen the monitor complains that it has "no input
> signal".
>
> Now what?
>
> :-(
>
> On 10/23/11, SwaJime <email address hidden> wrote:
>> I accepted that "solution" and now I have no wine.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/871083
>>
>> Title:
>> package libpam-modules 1.1.3-2ubuntu1 failed to install/upgrade:
>> ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is different from
>> the same file on the system
>>
>> Status in “pam” package in Ubuntu:
>> Fix Committed
>> Status in “pam” source package in Oneiric:
>> Fix Committed
>>
>> Bug description:
>> SRU justification:
>> libpam-modules is not multiarch-coinstallable because of unexplained
>> misgeneration of compressed manpages on the buildds in a previous run.
>> This
>> makes it impossible to use pam authentication for foreign-arch software
>> due
>> to the uninstallable modules, and also causes unpleasant upgrade issues.
>>
>> Test case:
>> 1) on amd64, enable armel in addition to i386 as a foreign architecture
>> by
>> running:
>> echo 'foreign-architecture armel' >>
>> /etc/dpkg/dpkg.cfg.d/multiarch-armel
>> echo 'deb [arch=armel] http://ports.ubuntu.com/ubuntu-ports/ oneiric
>> main' >> /etc/apt/sources.list
>> apt-get update
>> 2) try to install libpam-modules for the other two architectures by
>> running 'apt-get install libpam-modules:i386 libpam-modules:armel'
>> 3) confirm that both packages fail to install.
>> 4) enable oneiric-proposed (including for armel above).
>> 5) try to install libpam-modules for all three architectures by running
>> 'apt-get install libpam-modules libpam-modules:i386 libpam-modules:armel'
>> 6) confirm that the packages install successfully.
>>
>> error while updating
>>
>> ProblemType: Package
>> DistroRelease: Ubuntu 11.10
>> Package: libpam-modules 1.1.3-2ubuntu1
>> ProcVersionSignature: Ubuntu 2.6.38-11.50-generic 2.6.38.8
>> Uname: Linux 2.6.38-11-generic x86_64
>> NonfreeKernelModules: wl
>> ApportVersion: 1.23-0ubuntu2
>> Architecture: amd64
>> Date: Sat Oct 8 18:57:52 2011
>> ErrorMessage: ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is
>> different from the same file on the system
>> InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64
>> (20110427.1)
>> SourcePackage: pam
>> Title: package libpam-modules 1.1.3-2ubuntu1 failed to install/upgrade:
>> ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is different from
>> the
>> same file on the system
>> UpgradeStatus: Upgraded to oneiric on 2011-10-09 (0 days ago)
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/pam/+bug/871083/+subscriptions
>>
>
>
> --
> John Wesley Simpson
> SwaJime's Cove℠
> ...

Apparently the solution offered by aptitude was not a particularly good one .... it seems item #186 in the list was a bad choice.

grrrrrr

aptitude is not the recommended interface for commandline-based upgrades; it's recommended that you use apt-get instead. And aptitude is known to not work right with multiarch packages (which is what libpam-modules:i386 is). I'm sorry, but it sounds like aptitude has made a real mess of your system.

You may be able to fix this by running 'sudo apt-get install ubuntu-desktop'. But if that doesn't solve it, I think you'll want to take this question to one of the support forums. (ubuntuforums.org, askubuntu.com, etc)

I've been using aptitude for years without a problem, and the system upgrade
was not done with aptitude.
The upgrade was started from a dialog that kept popping up asking me to
upgrade the system.
That upgrade stopped and supposedly ran a "recovery", although I did not see
anything further happen.
I ran aptitude because that's what I've always used in the past.

I've reinstalled gnome-panel and upuntu-desktop and fglrx. I don't know
what else might be broken, or even how to tell, except that everything
related to the wine has become inoperative.

I've opened a bug report for that at
https://bugs.launchpad.net/ubuntu/+bug/880640

On Mon, Oct 24, 2011 at 12:30 PM, Steve Langasek <
<email address hidden>> wrote:

> aptitude is not the recommended interface for commandline-based
> upgrades; it's recommended that you use apt-get instead. And aptitude
> is known to not work right with multiarch packages (which is what
> libpam-modules:i386 is). I'm sorry, but it sounds like aptitude has
> made a real mess of your system.
>
> You may be able to fix this by running 'sudo apt-get install ubuntu-
> desktop'. But if that doesn't solve it, I think you'll want to take
> this question to one of the support forums. (ubuntuforums.org,
> askubuntu.com, etc)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/871083
>
> Title:
> package libpam-modules 1.1.3-2ubuntu1 failed to install/upgrade:
> ErrorMessage: './usr/share/man/man8/pam_shells.8.gz' is different from
> the same file on the system
>
> Status in “pam” package in Ubuntu:
> Fix Committed
> Status in “pam” source package in Oneiric:
> Fix Committed
>
> Bug description:
> SRU justification:
> libpam-modules is not multiarch-coinstallable because of unexplained
> misgeneration of compressed manpages on the buildds in a previous run. This
> makes it impossible to use pam authentication for foreign-arch software due
> to the uninstallable modules, and also causes unpleasant upgrade issues.
>
> Test case:
> 1) on amd64, enable armel in addition to i386 as a foreign architecture by
> running:
> echo 'foreign-architecture armel' >>
> /etc/dpkg/dpkg.cfg.d/multiarch-armel
> echo 'deb [arch=armel] http://ports.ubuntu.com/ubuntu-ports/ oneiric
> main' >> /etc/apt/sources.list
> apt-get update
> 2) try to install libpam-modules for the other two architectures by
> running 'apt-get install libpam-modules:i386 libpam-modules:armel'
> 3) confirm that both packages fail to install.
> 4) enable oneiric-proposed (including for armel above).
> 5) try to install libpam-modules for all three architectures by running
> 'apt-get install libpam-modules libpam-modules:i386 libpam-modules:armel'
> 6) confirm that the packages install successfully.
>
> error while updating
>
> ProblemType: Package
> DistroRelease: Ubuntu 11.10
> Package: libpam-modules 1.1.3-2ubuntu1
> ProcVersionSignature: Ubuntu 2.6.38-11.50-generic 2.6.38.8
> Uname: Linux 2.6.38-11-generic x86_64
> NonfreeKernelModules: wl
> ApportVersion: 1.23-0ubuntu2
> Architecture: amd64
> Date: Sat Oct 8 18:57:52 2011
> ErrorMessage: ErrorMessage: './u...

Is this because of the same gzip bug as in bug 889303 and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522 ?

Status changed to 'Confirmed' because the bug affects multiple users.

See duplicate bug 889303, we applied a workaround to libtasn1-3 to unbreak upgrades:

libtasn1-3 (2.10-1ubuntu1) precise; urgency=low

  * debian/rules: Disable compression of NEWS file for now to unbreak
    upgrades. Debugging the underlying gzip bug is quite hard and will take
    more time than a few hours. (LP: #889303)

 -- Martin Pitt <email address hidden> Mon, 14 Nov 2011 08:59:19 +0100

Actually, we should keep the libtasn1-3 task open to revert the workaround once gzip is fixed. Sorry for the spam.

Another workaround was uploaded for clutter-1.0: https://launchpad.net/ubuntu/+source/clutter-1.0/1.8.2-2ubuntu1

Adding a task to remember reverting it once gzip is fixed.

This bug was fixed in the package gzip - 1.4-1ubuntu2

---------------
gzip (1.4-1ubuntu2) precise; urgency=low

  * debian/patches/zeroify-buffers.diff: clear gzip buffers between
    files. Thanks to Cyril Brulebois <email address hidden>.
    Closes: #647522, LP: #871083.
 -- Steve Langasek <email address hidden> Wed, 08 Feb 2012 23:59:13 +0000

This bug was fixed in the package pam - 1.1.3-7ubuntu2

---------------
pam (1.1.3-7ubuntu2) precise; urgency=low

  * No-change rebuild with gzip 1.4-1ubuntu2 to get multiarch-clean
    compression of manpages. LP: #871083.
 -- Steve Langasek <email address hidden> Wed, 08 Feb 2012 17:15:39 -0800

Dropped the workaround in gtk+2.0 in bzr. Not urgent, dropping milestone.

Hello NightRider, or anyone else affected,

Accepted gzip into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

This bug was fixed in the package gtk+2.0 - 2.24.10-0ubuntu3

---------------
gtk+2.0 (2.24.10-0ubuntu3) precise; urgency=low

  * debian/patches/044_grips.patch:
    - dropped, our default theme has no grip and the patch leaded to issues
      with some applications like libreoffice lp: #749986
  * debian/control.in: update gir depends

  [ Martin Pitt ]
  * debian/rules: Revert forced non-compression of doc files, gzip has been
    fixed now. (LP: #871083)
 -- Sebastien Bacher <email address hidden> Thu, 23 Feb 2012 21:53:41 +0100

There's been a security update of pam recently that was built on a different amd64 buildd that apparently wasn't affected by the issue. And since gzip is now in -proposed, any new non-security uploads will be built using it. So I think we can probably consider this resolved for pam in oneiric.

Because reproducing this bug is dependent on filesystem order across multiple buildds, testing the fix in SRU is non-trivial. However, the same patch applied to precise has now resulted in all mismatched gzip-compressed docs being fixed after a rebuild (about 5 other packages were affected).

I think that, plus the fact that every subsequent SRU to oneiric has been using this version and there have been no complaints, is sufficient validation and the SRU should be promoted. Marking 'verification-done'.

When publishing to -updates, please note that this should also be published to -security in coordination with the security team, so that subsequent security updates don't risk regressing multiarch support.

Security team, I'll copy this to -updates now. Are you ok with copying to -security? I don't see a blocker, but you might have some special procedures there, or might require an USN. Want me to do the copying, or want to do yourself?

This bug was fixed in the package gzip - 1.3.12-9ubuntu1.2

---------------
gzip (1.3.12-9ubuntu1.2) oneiric-proposed; urgency=low

  * clear gzip buffers between files. Thanks to Cyril Brulebois
    <email address hidden>. Closes: #647522, LP: #871083.
 -- Steve Langasek <email address hidden> Fri, 10 Feb 2012 17:46:56 -0800

I discussed this with Steve and then I discussed with Marc last week and it is safe to copy to security. I did that just now.

I have a feeling that the very same bug affects python3.2 on Ubuntu 12.04. I have looked at the source code of the zlib module shipped with python and while it's not exactly the same as gzip it seems to lack the relevant memset to clear uninitialized garbage.

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release