Jaunty i386 popen() misbehaves on x86_64 kernel 2.6.26

Bug #339743 reported by Anders Kaseorg
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fedora
Fix Released
Medium
glibc (Ubuntu)
Invalid
Undecided
Unassigned
Intrepid
Invalid
Undecided
Unassigned
Jaunty
Invalid
Undecided
Unassigned
linux (Ubuntu)
Fix Released
Medium
Andy Whitcroft
Intrepid
Fix Released
Medium
Stefan Bader
Jaunty
Fix Released
Medium
Andy Whitcroft
linux-2.6 (Debian)
Fix Released
Unknown

Bug Description

SRU justification:

Impact: A bug in the syscall implementation can cause incorrect behavior when trying to use syscalls that are not implemented.

Fix: Cherry pick from upstream to correctly return an error code in that situation.

Testcase: see test below.

---

There is something very wrong with the popen() implementation in Jaunty on i386. It works fine when run on the Jaunty kernel, but when run on the Lenny 2.6.26-1-amd64 kernel, it misbehaves as follows. The output of the subcommand, instead of being fed to the pipe, is fed directly to stdout; the pipe either returns errors or hangs forever.

This causes weird errors in many programs running in a Jaunty i386 chroot on our Lenny build server.

Attached is a small 5-line test program. Run
  gcc -static popen-test.c -o popen-test
and then copy popen-test to a Lenny machine to see the problem.

Correct output:
  $ uname -r
  2.6.28-8-generic
  $ ./popen-test
  popen returned 0x1ec6400
  ferror returned 0
  fread returned 19
  ferror returned 0
  pclose returned 0
Incorrect output:
  $ uname -r
  2.6.26-1-amd64
  $ ./popen-test
  popen returned 0x935f688
  ferror returned 0
  SHOULD NOT DISPLAY

This has been reproduced independently by two of us with two different Jaunty i386 installs and two different Lenny installs.

Revision history for this message
In , Vitaly (vitaly-redhat-bugs) wrote :

+++ This bug was initially created as a clone of Bug #481682 +++

Created an attachment (id=330051)
proposed fix

in kernel-2.6.18-128.el5 and prior versions, arch/x86_64/ia32/ia32entry.S hunk of the linux-2.6-misc-utrace-update.patch contains incorrect optimization.
As result, out-of-table 32-bit syscalls on the x86_64 kernel do not return ENOSYS (unless the caller is being ptraced).

For example, glibc-2.9+ popen() goes mad when pipe2 syscall returns its number 331 instead of fail with ENOSYS.
As result, FC10+ 32-bit processes on RHEL5 x86_64 kernel break once popen(3) is called.

--- Additional comment from <email address hidden> on 2009-01-27 04:29:59 EDT ---

The issue was found while running Fedora 10 containers on an RHEL5+OpenVZ kernel.

Relative OpenVZ bug: http://bugzilla.openvz.org/show_bug.cgi?id=1150

--- Additional comment from <email address hidden> on 2009-02-06 20:59:52 EDT ---

Created an attachment (id=331182)
test case source

Simple test case, compile with -m32 and run on x86-64 kernel.

The RHEL5 code is the same (broken) as upstream. I'll fix it upstream and then RHEL5 should backport the change so it continues to match upstream.

--- Additional comment from <email address hidden> on 2009-02-06 21:02:48 EDT ---

Created an attachment (id=331183)
test case source

Test case fixed to exit 0 for correct and nonzero for bug.

Also, I forgot to note that to reproduce you have to have auditd disabled:
/sbin/chkconfig auditd off; reboot

--- Additional comment from <email address hidden> on 2009-02-06 21:34:13 EDT ---

Created an attachment (id=331187)
fix posted upstream

Wait to see if upstream takes this as is, but almost certain they will.
With s,/x86/,/x86_64/,g this same patch applies to RHEL5 fine.

--- Additional comment from <email address hidden> on 2009-02-07 04:57:47 EDT ---

> The RHEL5 code is the same (broken) as upstream.

RHEL5 is not broken, I wasn't able to reproduce it. As I understood, this is due to this check:

        testl $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP),threadinfo_flags(%r10)
        jnz ia32_tracesys

And we have TIF_SYSCALL_AUDIT set.

Also, it's possible to make the patch shorter:

--- a/arch/x86/ia32/ia32entry.S
+++ b/arch/x86/ia32/ia32entry.S
@@ -417,6 +417,7 @@ ENTRY(ia32_syscall)
        GET_THREAD_INFO(%r10)
        orl $TS_COMPAT,TI_status(%r10)
        testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags(%r10)
+ movq $-ENOSYS,RAX-ARGOFFSET(%rsp)
        jnz ia32_tracesys
 ia32_do_syscall:
        cmpl $(IA32_NR_syscalls-1),%eax

I've tested it yesterday, but didn't post upstream yet.

--- Additional comment from <email address hidden> on 2009-02-07 17:21:41 EDT ---

It is broken. See comment#3 on how to reproduce it.
Shorter patch does not mean shorter code path, which is what matters.
Anyway, I've already posted upstream.

--- Additional comment from <email address hidden> on 2009-02-07 17:25:55 EDT ---

Upstream fix was merged: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c09249f8d1b84344eca882547afdbffee8c09d14

Revision history for this message
In , Roland (roland-redhat-bugs) wrote :

I don't know what the point of this bug report is. The fix is already in upstream, and rawhide rebases from upstream daily.

Revision history for this message
In , Chuck (chuck-redhat-bugs) wrote :

I think you want to file this bug against Fedora 10 ??

Revision history for this message
In , Vitaly (vitaly-redhat-bugs) wrote :

(In reply to comment #2)
> I think you want to file this bug against Fedora 10 ??

Yes. I don't know if this will be backported to 2.6.27.x

Revision history for this message
In , Chuck (chuck-redhat-bugs) wrote :

Fixed in 2.6.27.18-170.2.28 and will send to -stable.

Revision history for this message
In , Fedora (fedora-redhat-bugs) wrote :

kernel-2.6.27.19-170.2.35.fc10 has been pushed to the Fedora 10 testing repository. If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with
 su -c 'yum --enablerepo=updates-testing update kernel'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2009-2003

Revision history for this message
In , Fedora (fedora-redhat-bugs) wrote :

kernel-2.6.27.19-170.2.35.fc10 has been pushed to the Fedora 10 stable repository. If problems still persist, please make note of it in this bug report.

Revision history for this message
Anders Kaseorg (andersk) wrote :
Revision history for this message
Evan Broder (broder) wrote :

I would be the second person who tested this on a second Jaunty install and a second Lenny install

Changed in glibc:
status: New → Confirmed
Revision history for this message
Anders Kaseorg (andersk) wrote :

I reproduced this with a vanilla v2.6.26 kernel (on x86_64 with defconfig). I used git bisect to find the first commit on which the problem is exhibited:

commit 8ab32bb89b5b9bf06147c31947eba65f0f21c3c0
Author: Roland McGrath <email address hidden>
Date: Sun Mar 16 21:57:41 2008 -0700

    x86: ia32 ptrace vs -ENOSYS

    When we're stopped at syscall entry tracing, ptrace can change the %eax
    value from -ENOSYS to something else. If no system call is actually made
    because the syscall number (now in orig_eax) is bad, then the %eax value
    set by ptrace should be returned to the user. But, instead it gets reset
    to -ENOSYS again. This is a regression from the native 32-bit kernel.

    This change fixes it by leaving the return value alone after entry tracing.

    Signed-off-by: Roland McGrath <email address hidden>
    Signed-off-by: Ingo Molnar <email address hidden>

Revision history for this message
Anders Kaseorg (andersk) wrote :

By defconfig, I meant defconfig plus CONFIG_IA32_EMULATION=y (without which it is not possible to run the i386 binary); that is the only option I changed.

Revision history for this message
Anders Kaseorg (andersk) wrote :

And the first commit after v2.6.26 on which the problem is NOT exhibited is:

commit ed8cae8ba01348bfd83333f4648dd807b04d7f08
Author: Ulrich Drepper <email address hidden>
Date: Wed Jul 23 21:29:30 2008 -0700

    flag parameters: pipe

    This patch introduces the new syscall pipe2 which is like pipe but it also
    takes an additional parameter which takes a flag value. This patch implements
    the handling of O_CLOEXEC for the flag. I did not add support for the new
    syscall for the architectures which have a special sys_pipe implementation. I
    think the maintainers of those archs have the chance to go with the unified
    implementation but that's up to them.

    The implementation introduces do_pipe_flags. I did that instead of changing
    all callers of do_pipe because some of the callers are written in assembler.
    I would probably screw up changing the assembly code. To avoid breaking code
    do_pipe is now a small wrapper around do_pipe_flags. Once all callers are
    changed over to do_pipe_flags the old do_pipe function can be removed.

Revision history for this message
Anders Kaseorg (andersk) wrote :

It seems that __have_pipe2 is being incorrectly set to 1 when running under kernels inside this range (8ab32bb..ed8cae8). Before 8ab32bb, __have_pipe2 is set to -1 and the program works correctly; starting with ed8cae8, pipe2() is actually implemented so the program works correctly.

Revision history for this message
Anders Kaseorg (andersk) wrote :

Fascinating. The problem is that pipe2() is not actually returning -1 and setting errno = ENOSYS like it should. Instead, it is returning 331, which is __NR_pipe2, and leaving errno == 0.

This affects every x86_64 kernel from v2.6.26 (8ab32bb) through v2.6.29-rc3, and was fixed by

commit c09249f8d1b84344eca882547afdbffee8c09d14
Author: Roland McGrath <email address hidden>
Date: Fri Feb 6 18:15:18 2009 -0800

    x86-64: fix int $0x80 -ENOSYS return

    One of my past fixes to this code introduced a different new bug.
    When using 32-bit "int $0x80" entry for a bogus syscall number,
    the return value is not correctly set to -ENOSYS. This only happens
    when neither syscall-audit nor syscall tracing is enabled (i.e., never
    seen if auditd ever started).

    The fix makes the int $0x80 path match the sysenter and syscall paths.

    Reported-by: Dmitry V. Levin <email address hidden>
    Signed-off-by: Roland McGrath <email address hidden>

Revision history for this message
Anders Kaseorg (andersk) wrote :

To fix this on the kernel side, please add commit c09249f8d1b84344eca882547afdbffee8c09d14 to the Intrepid and Jaunty kernels.

It may be possible for glibc to work around this problem on unpatched kernels, though it would not be very pretty.

Revision history for this message
Andy Whitcroft (apw) wrote : Re: [Bug 339743] [NEW] Jaunty i386 popen() misbehaves on x86_64 kernel 2.6.26

> Correct output:
> $ uname -r
> 2.6.28-8-generic
> $ ./popen-test
> popen returned 0x1ec6400
> ferror returned 0
> fread returned 19
> ferror returned 0
> pclose returned 0
> Incorrect output:
> $ uname -r
> 2.6.26-1-amd64
> $ ./popen-test
> popen returned 0x935f688
> ferror returned 0
> SHOULD NOT DISPLAY

I am confused. Your own output above shows that this works fine on the
Ubuntu Jaunty kernel. We don't need to patch Jaunty kernels to get the
right result?

Revision history for this message
Andy Whitcroft (apw) wrote :

It is possible this is triggered by the fact that we do indeed default to running with auditing. That would make the problem moot for a default Intrepid and Jaunty install. I would need a complelling use case to support and SRU of this change given the default install is unaffected.

Revision history for this message
Anders Kaseorg (andersk) wrote :

Kernels starting with 2.6.27 do not have that particular problem with popen() because they implement the pipe2() system call, so popen() does not tickle the missing syscall detection bug. But the attached source can be used to reproduce the missing syscall detection bug directly, by displaying the return value of syscall(6666666).

There are likely to be more and more problems arising from this bug as new syscalls get implemented.

To reproduce, statically compile this on an Intrepid or Jaunty i386 system:
  gcc -static bad-syscall-test.c -o bad-syscall-test
and then run the resulting binary on an Intrepid or Jaunty amd64 system.

Correct output:
  $ uname -rm
  2.6.24-etchnhalf.1-amd64 x86_64
  $ ./bad-syscall-test
  syscall(6666666) returned -1
  errno is 38 (Function not implemented)
Incorrect output:
  $ uname -rm
  2.6.28-8-generic x86_64
  $ ./bad-syscall-test
  syscall(6666666) returned 6666666
  errno is 0 (Success)

Changed in linux:
importance: Undecided → Medium
status: New → Triaged
Changed in linux-2.6:
status: Unknown → New
Revision history for this message
Andy Whitcroft (apw) wrote :

It seems that the only sensible fix is to fix the kernel. If this will affect new system calls then this does seem like something worth fixing correctly.

Changed in linux:
assignee: nobody → apw
status: Triaged → In Progress
Changed in glibc:
status: Confirmed → Invalid
Revision history for this message
Andy Whitcroft (apw) wrote :

@Anders -- I have pulled down that kernel fix and applied it to Jaunty and Intrepid, could you test the kernels at the URL below and confirm you are happy that they fix the issue to your satisfaction. Please report any testing back here. Kernels are at the URLs below:

    http://people.ubuntu.com/~apw/lp339743-intrepid/
    http://people.ubuntu.com/~apw/lp339743-jaunty/

Changed in linux:
status: In Progress → Incomplete
Revision history for this message
Anders Kaseorg (andersk) wrote :

Thanks. I tested the Jaunty amd64 kernel using the same bad-syscall-test binary; it works correctly and fixes the problem. I will test the Intrepid kernel soon.

Revision history for this message
Anders Kaseorg (andersk) wrote :

Yep, the Intrepid amd64 kernel also fixes the problem.

Changed in linux:
status: Incomplete → In Progress
Changed in openvz-kernel:
status: Unknown → Confirmed
Revision history for this message
Tim Gardner (timg-tpi) wrote :
Changed in linux:
milestone: none → ubuntu-9.04-beta
status: In Progress → Fix Committed
Changed in linux (Ubuntu Intrepid):
assignee: nobody → stefan-bader-canonical
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.28-10.32

---------------
linux (2.6.28-10.32) jaunty; urgency=low

  [ Amit Kucheria ]

  * Delete prepare-ppa-source script

  [ Andy Isaacson ]

  * SAUCE: FSAM7400: select CHECK_SIGNATURE
  * SAUCE: LIRC_PVR150: depends on VIDEO_IVTV
    - LP: #341477

  [ Ayaz Abdulla ]

  * SAUCE: forcedeth: msi interrupt fix
    - LP: #288281

  [ Brad Figg ]

  * Updating armel configs to remove PREEMPT

  [ Catalin Marinas ]

  * Fix the VFP handling on the Feroceon CPU

  [ Huaxu Wan ]

  * SAUCE: (drop after 2.6.28) [Jaunty] iwlagn: fix iwlagn DMA mapping
    direction

  [ Ike Panhc ]

  * squashfs: correct misspelling
    - LP: #322306

  [ Theodore Ts'o ]

  * SAUCE: (drop after 2.6.28) ext4: add EXT4_IOC_ALLOC_DA_BLKS ioctl
  * SAUCE: (drop after 2.6.28) ext4: Automatically allocate delay allocated
    blocks on close
  * SAUCE: (drop after 2.6.28) ext4: Automatically allocate delay allocated
    blocks on rename
    - LP: #317781

  [ Tyler Hicks ]

  * SAUCE: (drop after 2.6.28) eCryptfs: Don't encrypt file key with
    filename key
    - LP: #342128

  [ Upstream Kernel Changes ]

  * ALS: hda - Add support of iMac 24 Aluminium
  * USB: fix broken OTG makefile reference
  * ALSA: hda - add another MacBook Pro 3,1 SSID
  * ALSA: hda - Add model entry for HP dv4
  * x86-64: fix int $0x80 -ENOSYS return
    - LP: #339743

 -- Tim Gardner <email address hidden> Thu, 12 Mar 2009 19:16:07 -0600

Changed in linux:
status: Fix Committed → Fix Released
Matthias Klose (doko)
Changed in glibc (Ubuntu Intrepid):
status: New → Invalid
Stefan Bader (smb)
description: updated
Changed in linux:
status: In Progress → Fix Committed
Andy Whitcroft (apw)
Changed in linux:
importance: Undecided → Medium
Changed in openvz-kernel:
status: Confirmed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted linux into intrepid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
Revision history for this message
Steve Beattie (sbeattie) wrote :

I have reproduced the bad return value of int $0x080 calls on the amd64 kernels in intrepid-updates, currently 2.6.27-14.37, and confirmed that the kernels in intrepid-proposed, version 2.6.27-14.36 (needing to be updated), fixes the issue. I've added a slightly modified testcase to the lp:qa-regression-testing bzr tree (I invoke int $0x080 directly via inline asm rather than call syscall(2) so that binaries can be built directly on amd64). Marking verification-done.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.27-14.41

---------------
linux (2.6.27-14.41) intrepid-proposed; urgency=low

  [ Stefan Bader ]

  * Revert "SAUCE: input: Blacklist digitizers from joydev.c"
    - LP: #300143

linux (2.6.27-14.40) intrepid-proposed; urgency=low

  [ Amit Kucheria ]

  * Disable DEVKMEM for all archs on Intrepid
    - LP: #354221
  * SAUCE: Quirk for BT USB device on MacbookPro to be reset before use
    - LP: #332443

  [ Andy Isaacson ]

  * LIRC_PVR150: depends on VIDEO_IVTV
    - LP: #341477
  * SAUCE: FSAM7400: select CHECK_SIGNATURE
    - LP: #341712

  [ Andy Whitcroft ]

  * SAUCE: hotkey quirks for various Zepto Znote and Fujitsu Amilo laptops
    - LP: #330259
  * SAUCE: unusual devs: add an entry for the ScanLogic SL11R-IDE 0.78
    - LP: #336189

  [ Anton Veretenenko ]

  * SAUCE: sony-laptop: add support for Sony Vaio FW series function/media
    keys
    - LP: #307592

  [ Ayaz Abdulla ]

  * SAUCE: forcedeth: msi interrupt fix
    - LP: #288281

  [ Chuck Short ]

  * SAUCE: [USB] Unusual Device support for Gold MP3 Player Energy
    - LP: #125250

  [ Ike Panhc ]

  * squashfs: correct misspelling
    - LP: #322306
  * SAUCE: Fixing symbol name in HECI module
    - LP: #336549
  * Copy header files for various kernel media driver
    - LP: #322732

  [ Stefan Bader ]

  * SAUCE: vgacon: Return the upper half of 512 character fonts
    - LP: #355057
  * SAUCE: input: Blacklist digitizers from joydev.c
    - LP: #300143

  [ Upstream Kernel Changes ]

  * libata: make sure port is thawed when skipping resets
    - LP: #269652
  * x86-64: fix int $0x80 -ENOSYS return
    - LP: #339743
  * rt2x00: Fix race conditions in flag handling
    - LP: #258985
  * USB: cdc-acm: Add another conexant modem to the quirks
    - LP: #323829
  * Bluetooth: Add fine grained mem_flags usage to btusb driver
    - LP: #268502
  * Bluetooth: Handle bulk URBs in btusb driver from notify callback
    - LP: #268502
  * Bluetooth: Submit bulk URBs along with interrupt URBs
    - LP: #268502
  * hwmon: (abituguru3) Match partial DMI board name strings
    - LP: #298798
  * x86: mtrr: don't modify RdDram/WrDram bits of fixed MTRRs
    - LP: #292619
  * sis190: add identifier for Atheros AR8021 PHY
    - LP: #247889
  * ath9k: implement IO serialization
    - LP: #373034
  * ath9k: AR9280 PCI devices must serialize IO as well
    - LP: #373034
  * acer-wmi: fix regression in backlight detection
    - LP: #333386

 -- Stefan Bader <email address hidden> Wed, 26 Aug 2009 11:48:11 +0200

Changed in linux (Ubuntu Intrepid):
status: Fix Committed → Fix Released
Revision history for this message
Arie Skliarouk (skliarie) wrote :

I just checked sources of karmic and don't see there any prebuilt openvz enabled kernels. I thought it was this bug that was preventing the openvz-enablde kernel to be built and put into the repository.

Changed in linux-2.6 (Debian):
status: New → Fix Released
Changed in fedora:
importance: Unknown → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.