NFS client : permission denied when trying to access subshare, since kernel 4.4.0-31

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

This change was made by a bot.

There's nothing interesting in the logs.

Can you confirm one thing for me - the error you get is "permission denied" and not "operation not permitted"? Those represent different error codes, so I want to be certain which one you're getting. Thanks.

Sorry, i forget to trigger the problem before sending apport-collect ...
I will send the right one and delete the old

When i try to login i got :

Could not chdir to home directory /home/VI/staff/fili/linux : Permission denied
...
-bash: /home/VI/staff/fili/linux/.bash_profile: Permisson denied

But when i try to mount the subshare directly in fstab i got :

mount.nfs: access denied by server while mounting ...

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Still nothing useful there. Our kernels have nfs/sunrpc debugging enabled, so let's try enabling that and see what information we can get. I haven't used this before, so there may be a bit of trial and error involved.

Start by running these commands:

  $ rpcdebug -m rpc -s all
  $ rpcdebug -m nfs -c all

Those very well may require sudo. After that try to mount the nfs shares. I believe the debug output will appear in dmesg, so check there, if not check the journal. Wherever they appear, capture that to a file and attach it to this bug. Let me know if you have questions. Thanks!

Oops, that second command should have been:

  $ rpcdebug -m nfs -s all

You can run the same commands with -c instead of -s to turn off the logging when you're done.

Ok, here the logs after login attempt with kernel 4.4.0-57-generic

Maybe i found something interesting ...

I create a structure of zfs filesystems on the server similar my home path, but with general file permissions ... chown root:root and chmod 775 and mount this on the client under /home/user/staff/fili/data

And i am able to access this directory

When i change the permissions and the owner, chown -R fili and chmod -R 700 /home/user/staff/fili
i can access /home/user/staff/fili but not /home/user/staff/fili/data

...

The logs show that the remote server is responding that access is denied. I have some ideas but I'll need to look at the code to see if they make sense.

I tested something else ...

Now i change only the group permissions ( chgrp staff and chmod 750 ... but the result is the same.
Fristlevel subshare with the need of checking permissions will work, the second level subshare not.

By the way ... one reason why we use a Solaris Fork or FreeBSD as a NFSv4 fileserver is the possibility of using NFS4-ACLs.

So when i change the trivial permissions on the fileserver with chmod the NFS4-ACLs also modified.
and on the client those permission are also active.

Maybe the problem is in the part of the NFS4-ACLs. This could be the reason why not so many users affected ?

On Wed, Dec 14, 2016 at 08:26:24AM -0000, Thomas Fili wrote:
> I tested something else ...
>
> Now i change only the group permissions ( chgrp staff and chmod 750 ... but the result is the same.
> Fristlevel subshare with the need of checking permissions will work, the second level subshare not.
>
> By the way ... one reason why we use a Solaris Fork or FreeBSD as a
> NFSv4 fileserver is the possibility of using NFS4-ACLs.
>
> So when i change the trivial permissions on the fileserver with chmod the NFS4-ACLs also modified.
> and on the client those permission are also active.
>
> Maybe the problem is in the part of the NFS4-ACLs. This could be the
> reason why not so many users affected ?

It's a possibility.

Just to double check - you said the problem first appeared in 4.4.0-31.
The last version we released before that was 4.4.0-28; are you certain
that the problem did not exist there? If so I can concentrate on changes
between those versions. It's clearely not the same problem as the other
two bugs you mention, even though the behavior is similar.

OK, i test to install these old kernels once again to say nothing wrong ...

The Kernel 4.4.0-31 was working definitive for us. The next Kernel 4.4.0-34 also seems to work and 4.4.0.36 too ...

Kernel 4.4.0-38 now seems to be the first kernel having the problem. Strange thing ... as i remember correctly i also tested 4.4.0-34 and 4.4.0.36 ...

Maybe i reboot the computer not before the update to 4.4.0-38 has come, so i noticed so late

To complete the list :

The 4.4 mainline kernel works without problem (4.4.38-040438-generic)

The versions of mainline kernels 4.8.13 and 4.9.0 RC8 not

Please give this kernel a try and let me know if it fixes the problem.

http://people.canonical.com/~sforshee/lp1649292/linux-4.4.0-57.78+lp1649292v201612140942/

Hi,

looks good :)
In this kernel the problem is fixed.

Thank you very much !!!

But for my personal interest, could you explain me in a few words in which part of the kernel this fix works ?

Ah ... to early :(

In the patch i saw "automount" and in fact i configured autofs to mount the nfs shares.

Now i disabled autofs and uncomment the appropriate entries in /etc/fstab and the kernel crash on boot time.

Ok, i can boot without the /etc/fstab entries and mount then the NFS shares without errors ...

I added the logs as an attachment ...

The d_automount is the callback into the filesystem to mount a sub-filesystem upon access to a directory. autofs does use it, but nfs uses it too for subshares even without autofs.

So far I'm not seeing any connection between my patch and the crash, I'm wondering if you've encountered another bug just by coincidence. I don't know the code in question well though so there may be something I'm missing.

The "fix" isn't really a final fix though, just testing an idea I had about what might be causing it. 4.4.0-38 contains a patch (backported from 4.8) which changes the current credentials to those of root before trying to automount a submount, so that a recently added capability check will pass. I guessed that the id changes might be causing kerberos authentication to fail, so the patch makes a copy of the user's credentials but with the required capability instead of using root's creds.

The root of the kernel crash messages looks to be a GPF when reading from an rpc pipe, which has nothing to do with the credentials that I can see. So I'm not sure how the two would be connected.

The fact that it does help tells me that I'm on the right track though.

> 4.4.0-38 contains a patch (backported from 4.8)

Hmm, when i remember correctly, after the first time the permission problem occurs a lot of kernel updates comes in quick succession ... and when i test this updates i had similar crashes on boot time and booting without fstab entries works and also mounting the shares after booting work without a problem.

Meanwhile i switched to the autofs version with some hope ... with no success but staying with this config
At this time i use the latest 4.4 mainline kernel ...

Now i test booting 4.4.0-57 with fstab entries from the offical repo (without your patch) resulting in the same kernel crash !!!

4.4.0-54 do not crash like this but certainly with the permission problem.

Maybe you cant build 4.4.0-54 with your patch for testing?

I'll have another kernel for you to test soon with a very simple fix. In the mean time could you test the 4.9 mainline build from the link below just to confirm that the problem hasn't been fixed upstream?

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/

Strange about the crash, there are no changes between 4.4.0-54 and 4.4.0-57 that look to be related at all. I suggest you open another bug for that issue (let me know the bug number).

I'll make the new test build based on 4.4.0-54.

No changing since 4.9-RC8 ... the kernel boots ... mounting shares from fstab ... but i got permission denied at the second subshare

Thanks. Here's the new test build.

http://people.canonical.com/~sforshee/lp1649292/linux-4.4.0-54.75+lp1649292v201612150908/

Ok,

your kernel 4.4.0.54.75... boots and works with no permission error :)

For the kernel crash i will create a new bug report.

I create a new bug report Bug #1650336 for the crash with the nfs /etc/fstab entry

To keep you updated - I sent the patch to the nfs maintainers, unfortunately the change is not correct for some other system calls. No suggestions though on what would be better, so I'm trying to get more feedback. I will let you know when I have something new to try.

The Bug #1650336 seems to have dependency with the number of nfs shares should be mounted in fstab (from the same server) ...

For your information :

Apart from this kernel crash problem with multible nfs mounts ... the 4.4 mainline kernels, including the 4.4.41 from yesterday seems not to have the permission problem.

Unfortunately we need at least two nfs mounts on our clients so we have to stay on 4.4.0-31 on xenial or 4.2.0-42 on trusty

I am wondering ... why no one else is affected from this problem

Sorry for the lack of progress, I was off for a couple weeks during the holidays and am getting caught up.

Upstream 4.4 doesn't have the permission problem because we've backported some patches from newer kernels to provide some additional features. It's one of those patches that causes the problem.

It is an upstream problem though, so I'm trying to work out a solution with the upstream maintainers. I'll keep you posted.

I finally have something new for you to test. This is for upstream, so I have a 4.10-rc6 build. Once it's fixed upstream we can backport the fix to xenial.

Please let me know whether or not this kernel fixes the permissions problem for you. Thanks!

http://people.canonical.com/~sforshee/lp1649292/linux-4.10-rc6/linux-image-4.10.0-rc6+_4.10.0-rc6+-7_amd64.deb

This looks very good.
Access to the subshares is possible without permission denied :)

One last build to test the backport. Please verify that the problem is fixed in the build below. Thanks!

http://people.canonical.com/~sforshee/lp1649292/linux-4.4.0-63.84+lp1649292v201702141041/

Yes ...

referring to the permission denied when accessing subshares, this kernel build ( 4.4.0-63 ) works fine.

I has to comment out the second share in /etc/fstab or set it to noauto.

I will work now with this kernel on my PC, hoping there is no other problem ;)

And also hoping the other bug will solved too ;)

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag 'verification-needed-yakkety' to 'verification-failed-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Looks good,

the 4.4.0-65.86 from -proposed do not have the bug for us any longer ... great work ... thank you very much.

I changed the tag 'verification-needed-xenial' to 'verification-done-xenial' but i do not run any instance of yakkety, so i can not test it on this ubuntu version.

But for this bug i can also verify a working -proposed kernel on trusty

On Wed, Mar 01, 2017 at 08:29:41AM -0000, Thomas Fili wrote:
> Looks good,
>
> the 4.4.0-65.86 from -proposed do not have the bug for us any longer ...
> great work ... thank you very much.
>
> I changed the tag 'verification-needed-xenial' to 'verification-done-
> xenial' but i do not run any instance of yakkety, so i can not test it
> on this ubuntu version.

If you're able to install a yakkety kernel on a xenial system that would
be sufficient.

Ok, thank you

i added the yakkety-proposed repo to my xenial computer and install the 4.8.0-40.43 kernel
This kernel works also well under xenial :)

I changed the tag 'verification-needed-yakkety' to 'verification-done-yakkety'

On Wed, Mar 01, 2017 at 02:14:16PM -0000, Thomas Fili wrote:
> Ok, thank you
>
> i added the yakkety-proposed repo to my xenial computer and install the 4.8.0-40.43 kernel
> This kernel works also well under xenial :)
>
> I changed the tag 'verification-needed-yakkety' to 'verification-done-
> yakkety'

Thanks!

This bug was fixed in the package linux - 4.10.0-9.11

---------------
linux (4.10.0-9.11) zesty; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1666214

  * linux: disable CONFIG_PCIEPORTBUS in the kernel (LP: #1665404)
    - [Config] CONFIG_PCIEPORTBUS=n for ppc64el

  * linux-lts-xenial 4.4.0-63.84~14.04.2 ADT test failure with linux-lts-xenial
    4.4.0-63.84~14.04.2 (LP: #1664912)
    - SAUCE: apparmor: fix link auditing failure due to, uninitialized var

  * Ubuntu 17.04: "Oops: Exception in kernel mode, sig: 5 [#1]" seen during
    fadump over ssh on Alpine machine. (LP: #1655241)
    - SAUCE: powerpc/fadump: set an upper limit for boot memory size

  * In Ubuntu 17.04 : after reboot getting message in console like Unable to
    open file: /etc/keys/x509_ima.der (-2) (LP: #1656908)
    - SAUCE: ima: Downgrade error to warning

  * NFS client : permission denied when trying to access subshare, since kernel
    4.4.0-31 (LP: #1649292)
    - fs: Better permission checking for submounts

  * Miscellaneous Ubuntu changes
    - SAUCE: (noup) Update spl to 0.6.5.9-1, zfs to 0.6.5.9-2
    - [Config] CONFIG_SCSI_HISI_SAS=m on arm64
    - d-i: Add hisi_sas_v2_hw to scsi-modules
    - d-i: Add hns_enet_drv to nic-modules
    - d-i: Add supporting modules for hns_enet_drv to nic-modules
    - rebase to v4.10

  [ Upstream Kernel Changes ]

  * rebase to v4.10

 -- Tim Gardner <email address hidden> Wed, 15 Feb 2017 11:18:07 -0700

This bug was fixed in the package linux - 4.8.0-40.43

---------------
linux (4.8.0-40.43) yakkety; urgency=low

  * linux: 4.8.0-40.43 -proposed tracker (LP: #1667066)

  [ Andy Whitcroft ]
  * NFS client : permission denied when trying to access subshare, since kernel
    4.4.0-31 (LP: #1649292)
    - fs: Better permission checking for submounts

  * shaking screen (LP: #1651981)
    - drm/radeon: drop verde dpm quirks

  * [0bda:0328] Card reader failed after S3 (LP: #1664809)
    - usb: hub: Wait for connection to be reestablished after port reset

  * linux-lts-xenial 4.4.0-63.84~14.04.2 ADT test failure with linux-lts-xenial
    4.4.0-63.84~14.04.2 (LP: #1664912)
    - SAUCE: apparmor: fix link auditing failure due to, uninitialized var

  * In Ubuntu 17.04 : after reboot getting message in console like Unable to
    open file: /etc/keys/x509_ima.der (-2) (LP: #1656908)
    - SAUCE: ima: Downgrade error to warning

  * 16.04.2: Extra patches for POWER9 (LP: #1664564)
    - powerpc/mm: Fix no execute fault handling on pre-POWER5
    - powerpc/mm: Fix spurrious segfaults on radix with autonuma

  * ibmvscsis: Add SGL LIMIT (LP: #1662551)
    - ibmvscsis: Add SGL limit

  * [Hyper-V] Bug fixes for storvsc (tagged queuing, error conditions)
    (LP: #1663687)
    - scsi: storvsc: Enable tracking of queue depth
    - scsi: storvsc: Remove the restriction on max segment size
    - scsi: storvsc: Enable multi-queue support
    - scsi: storvsc: use tagged SRB requests if supported by the device
    - scsi: storvsc: properly handle SRB_ERROR when sense message is present
    - scsi: storvsc: properly set residual data length on errors

  * Ubuntu16.10-KVM:Big configuration with multiple guests running SRIOV VFs
    caused KVM host hung and all KVM guests down. (LP: #1651248)
    - KVM: PPC: Book 3S: XICS cleanup: remove XICS_RM_REJECT
    - KVM: PPC: Book 3S: XICS: correct the real mode ICP rejecting counter
    - KVM: PPC: Book 3S: XICS: Fix potential issue with duplicate IRQ resends
    - KVM: PPC: Book 3S: XICS: Implement ICS P/Q states
    - KVM: PPC: Book 3S: XICS: Don't lock twice when checking for resend

  * ISST-LTE:pNV: ppc64_cpu command is hung w HDs, SSDs and NVMe (LP: #1662666)
    - blk-mq: Avoid memory reclaim when remapping queues
    - blk-mq: Fix failed allocation path when mapping queues
    - blk-mq: Always schedule hctx->next_cpu

  * systemd-udevd hung in blk_mq_freeze_queue_wait testing unpartitioned NVMe
    drive (LP: #1662673)
    - percpu-refcount: fix reference leak during percpu-atomic transition

  * [Yakkety SRU] Enable KEXEC support in ARM64 kernel (LP: #1662554)
    - [Config] Enable KEXEC support in ARM64.

  * [Hyper-V] Fix ring buffer handling to avoid host throttling (LP: #1661430)
    - Drivers: hv: vmbus: On write cleanup the logic to interrupt the host
    - Drivers: hv: vmbus: On the read path cleanup the logic to interrupt the host
    - Drivers: hv: vmbus: finally fix hv_need_to_signal_on_read()

  * brd module compiled as built-in (LP: #1593293)
    - CONFIG_BLK_DEV_RAM=m

  * regession tests failing after stackprofile test is run (LP: #1661030)
    - SAUCE: fix regression with domain change in compla...

This bug was fixed in the package linux - 4.4.0-65.86

---------------
linux (4.4.0-65.86) xenial; urgency=low

  * linux: 4.4.0-65.86 -proposed tracker (LP: #1667052)

  [ Stefan Bader ]
  * Upgrade Redpine RS9113 driver to support AP mode (LP: #1665211)
    - SAUCE: Redpine driver to support Host AP mode

  * NFS client : permission denied when trying to access subshare, since kernel
    4.4.0-31 (LP: #1649292)
    - fs: Better permission checking for submounts

  * [Hyper-V] SAUCE: pci-hyperv fixes for SR-IOV on Azure (LP: #1665097)
    - SAUCE: PCI: hv: Fix wslot_to_devfn() to fix warnings on device removal
    - SAUCE: pci-hyperv: properly handle pci bus remove
    - SAUCE: pci-hyperv: lock pci bus on device eject

  * [Hyper-V/Azure] Please include Mellanox OFED drivers in Azure kernel and
    image (LP: #1650058)
    - net/mlx4_en: Fix bad WQE issue
    - net/mlx4_core: Fix racy CQ (Completion Queue) free
    - net/mlx4_core: Fix when to save some qp context flags for dynamic VST to VGT
      transitions
    - net/mlx4_core: Avoid command timeouts during VF driver device shutdown

  * Xenial update to v4.4.49 stable release (LP: #1664960)
    - ARC: [arcompact] brown paper bag bug in unaligned access delay slot fixup
    - selinux: fix off-by-one in setprocattr
    - Revert "x86/ioapic: Restore IO-APIC irq_chip retrigger callback"
    - cpumask: use nr_cpumask_bits for parsing functions
    - hns: avoid stack overflow with CONFIG_KASAN
    - ARM: 8643/3: arm/ptrace: Preserve previous registers for short regset write
    - target: Don't BUG_ON during NodeACL dynamic -> explicit conversion
    - target: Use correct SCSI status during EXTENDED_COPY exception
    - target: Fix early transport_generic_handle_tmr abort scenario
    - target: Fix COMPARE_AND_WRITE ref leak for non GOOD status
    - ARM: 8642/1: LPAE: catch pending imprecise abort on unmask
    - mac80211: Fix adding of mesh vendor IEs
    - netvsc: Set maximum GSO size in the right place
    - scsi: zfcp: fix use-after-free by not tracing WKA port open/close on failed
      send
    - scsi: aacraid: Fix INTx/MSI-x issue with older controllers
    - scsi: mpt3sas: disable ASPM for MPI2 controllers
    - xen-netfront: Delete rx_refill_timer in xennet_disconnect_backend()
    - ALSA: seq: Fix race at creating a queue
    - ALSA: seq: Don't handle loop timeout at snd_seq_pool_done()
    - drm/i915: fix use-after-free in page_flip_completed()
    - Linux 4.4.49

  * NFS client : kernel 4.4.0-57 crash with nfsv4 enries in /etc/fstab
    (LP: #1650336)
    - SUNRPC: fix refcounting problems with auth_gss messages.

  * [0bda:0328] Card reader failed after S3 (LP: #1664809)
    - usb: hub: Wait for connection to be reestablished after port reset

  * linux-lts-xenial 4.4.0-63.84~14.04.2 ADT test failure with linux-lts-xenial
    4.4.0-63.84~14.04.2 (LP: #1664912)
    - SAUCE: apparmor: fix link auditing failure due to, uninitialized var

  * ibmvscsis: Add SGL LIMIT (LP: #1662551)
    - ibmvscsis: Add SGL limit

  * [Hyper-V] Bug fixes for storvsc (tagged queuing, error conditions)
    (LP: #1663687)
    - scsi: storvsc: Enable tracking of queue depth
    - scsi: storvsc: Remove the ...