Received: by 2002:a25:824b:0:0:0:0:0 with SMTP id d11csp1827988ybn; Thu, 26 Sep 2019 02:51:40 -0700 (PDT) X-Google-Smtp-Source: APXvYqz0i6svlX7nOuj2mKnFvr6mO6tNQ66Y1xUE2SvUM5WGT5UmIY+uc/XJBl+CSNpAQVSlC3CV X-Received: by 2002:a17:906:6445:: with SMTP id l5mr2258788ejn.12.1569491499981; Thu, 26 Sep 2019 02:51:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1569491499; cv=none; d=google.com; s=arc-20160816; b=UIZbxWnpukVhsDp1B5N1t5WFEe3M+z2/bOVBl5rX4hdAQfSARlJOobpxK/3VXgfUz7 BXRiwGjXhu1rk4Z3fffcFM86AOQEGK0w6vzYDtkAqynzMSl7ZTd8RXaPAHuvobnNZzgK aznBpAEEGLL/mgWHdlu5yJOGjzQBQHD6A+TzuSZ4mpiUgdgghPnTrKr3z9X1MxjA976u Dq3pGuuczhYydEW+8PmsN0EmLT1KT2pEqOPmev0dmSbgqWC0FyVx316gxmTU7q2y/9V8 QHurHxa0gE+L5bePlo5Svbr1mefX/uY6hyWUDweFlCPr/l+SXaTUzk/MlGwxErjV9r/X ME3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=/+3Q5JqAnWLC0kB4ZN2DWSe1cAaMlOsC1gqX6w2DM1Q=; b=qbBCo9YCIp74VqC4PM+917gZZZLKGs7/aZpXkie6rS1mrqJTIRqqyE88DkmWAECoxU K5SbADRwz/SjsoarA7jQvXCfowIkGcRf5083TNU7uzVLQxqV23we0QLGi0hSvsdo70b0 XN+OWjtrTK/Rhu3Ln16I36ZLYxYZ0s5cyRhoOfgo5vkigZez/U3A/Dv5sl+zk4Y0nUnu S8uEz0tajjGp+iocnDQs1D0xH9ozefD5pOTDjMnthJ9iFBbzbhRhdZxwFNu1uofKCKg3 8HKaZqhuwKL1EH3xq7tothL3xkYNNT9J8xWwwl96N14wKmUni8qjrts1T2ay9rJO5Xf1 V16A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Gx3Y+EhD; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y11si684365eje.365.2019.09.26.02.51.14; Thu, 26 Sep 2019 02:51:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Gx3Y+EhD; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2438107AbfIYSoT (ORCPT + 99 others); Wed, 25 Sep 2019 14:44:19 -0400 Received: from mail-io1-f65.google.com ([209.85.166.65]:36029 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2437582AbfIYSoT (ORCPT ); Wed, 25 Sep 2019 14:44:19 -0400 Received: by mail-io1-f65.google.com with SMTP id b136so1509859iof.3 for ; Wed, 25 Sep 2019 11:44:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=/+3Q5JqAnWLC0kB4ZN2DWSe1cAaMlOsC1gqX6w2DM1Q=; b=Gx3Y+EhDOe5PN3szgmEbU+l7+rAv4bhiVul1h0XkctBE7sldKUd5Gj90L1CW8YcqP3 x9hwv3jDOGwlmff6hFiBmLDzCfJeQwaead4NZBDd2PypKTDrvh8QiC8u/Y7ml9F/BAXM Hv7e/H+FICvMc97PA9rCcWRkg186YNpNJScUIn2V7WY275F+5oGGaiYqEl1W/Cv1oHEt akzGH8UUskrZyDO7W2hnmA1lTA2GzaobYpi+V1QfnlWXk9POQ8EyHpK5uc1FQ7xhueP5 xpq05KaNl/7yfaDKr6LmG8EyCCnQCwqIkZZqTd4GKT1WihDPFbtNTLGwj4PHw2pvmKIP lTbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=/+3Q5JqAnWLC0kB4ZN2DWSe1cAaMlOsC1gqX6w2DM1Q=; b=RP5NuKlYShS9YUJvDXVa3kvtdyu3Zo/vXBp5XFm8jV7bSqPlqisQwArLiBspKw13iI arY9qYGjNZscjo3SrW/PF9XupTBaa57whW4rEr+yA24OhUzBFuUa+zFtD/kTemcuoDJd O3vWwbSxjZvtLNs5slDKwNNn/+8zMlJO6zzI3CZLfx9Jg6nUtpaMN1ksSKPT62lwQrRr fWbbyKVeZ5gbOwYx1rLqPIRuHrAwenrpeZVE4FgGOmD6jA5LF43Zx23tNd9ke2/CwLW+ HDN7bzUzQ6gO6bPk21QhUJFOsMmlyiqQRrWTvU6Ao5wXlSdrKwABzKB6pItjiI9gPFxz QKmQ== X-Gm-Message-State: APjAAAVbcuto2zPFMZ2WweR38qBna05bKyVf/CclZQxaNI8bMvI5VD03 2+3E5jY4NQTyeMYSnJrugMn6cOoj+5Ilgfwrh+PuzIiC X-Received: by 2002:a6b:6a01:: with SMTP id x1mr854266iog.119.1569437057705; Wed, 25 Sep 2019 11:44:17 -0700 (PDT) MIME-Version: 1.0 References: <20190925164831.GA9366@fieldses.org> <57192382-86BE-4878-9AE0-B22833D56367@oracle.com> In-Reply-To: <57192382-86BE-4878-9AE0-B22833D56367@oracle.com> From: Kevin Vasko Date: Wed, 25 Sep 2019 13:44:06 -0500 Message-ID: Subject: Re: NFSv4 client locks up on larger writes with Kerberos enabled To: Chuck Lever Cc: Bruce Fields , Linux NFS Mailing List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Appreciate the information. I have tried different clients. The Centos and Ubuntu clients exhibit the same issue. I also did packet tracing. On small files the client will go through its "WRITE..." packets... and then will issue COMMIT and then do the CLOSE. On larger files it will do all of its WRITE... packets and then will issue COMMIT and then it will never do a CLOSE. I'm assuming whats happening is the client is waiting for the NFS server to authenticate with the Kerberos server to make sure the write is okay, however I assume it never does. I would like to try and figure out how to see that "authentication" packet to kerberos from the server (if there is one). Just for giggles, I just tried it with NFSv3 and there is a slight change and gives a little more information (albeit, what I kind of already assumed). Mounted the shared with krb5p, and vers=3D3. I write a file with strace -f dd if=3D/dev/zero of=3D/data/test.img bs=3D10M count=3D1 This is successful. No issues. Just like NFSv4 Then I change nothing but the size of the file being written (just like with NFSv4)=E2=80=A6however with NFSv3 it will halt for ~1 minute and then spit out a permission denied even though I just wrote out the SAME exact file in the SAME exact location (literally overwriting the file) a few seconds prior. The only difference in NFSv3 and NFSv4 is that NFSv3 fails with a permission issue, whereas NFSv4 never returns. strace -f dd if=3D/dev/zero of=3D/data/test.img bs=3D1000M count=3D1 This command with NFSv3, takes ~1min and will return file permissions error (even though it just wrote it and can run the above command again and it will finish with no errors). This makes sense in that my initial suspicion is that the Dell Unity 300 isn=E2=80=99t authenticating properly with larger writes (for whatever reason) with Kerberos. This is also why I don=E2=80=99t see the problem whe= n I disable Kerberos on the Unity 300 system. When should the NFS server be sending a packet to the Kerberos server to validate the write? Or should it be? I did do packet capture on the Unity box but I don=E2=80=99t see anything really useful regarding Kerberos authentication. What should I be looking for in the packet traces to look for the authentication packets? Debugging commands: $mount ...snip=E2=80=A6 10.75.37.2:/ds1 on /data type nfs (rw,relatime,vers=3D3,rsize=3D65536,wsize=3D65536,namlen=3D255,hard,proto= =3Dtcp,timeo=3D600,retrans=3D2,sec=3Dkrb5p,mountaddr=3D10.75.37.2,mountvers= =3D3,mountport=3D1234,mountproto=3Dudp,local_lock=3Dnone,addr=3D10.75.37.2) $ time dd if=3D/dev/zero of=3D/data/test.img bs=3D10M count=3D1 1+0 records in 1+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.112561 s, 93.2 MB/s real 0m0.188s user 0m0.000s sys 0m0.087s $ time dd if=3D/dev/zero of=3D/data/test.img bs=3D1000M count=3D1 dd: closing output file '/data/test.img': Permission denied real 1m9.093s user 0m0.000s sys 0m1.124s $ strace -f dd if=3D/dev/zero of=3D/data/test.img bs=3D10M count=3D1 execve("/bin/dd", ["dd", "if=3D/dev/zero", "of=3D/data/test.img", "bs=3D10M", "count=3D1"], 0x7ffec9914328 /* 30 vars */) =3D 0 brk(NULL) =3D 0x5595514e6000 access("/etc/ld.so.nohwcap", F_OK) =3D -1 ENOENT (No such file or dire= ctory) access("/etc/ld.so.preload", R_OK) =3D -1 ENOENT (No such file or dire= ctory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) =3D 3 fstat(3, {st_mode=3DS_IFREG|0644, st_size=3D125550, ...}) =3D 0 mmap(NULL, 125550, PROT_READ, MAP_PRIVATE, 3, 0) =3D 0x7f5e4e5d2000 close(3) =3D 0 access("/etc/ld.so.nohwcap", F_OK) =3D -1 ENOENT (No such file or dire= ctory) openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) =3D= 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\34\2\0\0\0\0\0"= ..., 832) =3D 832 fstat(3, {st_mode=3DS_IFREG|0755, st_size=3D2030544, ...}) =3D 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D 0x7f5e4e5d0000 mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =3D 0x7f5e4dfd9000 mprotect(0x7f5e4e1c0000, 2097152, PROT_NONE) =3D 0 mmap(0x7f5e4e3c0000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) =3D 0x7f5e4e3c0000 mmap(0x7f5e4e3c6000, 15072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) =3D 0x7f5e4e3c6000 close(3) =3D 0 arch_prctl(ARCH_SET_FS, 0x7f5e4e5d1540) =3D 0 mprotect(0x7f5e4e3c0000, 16384, PROT_READ) =3D 0 mprotect(0x55954fd03000, 4096, PROT_READ) =3D 0 mprotect(0x7f5e4e5f1000, 4096, PROT_READ) =3D 0 munmap(0x7f5e4e5d2000, 125550) =3D 0 rt_sigaction(SIGINT, NULL, {sa_handler=3DSIG_DFL, sa_mask=3D[], sa_flags=3D= 0}, 8) =3D 0 rt_sigaction(SIGUSR1, {sa_handler=3D0x55954faf60e0, sa_mask=3D[INT USR1], sa_flags=3DSA_RESTORER, sa_restorer=3D0x7f5e4e017f20}, NULL, 8) =3D 0 rt_sigaction(SIGINT, {sa_handler=3D0x55954faf60d0, sa_mask=3D[INT USR1], sa_flags=3DSA_RESTORER|SA_NODEFER|SA_RESETHAND, a_restorer=3D0x7f5e4e017f20}, NULL, 8) =3D 0 brk(NULL) =3D 0x5595514e6000 brk(0x559551507000) =3D 0x559551507000 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) =3D = 3 fstat(3, {st_mode=3DS_IFREG|0644, st_size=3D11603728, ...}) =3D 0 mmap(NULL, 11603728, PROT_READ, MAP_PRIVATE, 3, 0) =3D 0x7f5e4d4c8000 close(3) =3D 0 openat(AT_FDCWD, "/dev/zero", O_RDONLY) =3D 3 dup2(3, 0) =3D 0 close(3) =3D 0 lseek(0, 0, SEEK_CUR) =3D 0 openat(AT_FDCWD, "/data/test.img", O_WRONLY|O_CREAT|O_TRUNC, 0666) =3D 3 dup2(3, 1) =3D 1 close(3) =3D 0 mmap(NULL, 10498048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D 0x7f5e4cac5000 read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".= .., 10485760) =3D 10485760 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"= ..., 10485760) =3D 10485760 close(0) =3D 0 close(1) =3D 0 openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) =3D = 0 fstat(0, {st_mode=3DS_IFREG|0644, st_size=3D2995, ...}) =3D 0 read(0, "# Locale name alias data base.\n#"..., 4096) =3D 2995 read(0, "", 4096) =3D 0 close(0) =3D 0 openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D 0 fstat(0, {st_mode=3DS_IFREG|0644, st_size=3D578, ...}) =3D 0 mmap(NULL, 578, PROT_READ, MAP_PRIVATE, 0, 0) =3D 0x7f5e4e5f0000 close(0) =3D 0 write(2, "1+0 records in\n1+0 records out\n", 311+0 records in 1+0 records out ) =3D 31 write(2, "10485760 bytes (10 MB, 10 MiB) c"..., 6010485760 bytes (10 MB, 10 MiB) copied, 0.105735 s, 99.2 MB/s) =3D 60 write(2, "\n", 1 ) =3D 1 close(2) =3D 0 exit_group(0) =3D ? +++ exited with 0 +++ $ strace -f dd if=3D/dev/zero of=3D/data/test.img bs=3D1000M count=3D1 execve("/bin/dd", ["dd", "if=3D/dev/zero", "of=3D/data/test.img", "bs=3D1000M", "count=3D1"], 0x7fff46397c58 /* 30 vars */) =3D 0 brk(NULL) =3D 0x55a768f09000 access("/etc/ld.so.nohwcap", F_OK) =3D -1 ENOENT (No such file or dire= ctory) access("/etc/ld.so.preload", R_OK) =3D -1 ENOENT (No such file or dire= ctory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) =3D 3 fstat(3, {st_mode=3DS_IFREG|0644, st_size=3D125550, ...}) =3D 0 mmap(NULL, 125550, PROT_READ, MAP_PRIVATE, 3, 0) =3D 0x7f604f357000 close(3) =3D 0 access("/etc/ld.so.nohwcap", F_OK) =3D -1 ENOENT (No such file or dire= ctory) openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) =3D= 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\34\2\0\0\0\0\0"= ..., 832) =3D 832 fstat(3, {st_mode=3DS_IFREG|0755, st_size=3D2030544, ...}) =3D 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D 0x7f604f355000 mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =3D 0x7f604ed5e000 mprotect(0x7f604ef45000, 2097152, PROT_NONE) =3D 0 mmap(0x7f604f145000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) =3D 0x7f604f145000 mmap(0x7f604f14b000, 15072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) =3D 0x7f604f14b000 close(3) =3D 0 arch_prctl(ARCH_SET_FS, 0x7f604f356540) =3D 0 mprotect(0x7f604f145000, 16384, PROT_READ) =3D 0 mprotect(0x55a767990000, 4096, PROT_READ) =3D 0 mprotect(0x7f604f376000, 4096, PROT_READ) =3D 0 munmap(0x7f604f357000, 125550) =3D 0 rt_sigaction(SIGINT, NULL, {sa_handler=3DSIG_DFL, sa_mask=3D[], sa_flags=3D= 0}, 8) =3D 0 rt_sigaction(SIGUSR1, {sa_handler=3D0x55a7677830e0, sa_mask=3D[INT USR1], sa_flags=3DSA_RESTORER, sa_restorer=3D0x7f604ed9cf20}, NULL, 8) =3D 0 rt_sigaction(SIGINT, {sa_handler=3D0x55a7677830d0, sa_mask=3D[INT USR1], sa_flags=3DSA_RESTORER|SA_NODEFER|SA_RESETHAND, a_restorer=3D0x7f604ed9cf20}, NULL, 8) =3D 0 brk(NULL) =3D 0x55a768f09000 brk(0x55a768f2a000) =3D 0x55a768f2a000 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) =3D = 3 fstat(3, {st_mode=3DS_IFREG|0644, st_size=3D11603728, ...}) =3D 0 mmap(NULL, 11603728, PROT_READ, MAP_PRIVATE, 3, 0) =3D 0x7f604e24d000 close(3) =3D 0 openat(AT_FDCWD, "/dev/zero", O_RDONLY) =3D 3 dup2(3, 0) =3D 0 close(3) =3D 0 lseek(0, 0, SEEK_CUR) =3D 0 openat(AT_FDCWD, "/data/test.img", O_WRONLY|O_CREAT|O_TRUNC, 0666) =3D 3 dup2(3, 1) =3D 1 close(3) =3D 0 mmap(NULL, 1048588288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D 0x7f600fa4a000 read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".= .., 1048576000) =3D 1048576000 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"= ..., 1048576000) =3D 1048576000 close(0) =3D 0 close(1) =3D -1 EACCES (Permission denied) openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) =3D = 0 fstat(0, {st_mode=3DS_IFREG|0644, st_size=3D2995, ...}) =3D 0 read(0, "# Locale name alias data base.\n#"..., 4096) =3D 2995 read(0, "", 4096) =3D 0 close(0) =3D 0 openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D 0 fstat(0, {st_mode=3DS_IFREG|0644, st_size=3D578, ...}) =3D 0 mmap(NULL, 578, PROT_READ, MAP_PRIVATE, 0, 0) =3D 0x7f604f375000 close(0) =3D 0 write(2, "dd: ", 4dd: ) =3D 4 write(2, "closing output file '/data/test."..., 36closing output file '/data/test.img') =3D 36 openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory) write(2, ": Permission denied", 19: Permission denied) =3D 19 write(2, "\n", 1 ) =3D 1 close(2) =3D 0 exit_group(1) =3D ? +++ exited with 1 +++ On Wed, Sep 25, 2019 at 12:06 PM Chuck Lever wrote= : > > > > > On Sep 25, 2019, at 9:48 AM, bfields@fieldses.org wrote: > > > > On Wed, Sep 18, 2019 at 06:36:13PM -0500, Kevin Vasko wrote: > >> We have a new Dell EMC Unity 300 acting as NAS Server that is > >> presenting a NFSv4 NFS Share. Our clients are mostly Ubuntu 18.04.3 > >> but issue is also present on CentOS 7.6 systems. We have been > >> struggling with this issue for over a week now and not sure how to > >> resolve it. > >> > >> > >> > >> We are having trouble with NFS Clients completing their writes to the > >> Dell EMC Unity 300 NFS Server when Kerberos is enabled on the NFS > >> Share. I created the NFS Share on the U300, associated it with our > >> FreeIPA (Kerberos/LDAP server) and everything shows successful. > > > > Troubleshooting ideas off the top of my head: > > > > It might be worth trying some other client versions if it's not hard. > > > > It'd be interesting to know what's happening on the network.... > > Unfortunately big krb5p writes won't be fun to try to capture and > > examine. > > Wireshark is supposed to have a mechanism for giving it the keys > so that captured GSS data can be decrypted. I've never gotten it > to work, but I didn't try hard. Should be appropriately documented. > > > > Maybe some network or rpc-level statistics would help show if > > there are an unusual number of retries or failures. > > > -- > Chuck Lever > > >