Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <3447df77-1b2f-6d36-0516-3ae7267ab509@genome.arizona.edu>
 <20191023171523.GA18802@fieldses.org> <b6248a82-1f1a-7329-5ee0-6e026f6db697@genome.arizona.edu>
 <YTBPR01MB2845B12E9C59F837FE35F40DDD650@YTBPR01MB2845.CANPRD01.PROD.OUTLOOK.COM>
 <82ee292f-f126-9e9f-d023-deb72d1a3971@genome.arizona.edu> <1079a074-7580-e257-8b52-6e48f8822176@genome.arizona.edu>
In-Reply-To: <1079a074-7580-e257-8b52-6e48f8822176@genome.arizona.edu>
From:   Olga Kornievskaia <aglo@umich.edu>
Date:   Tue, 5 Nov 2019 11:24:01 -0500
Message-ID: <CAN-5tyExtO_HRGYrqq7UTObrHNsTp7UqwW=Kg4CFM3q-OnaUiQ@mail.gmail.com>
Subject: Re: NFS hangs on one interface
To:     Chandler <admin@genome.arizona.edu>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

It's too hard to read this tcpdump-style network trace with multiple
nfs streams (a full .cap file would be much better) (internals of the
packets are hidden).

Some things that stick out. If you are doing a v4.0 mount, it
typically would start with a SETCLIENTID. Yours starts with a
PUTROOTFH which means you already have a 4.0 mount going to this
server. "cat /proc/fs/nfsfs/server" would show you mounts to that
server. If you are not expecting that you already had an existing 4.0
mount (ie., your "mount" command doesn't show that server mounted),
then things have gone wrong already and you have a stuck mount which
might be interfering with further mounts.

Are you experiencing issues with a fresh boot ? do you have an
ability/luxury to reboot the client machine?

Your problem description is confusing. Your last network trace is
about a failing v4.0 mount. Your initial description is talking about
mounting with "vers=3" or "vers=2". So is the problem with a specific
nfs version or is the problem with mounting over 10GB interface with
any NFS versions?

You can also turn on rpcdebug messages (if your client machine isn't
getting a lot of NFS traffic) but given your trace I see multiple
streams so you'll have to dig thru lots of output to follow your own
NFS operations.

On Mon, Nov 4, 2019 at 7:29 PM Chandler <admin@genome.arizona.edu> wrote:
>
> Any ideas what's going on here?
> Thanks