Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Rick Macklem <rmacklem@uoguelph.ca>
To:     Trond Myklebust <trondmy@hammerspace.com>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: weird Linux client behaviour when there are multiple concurrent
 CB_RECALLs
Thread-Topic: weird Linux client behaviour when there are multiple concurrent
 CB_RECALLs
Thread-Index: AQHXOXfnjixeaFEWH0C9YjMT3U0+MarFfYOAgABHYb8=
Date:   Sun, 25 Apr 2021 21:43:24 +0000
Message-ID: <YQXPR0101MB096801BCED45A39CB05CA884DD439@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
References: <YQXPR0101MB0968124A7F9024769EC90F23DD439@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>,<9dfaa112431882110d6795f93f52d7561b466b80.camel@hammerspace.com>
In-Reply-To: <9dfaa112431882110d6795f93f52d7561b466b80.camel@hammerspace.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: a377d0af-4a9c-46c1-11c6-08d90833273f
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2021 21:43:24.3445
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: F/vzvzp7xQxu1Ps0tDglh0fLJoBIobZnX1CewallSkW5GzD+hQ4WUg8ydLHUwuzfwmNwRXfaTg6sw3DPC+lCtg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR0101MB2248
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Thanks Trond. I upgrade my kernel one of these days
and test it.

rick

________________________________________
From: Trond Myklebust <trondmy@hammerspace.com>
Sent: Sunday, April 25, 2021 1:27 PM
To: linux-nfs@vger.kernel.org; Rick Macklem
Subject: Re: weird Linux client behaviour when there are multiple concurren=
t CB_RECALLs

CAUTION: This email originated from outside of the University of Guelph. Do=
 not click links or open attachments unless you recognize the sender and kn=
ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo=
guelph.ca


On Sun, 2021-04-25 at 02:29 +0000, Rick Macklem wrote:
> Hi,
>
> I have been running a simple test using two clients (one FreeBSD
> and the other Linux, Ferdora Core30, 5.2 kernel) with delegations
> enabled in the server.
>
> The test consists of running the connectathon general tests
> alternately om each client, using the same directory on the
> server.
> --> As such, each one results in CB_RECALLs of delegations
>       from the other client.
> Everything seems fine until the server does multiple concurrent
> CB_RECALLs for different files/delegations using different
> callback session slots.
> --> Then the Linux client decides it must create a new connection,
>        which breaks the back channel.
>        After 0.1sec, the FreeBSD server notices the broken back
>        channel and starts setting SEQ4_STATUS_CB_PATH_DOWN.
>        --> 15sec after that, the Linux client does a
> BindConnectionToSession
>               and things start working again.
>
> The mystery to me is why the client decides to create a new TCP
> connection, forcing this 15sec hickup each time it happens?
>
> If you are interested in looking at a packet capture. you can
> % fetch https://people.freebsd.org/~rmacklem/twoclientdeleg.pcap
> There are multiple examples in it. One is at:
> packet# 3518, 3520, 3521 CB_RECALL requests for 3 different
> delegations
>                          time 137.5
> --> This is followed by a close and open of a new TCP connection...
> packet# 3582 - first one with SEQ4_STATUS_CB_PATH_DOWN at
>                          time 137.6
> packet# 3604 - client does a bindconnectiontosession at
>                          time 152.7
> Then things start to happen again...
> 192.168.1.5 - FreeBSD server
> 192.168.1.6 - Linux client
> 192.168.1.13 - FreeBSD client
>
> If this is a known issue that you think is fixed in a more recent
> Linux kernel, then sorry about the noise.
>

Should have been fixed in Linux 5.3 by commit 7402a4fedc2b ("SUNRPC:
Fix up backchannel slot table accounting") AFAICT.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com