Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Rick Macklem <rmacklem@uoguelph.ca>
To:     Olga Kornievskaia <aglo@umich.edu>,
        "J. Bruce Fields" <bfields@fieldses.org>
CC:     Linux-NFS <linux-nfs@vger.kernel.org>
Subject: Re: Linux NFSv4.1 client session seqid sometimes advances by 2
Thread-Topic: Linux NFSv4.1 client session seqid sometimes advances by 2
Thread-Index: AQHXL9rIYn3uMMvy90evx7HGhIWLsKqycvKAgAA/JgCAABxygIAACEwAgAAXNwCAAB/6ZA==
Date:   Tue, 13 Apr 2021 23:08:51 +0000
Message-ID: <YQXPR0101MB096851380C58EA8F671427E0DD4F9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
References: <YQXPR0101MB0968E44CE6FB05F22DE27716DD709@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
 <CAN-5tyEWwAtBG41VByps+zS6Sw_z9U7Q8HZb84SqyXp4woUiPA@mail.gmail.com>
 <20210413171738.GA28230@fieldses.org>
 <CAN-5tyF=NBaZkv-CMKdK2JGMCRQ7cO2KNPTwO5KEPFG-JO4D4g@mail.gmail.com>
 <20210413192908.GD28230@fieldses.org>,<CAN-5tyEWVk9zH-aRJ7FSS5=nTLtVqfqFO9JEjQnp5PnCtmn=aQ@mail.gmail.com>
In-Reply-To: <CAN-5tyEWVk9zH-aRJ7FSS5=nTLtVqfqFO9JEjQnp5PnCtmn=aQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: da22b6e1-b4e8-41bf-3ad5-08d8fed11a51
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Apr 2021 23:08:51.4796
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 9Dp5KPsMTELXCkt4I7jA/NajAKS+ILnz4G5pQK+xToV+9g2yxRzy131nhaYabxFT6AkeQURtHwNwhbQcFv+2dg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3592
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Olga Kornievskaia wrote:=0A=
>On Tue, Apr 13, 2021 at 3:29 PM J. Bruce Fields <bfields@fieldses.org> wro=
te:=0A=
>>=0A=
>> On Tue, Apr 13, 2021 at 02:59:27PM -0400, Olga Kornievskaia wrote:=0A=
>> > On Tue, Apr 13, 2021 at 1:17 PM J. Bruce Fields <bfields@fieldses.org>=
 wrote:=0A=
>> > >=0A=
>> > > On Tue, Apr 13, 2021 at 09:31:37AM -0400, Olga Kornievskaia wrote:=
=0A=
>> > > > On Tue, Apr 13, 2021 at 3:08 AM Rick Macklem <rmacklem@uoguelph.ca=
> wrote:=0A=
>> > > > >=0A=
>> > > > > Hi,=0A=
>> > > > >=0A=
>> > > > > During testing of a Fedora Core 30 (5.2.10 kernel) against a Fre=
eBSD=0A=
>> > > > > server (4.1 mount), I have been simulating a network partitionin=
g=0A=
>> > > > > for a few minutes (until the TCP connection goes to SYN_SENT on=
=0A=
>> > > > > the Linux client).=0A=
>> > > > >=0A=
>> > > > > Sometimes, after the network partition heals, the FreeBSD server=
=0A=
>> > > > > replies NFS4ERR_SEQ_MISORDERED.=0A=
>> > > > > Looking at the packet trace, the seqid for the slot has advanced=
 by=0A=
>> > > > > 2 instead of 1. An RPC request for old-seqid + 1 never seems to =
get=0A=
>> > > > > sent.=0A=
>> > > > > --> Since sending an RPC with "seqid + 2" but never sending one=
=0A=
>> > > > >        that is "seqid + 1" for a slot seems harmless, I have add=
ed an optional=0A=
>> > > > >        hack (can be turned off), to allow this case instead of r=
eplying=0A=
>> > > > >        NFS4ERR_SEQ_MISORDERED for it. The code will still reply=
=0A=
>> > > > >        NFS4ERR_SEQ_MISORDERED if an RPC for the slot with=0A=
>> > > > >        "old seqid + 1" in it.=0A=
>> > > > >        --> Yes, doing this hack is a violation of RFC5661, but I=
've=0A=
>> > > > >              done it anyhow.=0A=
>> > > > >=0A=
>> > > > > If you are interested in a packet capture with this in it:=0A=
>> > > > > fetch https://people.freebsd.org/~rmacklem/linuxtofreenfs.pcap=
=0A=
>> > > > > - then look at packet #1945 and #2072=0A=
>> > > > >   --> You'll see that slot #1 seqid goes from 4 to 6. There is n=
o=0A=
>> > > > >          slot#1 seqid 5 RPC sent on the wire.=0A=
>> > > > >          (This packet capture was taken on the Linux client usin=
g=0A=
>> > > > >           tcpdump.)=0A=
>> > > > > --> Btw, the "RST battle" you'll see in the above trace between=
=0A=
>> > > > >        #2005 and #2068 that goes on until the FreeBSD=0A=
>> > > > >        krpc/NFS times out the connection after 6min. seems to be=
 a recent=0A=
>> > > > >        FreeBSD TCP bug.=0A=
>> > > > >        I have reproduced this seqid advances by 2 on an older sy=
stem=0A=
>> > > > >        that does not "RST battle" and allows the reconnect right=
 away,=0A=
>> > > > >        once the network partition is healed, so it does seem to =
be=0A=
>> > > > >        relevant to this bug.=0A=
>> > > > >=0A=
>> > > > > Someday, I will get around to upgrading to a more recent Linux=
=0A=
>> > > > > kernel and will test to see if I can still reproduce this bug.=
=0A=
>> > > > > On 5.2.10, it is intermittent and does not occur every time I=0A=
>> > > > > do the network partitioning test.=0A=
>> > > > >=0A=
>> > > > > Mostly just fyi, rick=0A=
>> > > >=0A=
>> > > > Hi Rick,=0A=
>> > > >=0A=
>> > > > I think this is happening because slotid=3D1 had something queued =
up=0A=
>> > > > using seqid=3D5 and that was interrupted because the connection wa=
s=0A=
>> > > > RSTed. For the interrupted slot, the client would send solo SEQUEN=
CE=0A=
>> > > > with +1 seqid.=0A=
>> > >=0A=
>> > > Doesn't the client send the solo SEQUENCE with seqid 5 in that case?=
=0A=
>> >=0A=
>> > No it sends with seq+1 because NFS layer client doesn't know if seqid=
=0A=
>> > actually was actually transmitted before the connection got caught=0A=
>> > (and/or received by the server).=0A=
>>=0A=
>> But then the MISORDERED tells the client it wasn't received, so the=0A=
>> client follows up with a call with seqid 5--is that what happens?=0A=
>=0A=
>Correct. If there were no error then the server did indeed consume=0A=
>seqid. And if an error was returned then the client knows to=0A=
>decrement.=0A=
Ok. Yes. I took a closer look at the packet capture and that is what happen=
ed.=0A=
On slot#1...(server expecting seqid =3D=3D 5):=0A=
- Sequence with seqid 6=0A=
--> reply NFS4ERR_SEQ_MISORDERED=0A=
- next use of slot#1, Sequence with seqid 5=0A=
--> reply NFS_OK=0A=
- next use of slot#1, Sequence with seqid 6=0A=
--> reply NFS_OK=0A=
- then slot#1 gets used normally by other RPCs=0A=
=0A=
So it's an intentional "probe to re-synchronize the=0A=
seqid for the slot"?=0A=
Clever.=0A=
=0A=
I'll back my "hack" out, since I now see it is not necessary=0A=
(and does violate the RFC'-).=0A=
=0A=
So, is this how the Linux client deals with "soft,intr" NFSv4.1=0A=
mounts?=0A=
--> I've never solved that for FreeBSD and just note in the BUGS=0A=
       section of the man page that "soft,intr" mount options break=0A=
       an NFSv4 mount.=0A=
I might *borrow* this idea. I'll attribute it to whoever thought of=0A=
the idea, if you let me know who that is?=0A=
=0A=
Thanks for the clarification, rick=0A=
=0A=
> Sorry, I seem to recall we went through this all a couple years ago, but=
=0A=
> now I've forgotten how it works.=0A=
>=0A=
> --b.=0A=