Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Rick Macklem <rmacklem@uoguelph.ca>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFSv4.1/4.2 server returns same sessionid after
 DestroySession/CreateSession
Thread-Topic: NFSv4.1/4.2 server returns same sessionid after
 DestroySession/CreateSession
Thread-Index: AQHYlKIG8hnDNDQzXUuZ/p7iJwgH2615RPsAgAAqTQCAAAlZgA==
Date:   Mon, 11 Jul 2022 18:06:31 +0000
Message-ID: <24C858F4-5334-4417-BFA5-4D580274F47E@oracle.com>
References: <YQBPR0101MB97421B80206B30FC32170C25DD849@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM>
 <DCE64EDD-FCE4-4C21-8B00-7833D5EB4EB2@oracle.com>
 <89044942-DAFE-4E9B-BC70-A8D2C847A422@oracle.com>
In-Reply-To: <89044942-DAFE-4E9B-BC70-A8D2C847A422@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rKYYyn2PS8ZOPXjvoKQCyu8qnMYt+MGwME6AJdvo7jed2mwpaht70ttxlwd8?=
 =?us-ascii?Q?QUpnWTzfb7T3F3cL2Ak/TsvFI5M0gk5a5USQyB8kXQlPQ/7ik9fmGaidyTMW?=
 =?us-ascii?Q?Nr6Fyvlc1cDtWhKTrp8aqurIfKxtW53+msOFD+GvHemppje26fYH3GwDq2xP?=
 =?us-ascii?Q?/a+hpI2hvq2Sv8srVLIWeP4rqHktw20K4awDzNfh1bt2GSKMPPBMriIYpGg7?=
 =?us-ascii?Q?BkO1j/30H7Ndig6zzVq3TPr6ayGeV+jI9Of8imU9wc19DwLE79DSqMd9YfhA?=
 =?us-ascii?Q?7aCHtRWe6FvFu+QZfPvyXD7dJVuC9zOS/iSXNiFDKvTJzkMpWpqMxfKugXPV?=
 =?us-ascii?Q?4TuBsJKBAw81HDGElkRUxMXAVm9pzdtJAazv99SSpRjRB+01ftF+SqCEeViZ?=
 =?us-ascii?Q?JYXF4TPacdr0ip6CcO478ggxqZW4IYPFQMVGMknqL1XxNM0iE31POaAfn1/x?=
 =?us-ascii?Q?+nrLxeybT4TXZsWb3CdQfa8NBlsathh6L7Ov+ZQavIytaAjjmEF1tA5Oi9+a?=
 =?us-ascii?Q?nbQbN1JMGTpX+ur2+MFcIgmYoFTrt5yKm9jRKeHhq/C7eRzw4yiKGlRL1yf2?=
 =?us-ascii?Q?3O/GyaodN0B0TC2Q8A90ATE7peplQl2FPKf/q9H6IuAjqTUJTJuJCt+KrUDW?=
 =?us-ascii?Q?JuDeVy0TjQeXKBuoJyUwN0/8g1e7KP6p7/XRVIFKunuhkTToIOyapQtn4058?=
 =?us-ascii?Q?sCiCpuSBXOqsKMg72qFTTQU3aJoUhV2pd6AFvdWaM+o2MD2JH2WB+HGB5ZJA?=
 =?us-ascii?Q?h13t4642hLbb2JbPwCrSu0U6gC7Uv8tt6gevionNMBCI+Q6F0b7WGb/WIlNc?=
 =?us-ascii?Q?FFAa1AcuCUAL8XHb2UCQ78SYUhw+FFBMizqW8FkTvq1N3f9ZzmjYelDYBjvw?=
 =?us-ascii?Q?jJony8nc1NSgPeVpnLwAFKfA0s4PXV553qkL3/2pbjX/JRarVdCVeI09NQIw?=
 =?us-ascii?Q?li5y4aOo0stAJAsCQtK0xsNAt7OhjMpw2lRXBwEMn/lGYoTeB26YtvgDMQ+/?=
 =?us-ascii?Q?J9GesWLiFRHGwEHzPjurbS1boQB8ji6UxlL0Ua800wNlGfvo8Bw1sSZocmQ/?=
 =?us-ascii?Q?dhYL1Lp/dC0zPWieBBkQBypMWdN0zmlrS0w9m3yoURonJmn3xKc75uLwOyJV?=
 =?us-ascii?Q?87IL9gz6ImDFi6x9LzHLabhKfpS/zj1dj2WWpEUmaBnor/gveo58TNtHA/Mg?=
 =?us-ascii?Q?zGZmFReNqYI2pT2MyKDLOSVaQuxdaW4+/u62HrFrsTzEHku0g7Kx+mcs7fXD?=
 =?us-ascii?Q?QCQQpNDEnd3sgPnprfSwSl30nSrfL8CBrA+uZz+QHoJZj08IWAIlFRZc1ea5?=
 =?us-ascii?Q?0qoqgk5maRMRvPxD6HIMPvctfub5iXZh9FoSwLCRgbMqvJAnwRJr9OyIzS2X?=
 =?us-ascii?Q?Zy6xxjYJYeZnrNB54GYvrH6Ne72UwjezSSQSjVq0yz0+wZrNsKh4fOrNl7ej?=
 =?us-ascii?Q?v0h4mWCnGUr/8CJpOWAbIftcYd0MMKtyEC0W8pnwbZ03TBzElmozdEqE8CMs?=
 =?us-ascii?Q?ZrU1VWqjqaSahBl/N+U8W7Y6jOR+0P0GQ5Bnarf0avTQVndkflVoXCPxUC9T?=
 =?us-ascii?Q?8PFr5yAHV97bm2ysdxoavczWAZSFeis8+jTsVI0ZVxCY9wzbKM/6BAurUVLU?=
 =?us-ascii?Q?mg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C2D1262C0324F3408C3E0144B20DBA63@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 16b5828d-28b1-4510-89d5-08da636815d6
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jul 2022 18:06:31.9851
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: aDJ68ys29c+GmrJRZAgJl5iS6+cIafQ84Ow4yREAxdtry3b5mAFle/TPRmza2wxSlUmA33ZBn+OTBWI+IIXS3g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1786
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-07-11_23:2022-07-08,2022-07-11 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0
 suspectscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 malwarescore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2206140000 definitions=main-2207110077
X-Proofpoint-GUID: _EgRJWJbTtciIBi68B_8zlmoz_edZAKu
X-Proofpoint-ORIG-GUID: _EgRJWJbTtciIBi68B_8zlmoz_edZAKu
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 11, 2022, at 1:33 PM, Chuck Lever III <chuck.lever@oracle.com> wro=
te:
>=20
>=20
>=20
>> On Jul 11, 2022, at 11:01 AM, Chuck Lever III <chuck.lever@oracle.com> w=
rote:
>>=20
>>=20
>>=20
>>> On Jul 10, 2022, at 6:10 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
>>>=20
>>> Hi,
>>>=20
>>> I have been trying to improve the behaviour of the FreeBSD
>>> NFSv4.1/4.2 client when using the "intr" mount option.
>>>=20
>>> I have come up with the following scheme:
>>> - When RPCs are interrupted, mark the session slot as potentially bad.
>>> - When all session slots are marked potentially bad, do a
>>> DestroySession (only op in RPC) to destroy the session.
>>> - When the server replies NFS4ERR_BAD_SESSION,
>>> do a CreateSession (only op in RPC) to acquire a new session and
>>> continue on.
>>>=20
>>> When testing against a Linux 5.15 server, the CreateSession
>>> succeeds, but returns the same sessionid as the old session.
>>> Then all subsequent RPCs get the NFS4ERR_BAD_SESSION reply.
>>> (The client repeatedly does CreateSession RPCs that reply NFS_OK,
>>> but always with the same sessionid as the destroyed one.)
>>>=20
>>> Here's what I see in the packet trace:
>>> (everything works normally until all session slots are marked
>>> potentially bad at packet# 14216)
>>> packet# RPC
>>> 14216 DestroySession request for sessionid 2725cb62002ed418040...0
>>> 14302 DestroySession reply NFS_OK
>>> 14304 Getattr request (using above sessionid)
>>> 14305 Getattr reply NFS4ERR_BAD_SESSION
>>> 14306 CreateSession request
>>> *** Now here is where I see a problem...
>>> 14307 CreateSession reply NFS_OK with sessionid=20
>>> 2725cb62002ed418040...0 (same as above)
>>> 14308 Getattr request (using above sessionid)
>>> 14309 Getattr reply NFS4ERR_BAD_SESSION
>>> - and then this just repeats...
>>> The whole packet trace can be found here, in case you are interested:
>>> https://people.freebsd.org/~rmacklem/linux.pcap
>>>=20
>>> It seems to me that a successful CreateSession should always return
>>> a new unique sessionid?
>>=20
>> Hi Rick, thanks for the bug report.
>>=20
>> CREATE_SESSION has a built-in reply cache to thwart replay attacks.
>> It can legitimately return the same sessionid as a previous request.
>> Granted, DESTROY_SESSION is supposed to wipe that reply cache...
>>=20
>> I'd like to see if there's a test in pynfs that replicates or is close
>> to the series of operations in your trace so that I can reproduce on
>> my lab systems and watch it fail up close.
>=20
> I constructed a pynfs test that does something similar to your
> reproducer:
>=20
> diff --git a/nfs4.1/server41tests/st_destroy_session.py b/nfs4.1/server41=
tests/st_destroy_session.py
> index b8be62582366..014330e7d623 100644
> --- a/nfs4.1/server41tests/st_destroy_session.py
> +++ b/nfs4.1/server41tests/st_destroy_session.py
> @@ -1,12 +1,33 @@
> from .st_create_session import create_session
> from xdrdef.nfs4_const import *
> -from .environment import check, fail, create_file, open_file
> +from .environment import check, fail, create_file, open_file, close_file
> from xdrdef.nfs4_type import open_owner4, openflag4, createhow4, open_cla=
im4
> import nfs_ops
> op =3D nfs_ops.NFS4ops()
> import threading
> import rpc.rpc as rpc
>=20
> +def testDestroyBasic(t, env):
> + """Ensure operations outside a session fail with BADSESSION
> +
> + FLAGS: destroy_session all
> + CODE: DSESS1
> + """
> + c =3D env.c1.new_client(env.testname(t))
> + sess1 =3D c.create_session()
> + sess1.compound([op.reclaim_complete(FALSE)])
> + res =3D c.c.compound([op.destroy_session(sess1.sessionid)])
> + res =3D create_file(sess1, env.testname(t),
> + access=3DOPEN4_SHARE_ACCESS_READ)
> + check(res, NFS4ERR_BADSESSION)
> + sess2 =3D c.create_session()
> + res =3D create_file(sess2, env.testname(t),
> + access=3DOPEN4_SHARE_ACCESS_READ)
> + check(res)
> + fh =3D res.resarray[-1].object
> + open_stateid =3D res.resarray[-2].stateid
> + close_file(sess2, fh, stateid=3Dopen_stateid)
> +
> def testDestroy(t, env):
> """
> - create a session
>=20
> I'm not able to reproduce the problem on 5.19-rc5, but that
> probably means there's something going on that we haven't
> discovered yet.

My guess is that your client is sending CREATE_SESSION operations
with the same sequence ID (1) and that is hitting in NFSD's
CREATE_SESSION reply cache. So it's treating the client's new
requests as replays and returning an old (stale) sessionid.


--
Chuck Lever