From:   "Andrew J. Romero" <romero@fnal.gov>
To:     Chuck Lever III <chuck.lever@oracle.com>
CC:     Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: RE: Zombie / Orphan open files
Thread-Topic: Zombie / Orphan open files
Thread-Index: AQHZNPfbZsyiVLox/EunGMra5E7rNa64haaAgAATyXCAACB5gIAAAJ1QgAAY44CAAAHUIIAACwaAgAACmOA=
Date:   Tue, 31 Jan 2023 19:32:42 +0000
Message-ID: <SA1PR09MB75522342729588745CBAA834A7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
References: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <YQBPR01MB10724AEE306F99C844101EED086CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <YQBPR01MB10724F79460F3C02361279E8686CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <654e3b7d15992d191b2b2338483f29aec8b10ee1.camel@kernel.org>
 <YQBPR01MB10724B36E378F493B9DED3C7E86D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <3c02bd2df703a68093db057c51086bbf767ffeb1.camel@kernel.org>
 <YQBPR01MB1072428BC706EE8C5CC34341186D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <936efa478e786be19cb9715eba1941ebc4f94a1b.camel@kernel.org>
 <SA1PR09MB75521717AA00DCAD6CAB5118A7D39@SA1PR09MB7552.namprd09.prod.outlook.com>
 <2bc328a4a292eb02681f8fc6ea626e83f7a3ae85.camel@kernel.org>
 <SA1PR09MB75528A7E45898F6A02EDF82EA7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
 <0BBE155A-CE56-40F7-A729-85D67A9C0CC3@oracle.com>
 <SA1PR09MB755212AB7E5C5481C45028A8A7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
 <25395E08-073E-4572-A46D-A2228DB0212E@oracle.com>
 <SA1PR09MB7552B00147DBC9200BB8CDBAA7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
 <884655AA-804E-4AD8-8469-04EE3AE75476@oracle.com>
In-Reply-To: <884655AA-804E-4AD8-8469-04EE3AE75476@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?IbO29fi7Wx+u+boyVcXK/t+LQfYNl6SAEwKQ0PGUe7HfmJsqy/FtqvDEZFOx?=
 =?us-ascii?Q?UxoXFuxP+0nLCtXs+WAlwVctLpQjJb+T2vD7Id3YMYJ1euZ1JALLMpN+A/nY?=
 =?us-ascii?Q?VMKuso9hWUGfuIeIxbNrxSQeMw/iNCkgowlY+7ZXyVVYNRMdhOlYheoUe1Se?=
 =?us-ascii?Q?37JSGdaBXVym224+ttREAXt+NwqIDblfDAcVXTSkzRBPf7rwha78Db6uJkDA?=
 =?us-ascii?Q?UW7A94iIr1qiQPUdqw5R2frY2VFrXYIzsFMN5eOmM2A1q0bNKdF1TF0qaA9A?=
 =?us-ascii?Q?dIdMeQILgu1IOC+/gWSVPlD5sakcSME+bfFj3WBMd3LQ6QMwEOQtdwyRo1Ue?=
 =?us-ascii?Q?sOKRTzi4gJ2ya9tBUrDzn4Hn12LNXQn6Z+kHPwcG17DLwmIc6ABVdtmgu4pK?=
 =?us-ascii?Q?/BXNwsFSLEskyNWjkecRGGchc22Ka5vlR+EmOOAQckiHy7wTtqK9XTSOt3Vk?=
 =?us-ascii?Q?FLCW9JzGi9YHbbgeWH1i1kozUt3VP8H1fJTD1jjIGzjipgbz/YXzMXnYc1LS?=
 =?us-ascii?Q?56jl23IbbN0Gtw5+XTakT1W4Xjzay6AUgBDaryXbuzf01jYt1warWR5yVsPn?=
 =?us-ascii?Q?NcOns0SR7q2KkGFDAOZINTYEhyx4Kh46eesT784FtNMETYSRQ79jELEoJJ0S?=
 =?us-ascii?Q?D/QbRwScwcTG0iJKb+IuR9DWlfufDDz17gw6eZjeETCiI4Wuc38gvENRE1Vc?=
 =?us-ascii?Q?x5y6316Un52rigzMZRyTXtNzPA3vijtbtUVT/ss8XYhepfQVjjMHeD5c3VBU?=
 =?us-ascii?Q?MOd4bBmGKxVAWYyfdjwm9PYrBo9V4WEalfaoMVk8n+4BRZOXqUu6fDY05EYc?=
 =?us-ascii?Q?5iQdTYTStNLCQF+xC6F4bBI8HQg0L2jrUWUUtMQTpMwUaMyG2mj9lkmA5TpV?=
 =?us-ascii?Q?9AfZ8WEGB/wXBzDp4Yh+nb4kCoJ4RwihQGWzwypwOUu/NyWsypOY0NAI8nA6?=
 =?us-ascii?Q?6jTBXvl2sYffn9u30246NxD8B6UR7oZZr2nPBe/fMG6JP5hfiCf9brZkFttq?=
 =?us-ascii?Q?gPRhiuFmYyYtowELxg5R138K4l+38EKJPfl2i2S9NAQW10ITpxRyyDWvrvPf?=
 =?us-ascii?Q?HE65aRlId+6JW+TyGtbo33sPVNcIRR+IOExZ2Cw3UAtFT5PExeXHtDQE+kBO?=
 =?us-ascii?Q?H5AJpLJR1YEcc+zSdXaD5COB9wFUmFpN4RuvP0au8Kto3GmY4+wsYp88e+Fq?=
 =?us-ascii?Q?W8sQkkP2EwPxw4nviwJXpOf8EFQdvsA7shngyAyuJoFQ3vG3rJdJoFbmE0nC?=
 =?us-ascii?Q?Gg5M4BtYWgm+lgq4C6gWRxVRV0CpIdEgpkM2BFEWjHzdQrXAJUDMz7W3ApvX?=
 =?us-ascii?Q?VHFglA4yDmzEwivWxN5sN3DhXUdwhSKc4jMqQwTze3M2DMwyXolpSpTb1Mlm?=
 =?us-ascii?Q?hm09NuJvfy7ICK3dPhc9iSuJOuvINsnB+hk1+AssI9RZhEnd2LqJorIMUSJ7?=
 =?us-ascii?Q?GFZlYCjvhsPt1bxV2NKrtWgHMKjU8zc45Rh9NVi7sq4J5ZF73ZRhk0Wp4EQ+?=
 =?us-ascii?Q?kcUSmBrcOIMArRZUeX6v8xj/ep/101ottHkBeUOya1D9TDc4pW+0txgt0lPY?=
 =?us-ascii?Q?vYFHTlNL4CE96Z9wnFlhpww2Glt07Z5cghD67Uux?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: fnal.gov
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR09MB7552.namprd09.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 21303dfe-6133-4b35-64a3-08db03c1ec21
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Jan 2023 19:32:42.7767
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 9d5f83d3-d338-4fd3-b1c9-b7d94d70255a
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR09MB10187
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> -----Original Message-----
> From: Chuck Lever III <chuck.lever@oracle.com>
>=20
> Almost. The protocol requires:
>=20
> After the client reboots, when it opens its first file, the client
> does a SETCLIENTID or EXCHANGE_ID to establish its lease on the
> server. All OPEN and LOCK state is managed under the umbrella of
> that lease (and that includes all files that client is managing).
> The client keeps the lease alive by renewing the lease every minute.
>=20
> If the client reboots (ie, does a subsequent SETCLIENTID or
> EXCHANGE_ID with a new boot verifier), the server has to purge all
> open file state for that client.
>=20
> If the client fails to renew its lease, the server is free to do
> what it wants -- it can purge the client's lease immediately, or
> it can wait until conflicting opens or locks come from other
> clients and then purge some or all of that client's lease.
>=20
> If the client can't or doesn't CLOSE that file, it will remain
> on the server until the client tells it (implicitly by not
> renewing or explicitly with a fresh ID) that the state is no
> longer needed; or until the server reboots and the client does
> not re-establish the OPEN state.

So , in general, this is true:
  - A lease is not "issued" for every file opened
  - A lease is not "issued" for every user running on an NFS-client host
  - In general. one lease is issued / managed for each NFS-client host
( if this is true,  my server vendor is probably not forgetting to do
  something they should be doing )


> But again, we need some way to confirm exactly how this is
> happening. Can you post your script, or capture client-server
> network traffic while the script does its thing?
>=20

The script is about simple as "hello world":

import sys
import fileinput
import os.path
import re
import time

def main():

   StartID=3Dint(raw_input("Enter Start ID: "))

   TestDir=3Dos.path.normcase('/nashome/r/romero/stuckopentest/dataout')

   FPlist=3D[]

   # open 2000 files and leave them open
   for x in range(StartID, StartID+2000):

      TestFilePath=3Dos.path.join(TestDir, "TestFile-" + str(x))
      print(TestFilePath)

      # open file append file pointer to list
      FPlist.append(open(TestFilePath,"w"))


   # sleep for greater than Krb ticket life time
   # 2000 files will be "stuck open" on the server
   time.sleep(60*60*24)


main()


NOTE:

I don't expect people on this list to debug my issue.

My reason's for posting:

- Determine If my NAS vendor might be accidentally
  not doing something they should be.
  (  I now don't really think this is the case. )


- Determine if this is a known behavior common to all NFS implementations
   ( Linux, ....etc ) and if so have you determine if this is a problem tha=
t should be addressed
   in the spec and the implementations. =20