From:   "Andrew J. Romero" <romero@fnal.gov>
To:     Jeff Layton <jlayton@kernel.org>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: RE: Zombie / Orphan open files
Thread-Topic: Zombie / Orphan open files
Thread-Index: AQHZNPfbZsyiVLox/EunGMra5E7rNa64haaAgAATyXA=
Date:   Tue, 31 Jan 2023 14:42:18 +0000
Message-ID: <SA1PR09MB75528A7E45898F6A02EDF82EA7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
References: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
         <YQBPR01MB10724AEE306F99C844101EED086CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
         <YQBPR01MB10724F79460F3C02361279E8686CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
         <654e3b7d15992d191b2b2338483f29aec8b10ee1.camel@kernel.org>
         <YQBPR01MB10724B36E378F493B9DED3C7E86D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
         <3c02bd2df703a68093db057c51086bbf767ffeb1.camel@kernel.org>
         <YQBPR01MB1072428BC706EE8C5CC34341186D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
         <936efa478e786be19cb9715eba1941ebc4f94a1b.camel@kernel.org>
         <SA1PR09MB75521717AA00DCAD6CAB5118A7D39@SA1PR09MB7552.namprd09.prod.outlook.com>
 <2bc328a4a292eb02681f8fc6ea626e83f7a3ae85.camel@kernel.org>
In-Reply-To: <2bc328a4a292eb02681f8fc6ea626e83f7a3ae85.camel@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?d6AJkLgMcvDJv3aZ7qti1fQ1GM7edOC5n/Ssc2zXA1kDUSrx9P71EiQcUVrQ?=
 =?us-ascii?Q?9+n5cx+UfX2K1dD0xC3P9w69oQjdxudH2j6fqjmAdlxYa7QNftrnNYgBW/Ep?=
 =?us-ascii?Q?q9vnCEjRcLSswMTML+RarA6bwko+CasgDPVEvohaQZEiZaHU7r9n0v44XUQn?=
 =?us-ascii?Q?Xrwnkq/unMdW3RaMRikYS2P4I2oARIATu4SK4fZVEC814WPL7w+DtagRf0nC?=
 =?us-ascii?Q?zcG/gJcUWifLwNSkpz9mEoqJT7ata9eYGbfYlpu/mgW0w+UFZvunL7QNq6G8?=
 =?us-ascii?Q?rWj2e/ps5zbJcXxzL4ur9ljZfrcjWPkVer6E+6OKiSf7XyiGc0T0jz9PFfQU?=
 =?us-ascii?Q?gXzw5aJtWfs0LPEg5ijbV19tWZ/gxkSuwfcagFEjn45rCPO+NeFoF30y+j8L?=
 =?us-ascii?Q?jZOKtgzbh86kJ46UQpDLZ8SK6M9bFs/B6GH2QbOsT1XO1ZW4l1LRKdbMg4IS?=
 =?us-ascii?Q?prwyKUdyFv7EnpdlN5m0+PTd8P4IGZUK14MLtlNPn3Eou5hKsCAG3RMdp8FB?=
 =?us-ascii?Q?b/X1Mapy7XIQmWaJpRCOodT0rvlMaEqpBWr0VAoNkfoYiX3aeR8blJoNLgw1?=
 =?us-ascii?Q?vfrJwlsZnNjC2hcTOzTUY5qyqHDU4PfwhzTnN28ZQNdiaPrVxFMxJKRol/k6?=
 =?us-ascii?Q?PebT3s3ZrQ/ZIFCBqGR79uqa4Wa5DKnEZrZmxPSICdbUfWPPFvZAiRHGSKMe?=
 =?us-ascii?Q?15nsUNIDmV+919w/urtlC+JeEyLPwUPtT+wDeW2ME1EaShcOrmIhfhyULP39?=
 =?us-ascii?Q?sKBnWhrpk+WQBlIML+d3xo4talWl22y96TnqQf8aPto4sgrwLsqZqkyMKHBS?=
 =?us-ascii?Q?gRjmIeriHtjG/fUodtQ3QZUktolY5AcyCEnoKnZy9L4Qx+QoiAYpr2FF2u4R?=
 =?us-ascii?Q?sGvdwGwqGy8sP6+trgm0b/sNYtM85cK4lVtZBc3vq7UxfrV9LVRO4wPmfmH6?=
 =?us-ascii?Q?3uua3ob1I5/D22iWjE/zLfYhI6rYZqZyZr5UuNGwKV4+18LzPEWW9KqvhYVe?=
 =?us-ascii?Q?nupqdaqJVp2dYTWuzRNhdptZAK/EbrYGkanZ7xu4xp13LOQZ0BgngVggT4iC?=
 =?us-ascii?Q?bwLjsfPzwH1CmT0wKZvIArG47wbTcmU0862HqhYSjrvQukyrAHdn+yuSms9/?=
 =?us-ascii?Q?6plK0Xok8tUF9LP/4B/0dwzABonDdhUxuwhd5z6jNXbviVGI9maQtIRf7U2h?=
 =?us-ascii?Q?hPLWW01qJfKJl3TzUOr+KwChTqqWosvq2qhJatHkoBoET7jIK3uy7b5cl6xe?=
 =?us-ascii?Q?BU2X2W8WjXIFQIEhV24n9vzCs/vTlaxSoeKaQPWDlcec2NGFxycnYFFHkTYz?=
 =?us-ascii?Q?kqFOGcranef66EODSPDz1uW9e6ZvcCRy57MIYXlcsFBQF3ugZh7c9Cdh6+sr?=
 =?us-ascii?Q?dz0LJseaAbwlhlC13prRJd0Hj1yvOHifDrosJd3ULYH9W95pwcsPzBcxLXZE?=
 =?us-ascii?Q?/UpswBa74yLef3PeCFUlT5n2g7p+Jeo5ReAI8haV/YllHCCz8DKJ4dAGcobs?=
 =?us-ascii?Q?NEfxqBErU2apLD9xmMx8jWdPONn/Jw5k6TaJpVk6utNH5Jzh+7ONlGNJLBN0?=
 =?us-ascii?Q?wAO2lOyXKkN4ztAhnLMoiNNEopH7DiGi4cx1e8qp?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: fnal.gov
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR09MB7552.namprd09.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 105aa748-79f0-41de-2543-08db03995a9c
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Jan 2023 14:42:18.7650
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 9d5f83d3-d338-4fd3-b1c9-b7d94d70255a
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR09MB6280
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> From: Jeff Layton <jlayton@kernel.org>
=20
> What do you mean by "zombie / orphan" here? Do you mean files that have
> been sillyrenamed [1] to ".nfsXXXXXXX" ? Or are you simply talking about
> clients that are holding files open for a long time?

Hi Jeff

.... clients that are holding files open for a long time

Here's a complete summary:

On my NAS appliances , I noticed that average usage of the relevant memory =
pool
never went down. I suspected some sort of "leak" or "file-stuck-open" scena=
rio.

I hypothesized that if NFS-client to NFS-server communications were frequen=
tly disrupted,
this would explain the memory-pool behavior I was seeing.
I felt that Kerberos credential expiration was the most likely frequent dis=
ruptor.

I ran a simple python test script that (1) opened enough files that I could=
 see an obvious jump
in the relevant NAS memory pool metric, then (2) went to sleep for shorter =
than the
Kerberos ticket lifetime, then (3) exited without explicitly closing the fi=
les.
 The result:  After the script exited,  usage of the relevant server-side m=
emory pool decreased by
the expected amount.

Then I ran a simple python test script that (1) opened enough files that I =
could see an obvious jump
in the relevant NAS memory pool metric, then (2) went to sleep for longer t=
han the
Kerberos ticket lifetime, then (3) exited without explicitly closing the fi=
les.
 The result:  After the script exited,  usage of the relevant server-side m=
emory pool did not decrease.
( the files opened by the script were permanently "stuck open" ... depletin=
g the server-side pool resource)

In a large campus environment, usage of the relevant memory pool will event=
ually get so
high that a server-side reboot will be needed.

I'm working with my NAS vendor ( who is very helpful ); however, if the NFS=
 server and client specifications
don't specify an official way to handle this very real problem, there is no=
t much a NAS server vendor can safely / responsibly do.

If there currently is no formal/official way of handling this issue ( serve=
r-side pool exhaustion due to "disappearing" client )
is this a problem worth solving ( at a level lower than the application lev=
el )?

If client applications were all well behaved ( didn't leave files open for =
long periods of time ) we wouldn't have a significant issue.
Assuming applications aren't going to be well behaved, are there good gener=
al ways of solving this on either the client or server side ?

Thanks

Andy