From:   Chuck Lever III <chuck.lever@oracle.com>
To:     "Andrew J. Romero" <romero@fnal.gov>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: Zombie / Orphan open files 
Thread-Topic: Zombie / Orphan open files 
Thread-Index: AQHZNPfk4CJ0DwDhNkGXE9jEOOJtYK63pvcA
Date:   Tue, 31 Jan 2023 00:10:41 +0000
Message-ID: <0D998626-5934-47B1-BAC5-3D044FAB4926@oracle.com>
References: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <YQBPR01MB10724AEE306F99C844101EED086CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <YQBPR01MB10724F79460F3C02361279E8686CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <654e3b7d15992d191b2b2338483f29aec8b10ee1.camel@kernel.org>
 <YQBPR01MB10724B36E378F493B9DED3C7E86D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <3c02bd2df703a68093db057c51086bbf767ffeb1.camel@kernel.org>
 <YQBPR01MB1072428BC706EE8C5CC34341186D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <936efa478e786be19cb9715eba1941ebc4f94a1b.camel@kernel.org>
 <SA1PR09MB75521717AA00DCAD6CAB5118A7D39@SA1PR09MB7552.namprd09.prod.outlook.com>
In-Reply-To: <SA1PR09MB75521717AA00DCAD6CAB5118A7D39@SA1PR09MB7552.namprd09.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ZtjPAHpIwLTYHbSO46ydDZwQg87MKpqBdwfsNcWDZsp2t1TXCobMkBCAMvlq?=
 =?us-ascii?Q?weZMZg2tCLblHBj8ZDImWeKn0Wxe7gvjzDLZ9gQAI+RmgoNTEh8cqFcwEKyA?=
 =?us-ascii?Q?gw7scbxCg7Qp+gqdGQKe8K/A7lrDIb2OQL1J7m60WNEI9YM/85Td+sevBaXH?=
 =?us-ascii?Q?kYvTPuYXttn04ohNA9HgrvNI24BEGD9FOCgvKNEV82JmCO3Kj152n6sbCoTe?=
 =?us-ascii?Q?bH4T1LG2KPy5+8IkOeLCjNlPkAxCb9i3DJjVMK0Euu+V9/IKUwa+dUsHOEfs?=
 =?us-ascii?Q?uA8P2go2ZXmyIXuoUWxBWl9boDlt9sb6cinyZlRCamtrEzATGkA8iGxS/4mr?=
 =?us-ascii?Q?Awz6CVO8Ft4H+WA7/55plYu0Ohjp4ifXLHO9OihRZdFwqo4uDqB9huaJP2ar?=
 =?us-ascii?Q?hbVUw/fnBdLYI+3jZ4DvddyDwRVK7ZnQLKuEzABNGkZM00d0x3lk7E+Ul2Q2?=
 =?us-ascii?Q?a6pabdQx3FJPLWOWhsPeMN+QZOJZSryoGxxD2cLcrNrXGD6K3tIwMuooRW54?=
 =?us-ascii?Q?s9ObJ2vR1W7a6xHSsShUZO6sSG8J4O8ZueShPe884beog++SxLUwIbA2Kq6G?=
 =?us-ascii?Q?i65TRs1CovStDq130mPNZzgpQhKpv0/Mtmu1VNI0yIXXX96P82EjC6Mgimhk?=
 =?us-ascii?Q?DmobOUJqopuY+9dtdNhWynwdzxwwP8TUEWR1dAmHDJkdbiNAK+ftgFpAkqvb?=
 =?us-ascii?Q?/5QzEt81rPHO+lCfaTerU7rF7BEWB8vcpfJgDbHEy7Jzxl9GNjhCrpXgVGKD?=
 =?us-ascii?Q?WmkcW2VXhpO/BWQtGUfDEFW4IlVL8UUmvpmlQb0K6Bj2Gl0edWY3ye/gPeVz?=
 =?us-ascii?Q?ZixhRSCgCELaH1fds1MFqxfM/m2gKufNpNjliG7L8lXC175ncxEryfzyv8c6?=
 =?us-ascii?Q?Hqx0y2V3xdid4UxVccehivMoKbKa7X/VUlkMjZD7Fu2pBsT5dz5LhqHl+l3Y?=
 =?us-ascii?Q?mahCoPrTuAQ1H10SsKeNuRyHKsrGjxEf6h9HPidWjk8VZ4LNYzxcfjlX3tF5?=
 =?us-ascii?Q?zCP5r/PZSIjROvVA6KoSs23Cf0KVt/lm3fl0zfbZZ1EgNxOk8rIuMoErb+Bu?=
 =?us-ascii?Q?t3zVYxrw1yWz8vjMJLX5uW/Lt3q5hWxgipSjdBtZKtg8+GePtE9NBHtOe2Kf?=
 =?us-ascii?Q?281fDgLKpcenj9E8GNL3jetiJq8ydBUk+p424N04rAHiXzqm3MHy/cOdVXWg?=
 =?us-ascii?Q?tF6GCZq/JM+fK6zKApo9d14qzjkawRSNtioSKmsHHx+ed1W1eegH5gDLWV4k?=
 =?us-ascii?Q?AT2p26D2hcmXvCBtSrlOsFadqMsMnum6ZXdmI0d6pJJMgm+lLGlY1+9eT8j8?=
 =?us-ascii?Q?MuVe5fV5AjCZYDbBfwqxAa1sA+LZ9JPlyl+aj3pTRkC70rWoqDzR1ZezHwvv?=
 =?us-ascii?Q?/1UumA9eL1/cnqzYt5j2UHmvTWiz3hTtqbBbHEIdtU0CQoY2aJppf7iuw2mD?=
 =?us-ascii?Q?BoD7pl96B8VaixYrMxa5qqRdbRPYd8fVi7kl7b0nLsId3EgtTnTcaWIgmm9S?=
 =?us-ascii?Q?omZYta2Ceu4VJkt1I+2OClb2RpNt5MzLCxN/IiJ1F9212/xRNdrFi12HnQUr?=
 =?us-ascii?Q?jMNXUU6w83H0LnvIrfhSxC+zLDQynpG0VPQWfjFFqwVkVCIcAje4d0YwZWVf?=
 =?us-ascii?Q?Fg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9A494EC53037154ABC7EAD787404883F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Xwm/wkBehsHfNiZgME2EMmfF+LtzjPp4F2XP4isIdQv9ti4Pwg1GB+WaE4fhwHI0aaVa8hMiYy7cUIQIJwiOqFZdFUvYwljPXvsIjZJ+WFJ9U5KKv3n4zXlSP4XbCv/6Rj9in61TQZ6rnlhgM/QEKdu0JYdfvz7+rNKU68OWP7WoweKkytxlU0wpOJkvmO8hgkQBgyP9L4sy7EzQmd2rvy5FVTD9twaBMZMWUtyuAqS3ghy4Bbmw0FADf6nNJy2udQlvJIn3+GgvBD2NrklAJT0Bo/iaMzgDFgDttQKHg+2QM8vkxNAlCGXThWtwhR0WMW5lWt6d4/tI9YhH3c2MJ+Y4m+gSccoj1hy3eNjT4mjEqxx18ZYQ1eRVYBFZnN126rU6hRFA5faXtmA2c2UpiceHs5xpJxpQougUpnIuGbVCCgEoz7oqFHd8SZMarfAU0GSSk6ORPs2NCSvTsrmE15VQP2btHMqcqyZ8LZYJmQtu5dJi6Ewvv6qEao/1qLWKz0TobgCCcixbi/ejagnGKhpgZ+T/OQBEuwGe2FKXzyBFetRUw0vj1t2oiy6/+f7+PHZHYktRv8vLAXwZjzG0j7pK6Tnu+MDvDmPU6iH+x+itp+ZHKv7Xl7WsxWL+bkWpf8LRR0f4Y4Xi8NS36mOE0zOR2PO3PuFYMPyPMCK8vb5K0pDNfRInNzSpDAPifJi9PUnHoEV5VReLNHweYGEC1TccfkOmIsrJXdMdDVr+5idSbKwg9smETXGNObxab9kK8p9E8IQCRvV7nxLUQMLF9+Ewg6YWW9VyB5xUD7l7+Kw=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b4f9de63-4847-4373-44d3-08db031f96fc
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Jan 2023 00:10:41.4729
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: TA/Kqw1dGbUfXNTRKJIUXRec4FSFwNEvu9A2jlE9e/KWO13weYxPLuzmC2KO4yfqq0NyNo4Q+RJjPUvb0i8lfw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB6432
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1
 definitions=2023-01-30_18,2023-01-30_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0
 suspectscore=0 adultscore=0 mlxscore=0 phishscore=0 mlxlogscore=999
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2301300218
X-Proofpoint-ORIG-GUID: EgMEF5eKR03ZVRvY0jsGPmcSCsPnfbvw
X-Proofpoint-GUID: EgMEF5eKR03ZVRvY0jsGPmcSCsPnfbvw
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 30, 2023, at 5:11 PM, Andrew J. Romero <romero@fnal.gov> wrote:
>=20
> Hi
>=20
> This is a quick general NFS server question.
>=20
> Does the NFSv4x  specification require or recommend that:   the NFS serve=
r, after some reasonable time,=20
> should / must close orphan / zombie open files ?

No it does not. A server is supposed to leave open state alone
if the client continues to renew its lease.

A server has some recourse, though. It can recall delegations
to free up resources. We have some patches for v6.2 that do
that.

Servers can also free state where subsequent accesses by a
client indicate that the server administrator has revoked
that state. I don't believe the spec makes any statement
about when to use this facility or how to choose state to
purge, and I'm pretty sure Linux NFSD does not implement it.

A heavyweight tool would be to simulate a server reboot to
force clients to acknowledge which state they are still
using, via state recovery.


> On several NAS platforms I have seen large numbers of orphan / zombie ope=
n files "pile up"=20
> as a result of Kerberos credential expiration.
>=20
> Does the Red Hat NFS server "deal with" orphan / zombie open files ?

Not currently, nor does the upstream server.

Purging state is not terribly good for data integrity guarantees,
and I'm not sure how the server would make fair choices about
what OPEN stateids to purge.

So before going down that path I would like to see if the file
leakage might be the result of aberrant client behavior, and
try to address the issue from that side first. Do you have a
simple reproducer for this issue? How do you observe the
orphaned files?

--
Chuck Lever