MIME-Version: 1.0
References: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <YQBPR01MB10724AEE306F99C844101EED086CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <YQBPR01MB10724F79460F3C02361279E8686CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <654e3b7d15992d191b2b2338483f29aec8b10ee1.camel@kernel.org>
 <YQBPR01MB10724B36E378F493B9DED3C7E86D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <3c02bd2df703a68093db057c51086bbf767ffeb1.camel@kernel.org>
 <YQBPR01MB1072428BC706EE8C5CC34341186D39@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
 <936efa478e786be19cb9715eba1941ebc4f94a1b.camel@kernel.org>
 <SA1PR09MB75521717AA00DCAD6CAB5118A7D39@SA1PR09MB7552.namprd09.prod.outlook.com>
 <2bc328a4a292eb02681f8fc6ea626e83f7a3ae85.camel@kernel.org>
 <SA1PR09MB75528A7E45898F6A02EDF82EA7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
 <0BBE155A-CE56-40F7-A729-85D67A9C0CC3@oracle.com> <SA1PR09MB755212AB7E5C5481C45028A8A7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
 <CAN-5tyHOJ=qXUU73VsZC9Ezs7_-eZ46VDtiE_DWB3bdyr768gA@mail.gmail.com> <SA1PR09MB7552C7543CE6E9D263C070D4A7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
In-Reply-To: <SA1PR09MB7552C7543CE6E9D263C070D4A7D09@SA1PR09MB7552.namprd09.prod.outlook.com>
From:   Olga Kornievskaia <aglo@umich.edu>
Date:   Tue, 31 Jan 2023 17:14:05 -0500
Message-ID: <CAN-5tyGdaL_pYgqgS0TDwqCzVu=0rgLau8TDZMTe+hmC395UtQ@mail.gmail.com>
Subject: Re: Zombie / Orphan open files
To:     "Andrew J. Romero" <romero@fnal.gov>
Cc:     Chuck Lever III <chuck.lever@oracle.com>,
        Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, Jan 31, 2023 at 2:55 PM Andrew J. Romero <romero@fnal.gov> wrote:
>
>
>
> > What you are describing sounds like a bug in a system (be it client or
> > server). There is state that the client thought it closed but the
> > server still keeping that state.
>
> Hi Olga
>
> Based on my simple test script experiment,
> Here's a summary of what I believe is happening
>
> 1. An interactive user starts a process that opens a file or multiple files
>
> 2. A disruption, that prevents
>    NFS-client <-> NFS-server communication,
>    occurs while the file is open.  This could be due to
>    having the file open a long time or due to opening the file
>    too close to the time of disruption.
>
> ( I believe the most common "disruption" is
>   credential expiration )
>
> 3) The user's process terminates before the disruption
>      is cleared.  ( or stated another way ,  the disruption is not cleared until after the user
>     process terminates )
>
>    At the time the user process terminates, the process
>    can not tell the server to close the server-side file state.
>
>   After the process terminates, nothing will ever tell the server
>   to close the files.  The now zombie open files will continue to
>   consume server-side resources.
>
>   In environments with many users, the problem is significant
>
> My reasons for posting:
>
> - Are not to have your team  help troubleshoot my specific issue
>    ( that would be quite rude )
>
> they are:
>
> - Determine If my NAS vendor might be accidentally
>   not doing something they should be.
>   (  I now don't really think this is the case. )

It's hard to say who's at fault here without having some more info
like tracepoints or network traces.

> - Determine if this is a known behavior common to all NFS implementations
>    ( Linux, ....etc ) and if so have your team determine if this is a problem that should be addressed
>    in the spec and the implementations.

What you describe  --- having different views of state on the client
and server -- is not a known common behaviour.

I have tried it on my Kerberos setup.
Gotten a 5min ticket.
As a user opened a file in a process that went to sleep.
My user credentials have expired (after 5mins). I verified that by
doing an "ls" on a mounted filesystem which resulted in permission
denied error.
Then I killed the application that had an opened file. This resulted
in a NFS CLOSE being sent to the server using the machine's gss
context (which is a default behaviour of the linux client regardless
of whether or not user's credentials are valid).

Basically as far as I can tell, a linux client can handle cleaning up
state when user's credentials have expired.
>
>
>
> Andy
>
>
>
>
>