Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
Date:   Wed, 29 Sep 2021 06:06:47 +0200
From:   Salvatore Bonaccorso <bonaccos@ee.ethz.ch>
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     linux-nfs@vger.kernel.org, Chuck Lever <chuck.lever@oracle.com>
Subject: Re: nfsd4_process_open2 failed to open newly-created file!
 status=10008 ; warning at fs/nfsd/nfs4proc.c for nfsd4_open
In-Reply-To: <20210928193925.GH25415@fieldses.org>
References: <20210927061025.GA20892@varda.ee.ethz.ch>
 <20210927155338.GA30593@fieldses.org>
 <cd38d56c6824ac6776df838dfd66bccd@ee.ethz.ch>
 <20210928193925.GH25415@fieldses.org>
User-Agent: Roundcube Webmail/1.4.11
Message-ID: <84751bbb03137f6a578148c988eeb230@ee.ethz.ch>
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Bruce,

On 28.09.2021 21:39, J. Bruce Fields wrote:
> On Tue, Sep 28, 2021 at 07:11:41AM +0200, Salvatore Bonaccorso wrote:
>> Hi Bruce,
>> 
>> On 27.09.2021 17:53, J. Bruce Fields wrote:
>> >On Mon, Sep 27, 2021 at 08:10:31AM +0200, Salvatore Bonaccorso wrote:
>> >>We recently got the following traces on a NFS server, but I'm not sure
>> >>how to further debug this, any hints?
>> >
>> >The server creates and opens a file in two steps, though it should
>> >really be a single atomic operation.
>> >
>> >That means there's a small possibility somebody could intervene and do
>> >something like change the permissions:
>> >
>> >>
>> >>[5746893.904448] ------------[ cut here ]------------
>> >>[5746893.910050] nfsd4_process_open2 failed to open
>> >>newly-created file! status=10008
>> >
>> >10008 is NFS4ERR_DELAY, so maybe somebody managed to get a delegation
>> >before we finished opening?
>> >
>> >We should be able to prevent that....
>> >
>> >In your setup are there processes quickly opening new files created by
>> >others?
>> 
>> This is very possible. The NFS server is used as a "scratch" place
>> accessible from
>> compute cluster where people can have multiple jobs simultaneously
>> running through
>> Slurm and accessing the data. So it is possible that user create new
>> files from
>> one running instance and accessing it quickly from the other nodes.
>> 
>> I'm so far was unable to arificially trigger the issue but is there
>> anything I
>> can try out to get more information useful for you?
> 
> I think the problem's pretty obvious.  I'm not sure what the fix should
> be.

Hope you come up with the right solution! Thanks for looking into it,
much appreciated.

> You can work around it for now by turning off delegations (echo 0
>> /proc/sys/fs/leases_enable before starting nfsd).

Ack thank you.

Regards,
Salvatore