Received-SPF: pass (google.com: domain of linux-nfs+bounces-3350-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Precedence: bulk
MIME-Version: 1.0
Date: Thu, 23 May 2024 14:12:02 +0200
From: Richard Kojedzinszky <richard+debian+bugreport@kojedz.in>
To: linux-nfs@vger.kernel.org, 1071501@bugs.debian.org
Subject: Re: Linux NFS client hangs in nfs4_lookup_revalidate
In-Reply-To: <0473c552b6fd8e96ef2ffbf0435a7552@kojedz.in>
References: <0473c552b6fd8e96ef2ffbf0435a7552@kojedz.in>
Message-ID: <73e081764d06746be27c5f0d2f181938@kojedz.in>
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 8bit

Dear devs,

Now bisecting turned out that 3c59366c207e4c6c6569524af606baf017a55c61 
is the bad commit for me. Strangely it only affects my dovecot process 
accessing data over NFS.

Can you please confirm that this may be a bad commit?

My earlier attached programs may be used to demonstrate/trigger the 
issue. It even could be stripped down to minimal operations to trigger 
the bug.

Thanks in advance,
Richard


2024-05-23 09:10 időpontban Richard Kojedzinszky ezt írta:
> Dear NFS developers,
> 
> I am running multiple PODs on a Kubernetes node, they all mount 
> different NFS shares from the same nfs server. I started to notice 
> hangups in my dovecot process after I switched to Debian's kernel from 
> upstream 5.15. You can find Debian bugreport at 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071501.
> 
> So, effectively I am running dovecot in Kubernetes, and dovecot's data 
> directory is accessed over NFS. Eventually one dovecot process stucks 
> in nfs4_lookup_revalidate(). From that point, that process cannot be 
> killed, howewer, other processes can access NFS as normal. Also, 
> another dovecot process running on the very same node accessing the 
> same NFS share works too.
> 
> Now, I am still in the process of bisecting, howewer, I cannot reliably 
> trigger the bug. Originally it took a few days after I've noticed a 
> hanging process. Now I am trying to mimic file operations what dovecot 
> does in a faster way. Now it seems that it triggers the bug in a few 
> hours, howewer, during bisects, I can still make mistakes.
> 
> I've scheduled many of my applications which use NFS shares to the same 
> node, to have more NFS load on that node.
> 
> I am attaching my simple app which triggers the bug in a few hours, at 
> least in my lab. I have two dedicated NFS shares for this test case, 
> and I am running 3 instances of the applications for both shares. Also, 
> I am running other production applications on the same node which also 
> use NFS, howewer, I dont experience lockups with them. They are 
> librenms, prometheus, and a docker private registry. This way I dont 
> know if running the attached app only is enough to trigger the bug.
> 
> Once I have a suspectible commit based on my bisecting process, I will 
> report it here.
> 
> My NFS server is a TrueNAS, based on FreeBSD 13.3.
> 
> Thanks in advance,
> Richard