Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
From:   "NeilBrown" <neilb@suse.de>
To:     "Donald Buczek" <buczek@molgen.mpg.de>
Cc:     "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
        linux-fsdevel@vger.kernel.org
Subject: Re: Heisenbug: I/O freeze can be resolved by cat $task/cmdline of
 unrelated process
In-reply-to: <77184fcc-46ab-4d69-b163-368264fa49f7@molgen.mpg.de>
References: <77184fcc-46ab-4d69-b163-368264fa49f7@molgen.mpg.de>
Date:   Mon, 06 Nov 2023 08:51:11 +1100
Message-id: <169922107188.24305.7903791112230110428@noble.neil.brown.name>
Precedence: bulk

On Sun, 05 Nov 2023, Donald Buczek wrote:
....
>=20
>      for task in /proc/*/task/*; do
>          echo  "# # $task: $(cat $task/comm) : $(cat $task/cmdline | xargs =
-0 echo)"
>          cmd cat $task/stack
>      done
>=20
> which can further be reduced to
>=20
>      for task in /proc/*/task/*; do echo $task $(cat $task/cmdline | xargs =
-0 echo); done
>=20
> This is absolutely reproducible. Above line unblocks the system reliably.
>=20
> Another remarkable thing: We've modified above code to do the
> processes slowly one by one and checking after each step if I/O
> resumed.  And each time we've tested that, it was one of the 64 nfsd
> processes (but not the very first one tried).  While the systems
> exports filesystems, we have absolutely no reason to assume, that any
> client actually tries to access this nfs server.  Additionally, when
> the full script is run, the stack traces show all nfsd tasks in their
> normal idle state ( [<0>] svc_recv+0x7bd/0x8d0 [sunrpc] ).
>=20
> Does anybody have an idea, how a `cat /proc/PID/cmdline` on a specific
> assumed-to-be-idle nfsd thread could have such an "healing" effect?

/proc/PID/cmndline for an nfsd thread is empty.  So it probably isn't
accessing 'cmdline' specifically that unblocks, but any (or almost any)
proc file for the process might help.

You say that *after* accessing cmdline, the "stack" file shows a normal
stack trace.  It might be interesting to see if that same stack is
present *before* accessing cmdline.  But my guess is that nfsd is mostly
a distraction.

It would help to see the fully "echo t > /proc/sysrq-trigger" list of all
process stacks.  That should reveal where the blockage is.

NeilBrown


>=20
> I'm well aware, that, for example, a hardware problem might result in
> just anything and that the question might not be answerable at all.
> If so: please excuse the noise.
>=20
> Thanks
> Donald
> --=20
> Donald Buczek
> buczek@molgen.mpg.de
> Tel: +49 30 8413 1433
>=20
>=20