Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <284e5b86-1903-4073-fb3e-de60ad5d8f20@oracle.com>
Date:   Tue, 4 Jul 2023 10:48:42 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.11.2
Subject: Re: nfsd in D state, another case
Content-Language: en-US
To:     Miha Vrhovnik <miha.vrhovnik@visionect.com>,
        linux-nfs@vger.kernel.org
References: <CANbXbTt=smOsgJZHCkk8O0EzB+nBd=a7a40my9w59UKTtka0fg@mail.gmail.com>
From:   dai.ngo@oracle.com
In-Reply-To: <CANbXbTt=smOsgJZHCkk8O0EzB+nBd=a7a40my9w59UKTtka0fg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YXM5WFkxK2Zzd003V1lvZmxZaDViaXpxMVVkdVJhdmYvY0pUQWhFNXk4dHRZ?=
 =?utf-8?B?T2ZLdEtIWG56YlhxWXc1eG9LdjA1TnV3anlveEhjTDFma29RMGlWdmFLcnlw?=
 =?utf-8?B?Y29NNFVkM0krT0VydldTa0hkeCt5VFpMSWpzWUs5cmVjcW9VcUJEUkxjM0t1?=
 =?utf-8?B?ZnhhMGhKQmo1UGpxS3lHcFdTRVkyZGE4Uy9KQlNTL2pwK1ZaY0VGckxlQVp0?=
 =?utf-8?B?Y0J5ejV2RENJWmlja3g5dGY4R0JMbEhmcndUc2dldHZVc0s1RG5TdnFiMTdo?=
 =?utf-8?B?Znl1L0tyeGxOQ3p4Q1AyK2tkakU4U2tCRmFzSGlobTAvK2VVd1J0ZTlzNGhk?=
 =?utf-8?B?TzhaOFUrQTIzUTR3OGNnRStKWStSeGJqQ3FTR3VOb1c4YWdBZ1JvUFU5bnNO?=
 =?utf-8?B?WnZlcnNtMlNJRGZEdjZBaXRoOGt5TURyQ0U2Nm9uUjcvK1ZOMGpEZERmTzU2?=
 =?utf-8?B?akNHajI4ZGJSOE1xelFLRStBMVFYZmorNU8vSVJBTTFRUUR6dXJpSFVXOWpR?=
 =?utf-8?B?K1lVZWdrMmJueVZ4Y29ObGRGN3VaVFVXTGl6bWdjVTcvMnIrWEJBMHp0M3JF?=
 =?utf-8?B?VXIwd01hbXJyREpkNjhrUmJsdzhmdUVVTnJHRHc2Q0hGc05uVVovTU8wM1Zz?=
 =?utf-8?B?N2d5V25oTUJxei9OWXFuTzI2UzBjb1YzN1RReEFvSmFLNHkwekNjb0JlYnRN?=
 =?utf-8?B?SUFkMjlOY2hjbi9MQThZaHEwQ2h5b2RqcHM4NTJrRE5QL1ZKSUYwMmxvU0x2?=
 =?utf-8?B?YlNERnh0T2xHcTZ1MWVVS3BKb3JwYjVoYmRzR2dNUlI2TzAwRGlHYmg1WEtO?=
 =?utf-8?B?cXN1emRBYTBXdEZtK0RnNlZLS2x4SlArUDBPSXIwaERsRzBxZlVieWZxVFRL?=
 =?utf-8?B?YzFHRlc0bGZSQkJEUmtsOGQxV3pjalN3dUdOTy8vZitWVmJwa1djM0V2V25D?=
 =?utf-8?B?OGdvaXZzMWZ5ekJZdCtnSzU0L0RrQjNmd0x3Z1dTWXlWVVgrWWc0VjAzeDFG?=
 =?utf-8?B?dHNUaVMzL0RpVG1kY2RLdWhJMUpHYzU4OWlGZXgrRm5oNUxBeTJTSEFEbjYr?=
 =?utf-8?B?Q21SeWpSV2dJYzFYNG9QS0ZUTTV5L1pNQWpmc0c2b3VDRUQ4TVRaY0IrNDgy?=
 =?utf-8?B?MWpxK21CdGdUMzF6QU5GNit6MVV5OFEySHVwMGxNUmF3OVp2cGRYejdtZ2Jy?=
 =?utf-8?B?MXNCY2p6TnFYVHV1N3hKais5dlZKYmxwM1hRQjQ3aFRtc1FJeURaOWo2QzdK?=
 =?utf-8?B?czduWm5KTG1wUWV2M0NtREtudEtDQTNDTkxNdFhLdnliVkgxMHBrTk5CNDRB?=
 =?utf-8?B?N1hzZjgvNit6RC9zYllpRXI4ZUlWMG45T1VMLzhvU0NUaFRPWDB6cDlzNFN4?=
 =?utf-8?B?UTlxRDZZZTVXdGNFbjNlaGhJLzh5T2pQRkNwdERNdVUxUG9MSVhUbWUyQWtQ?=
 =?utf-8?B?b0FWTFRKaFZpdjlZTk5Mayt0bmUzN3Y2QVBwdzlJcml1Z3c5Sk9NUkVuVVdI?=
 =?utf-8?B?TFB1N01oRmR5bThFUjFtSUNKMXNyL2ZoNmtYUTNCSlZTZVZMWlUyc2RyaGdV?=
 =?utf-8?B?WERPRHd4b1p3WGZvdGlsL3lUenh5eW5KMDZiZE1pNkF1NVAyYzNtVXhSUENB?=
 =?utf-8?B?RGcwNndwSVRMNlNWeDY5N0drNlFBRXJjRUd6Q1NNdlNadldzdmJzbzVJRHhh?=
 =?utf-8?B?a29SQmtSUGhQQ3E1TnNaN2hNTkhLOFN3TnJkazRSYWJwTGVhd01jekV1RFRJ?=
 =?utf-8?B?aUp2VGd6d2hFeVcycHFHME5uSFpZdzBNOHQ3OEg5RzBISXd6OE04bkkzNkxI?=
 =?utf-8?B?SUJCWWVVZ3hSS0ZpTGV1ZmdZYlNTbkVGcXk3ZlZvN2JpOGFvNHFhRGJGcFJF?=
 =?utf-8?B?Y0k0NjNvczNCTXhhaDRVTy9UdXFPclJnSkxJdVJrY2lCV2FsbklYdXppM0R5?=
 =?utf-8?B?bDZGYm9SMTh3eEFaT0xvYnhRamoxZ1VUMy9waUZSUjNIWkVIU0JXMGpBdGdH?=
 =?utf-8?B?NGZpV285MXN0RTJQTDJSK3JybzZYYTgrVGxuT2UyREJNVGVGNG1DQVp2NytQ?=
 =?utf-8?B?TS9VcTdnUEU3VHRGam95QWQzZUd4bGt3VXljcTNwcG5adDFsbzdQTjZJcVdG?=
 =?utf-8?Q?mfGTaTjd9ysfgK9fKnVhQUX2E?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: UO4dQjaHyglXJwK2iEsK9/cGGWPgTNgoWrXPTTwDG5LP72iKp6rPlSZ+uFas/3OszxmheWZ+CIqkmIiGlImcwUrL/Y/QSoGJG+yzndANZvxItZeoo21EpOWGzI7A5rs2MmU7CMSLdh/LSxSOKHsODyR4Phu8Rg1PyuNoMe5ke8mFtROlwdKBEflPv8FQBwHBoWMPp48khBFQiYdqfrLRD6C7EbV6GOnEgrBCKvJmyEiQw7ePU8SwEZU4scMmZWvUwYxg9vTvcQYPynIflWgPyk3HdLG8teeGiHFhYXWPUCXwbdmR0aGcnTfH3nZhz6jTlpJ4YAZuEozYRus4oOI3UORjCOwWg91B6DYf2KFpvkZehN6SjbfiW/3R2nyaUFmVXab+/iKV1A9iy9fVpLDt+ggMrBYTAV6KXnK8zIxa9AiHlgmrLwThQwIRZ60Qr1sBcSJ6Y7nlwWX0PwxhVw5yorkDxJwxoDl66sqzEaOk87N0cjc1ho4i6ipVK3a8U0vOu4cd6eGPMRy+i8CbR2qWjJGCP7EDHrpEcQJb3/fnNmPcD+dUwAzGFQXE6mDNx0S9GaMJOy2qkIieoOgJhkdJAxIPM4PLAnu84/9GQ8r+RaL8YFu6Sa2bRqnOb4mg/qVIsuy3Xf9aNWOF+qjk9pjpOOxFXLY2b6HIVITsM1jbNifPDMtwTxi6diz629WO/xR0gw3/wE32YKt5Nqrozx+sMKhwDQetDwwGwW6Y0y78XMiT7LIP0zukhvQ5SNR3JwzHWJVzyLNVRF32kdfxf13ULD1f5GfvcLDEcXagBmwMyxg=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d68411f-6725-4c4a-0663-08db7cb6f0d0
X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4270.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 17:48:57.4364
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HbVAOuUMGfPG5WZSzAJ1ixHZE3nLsiag5Cbn/w0kEaFOLh+dNd9j08hH0ci0DDQwBZNovhKu+GbhOrsYxZ9xhg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5040
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-04_12,2023-07-04_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 suspectscore=0
 spamscore=0 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2307040153
X-Proofpoint-GUID: GkFqJq29EqV_IOXwbihcawhmA47B1Z9U
X-Proofpoint-ORIG-GUID: GkFqJq29EqV_IOXwbihcawhmA47B1Z9U
X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Miha,

I don't have any solution for this problem based on the provided data.
However next time when this problem occurs, please do the following on
the NFS server to gather some diagnostic info so we can investigate:

1. dump the workqueues:

#  echo t > /proc/sysrq-trigger

Output is written to /var/log/messages.

2. gather nfsd and rpc task info:

# (for tt in $(grep -l 'nfs' /proc/*/stack); do echo "${tt}:"; cat ${tt}; echo; done) > /tmp/nfs_threads.txt

# cat /sys/kernel/debug/sunrpc/rpc_clnt/*/tasks > /tmp/rpc_tasks.txt


You can put /var/log/messages, nfs_threads.txt and rpc_tasks.txt at some
public place where we can get to, or you can email them directly to me.

Thanks,
-Dai

On 7/3/23 2:05 AM, Miha Vrhovnik wrote:
> Hey everybody,
>
> we are experiencing a similar issue to Dr. Herzog [1]. the details and
> our configuration is a bit different than his, but I'd assume that the
> same bug is behind this.
>
> It's a single server per region, with NFS is on local storage formatted as ext4.
> The servers don't have a swap and have plenty of free memory. 16G-20G
> used (without caches) out of 64G.
> The amount of data stored is rather small about 1G, with about 100.000
> files over 17.000 directories
>
> We've upgraded the servers from Ubuntu 20.04 to Ubuntu 22.04 about a
> month ago and we had four outages since than.Our clients are still on
> Ubuntu 20.04 if this helps
> Our setup is rather small.But the impacts aprox. 10.000 users per region.
>
> And the packages in Ubuntu 22.04 are a bit older than the ones in
> Debian 12 (bookworm )
>
> apt list --installed '*nfs*'
> Listing... Done
> libnfsidmap1/jammy-updates,now 1:2.6.1-1ubuntu1.2 amd64 [installed,automatic]
> nfs-common/jammy-updates,now 1:2.6.1-1ubuntu1.2 amd64 [installed]
> nfs-kernel-server/jammy-updates,now 1:2.6.1-1ubuntu1.2 amd64 [installed]
>
> and the kernel
> Linux redacted 5.19.0-1026-gcp #28~22.04.1-Ubuntu SMP Tue Jun 6
> 07:24:26 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
>
> Stack-trace is a different from the one provided by Dr. Herzog. But
> the same between servers.
> so only full server restart helps. And once this happened almost
> immediately after the server was rebooted.
>
> This is the first time that it happened that day.
>
> Jun 26 11:34:07 redacted kernel: [326687.684554] receive_cb_reply: Got
> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000000dddae84 xid
> bb78d292
> Jun 26 11:34:07 redacted kernel: [326687.684634] receive_cb_reply: Got
> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000000dddae84 xid
> ba78d292
> Jun 26 11:41:38 redacted kernel: [327139.472177] receive_cb_reply: Got
> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004df70d45 xid
> 3bfdc2ea
> Jun 26 11:44:30 redacted kernel: [327310.678539] INFO: task nfsd:848
> blocked for more than 120 seconds.
> Jun 26 11:44:30 redacted kernel: [327310.685086]       Not tainted
> 5.19.0-1026-gcp #28~22.04.1-Ubuntu
> Jun 26 11:44:30 redacted kernel: [327310.691358] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 26 11:44:30 redacted kernel: [327310.699442] task:nfsd
> state:D stack:    0 pid:  848 ppid:     2 flags:0x00004000
> Jun 26 11:44:30 redacted kernel: [327310.699451] Call Trace:
> Jun 26 11:44:30 redacted kernel: [327310.699456]  <TASK>
> Jun 26 11:44:30 redacted kernel: [327310.699464]  __schedule+0x248/0x5d0
> Jun 26 11:44:30 redacted kernel: [327310.699478]  schedule+0x58/0x100
> Jun 26 11:44:30 redacted kernel: [327310.699482]  schedule_timeout+0x10d/0x140
> Jun 26 11:44:30 redacted kernel: [327310.699489]  __wait_for_common+0x99/0x1b0
> Jun 26 11:44:30 redacted kernel: [327310.699493]  ? usleep_range_state+0xa0/0xa0
> Jun 26 11:44:30 redacted kernel: [327310.699497]  wait_for_completion+0x24/0x40
> Jun 26 11:44:30 redacted kernel: [327310.699501]  __flush_workqueue+0x140/0x3e0
> Jun 26 11:44:30 redacted kernel: [327310.699512]
> nfsd4_probe_callback_sync+0x1a/0x30 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699566]
> nfsd4_destroy_session+0x184/0x230 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699597]
> nfsd4_proc_compound+0x42b/0x770 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699629]
> nfsd_dispatch+0x174/0x270 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699670]
> svc_process_common+0x2a5/0x610 [sunrpc]
> Jun 26 11:44:30 redacted kernel: [327310.699760]  ?
> svc_handle_xprt+0x166/0x350 [sunrpc]
> Jun 26 11:44:30 redacted kernel: [327310.699807]  ? nfsd_svc+0x1a0/0x1a0 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699835]  ?
> nfsd_shutdown_threads+0xa0/0xa0 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699863]
> svc_process+0xba/0x110 [sunrpc]
> Jun 26 11:44:30 redacted kernel: [327310.699906]  nfsd+0xd1/0x1a0 [nfsd]
> Jun 26 11:44:30 redacted kernel: [327310.699932]  kthread+0xce/0xf0
> Jun 26 11:44:30 redacted kernel: [327310.699937]  ?
> kthread_complete_and_exit+0x20/0x20
> Jun 26 11:44:30 redacted kernel: [327310.699941]  ret_from_fork+0x1f/0x30
> Jun 26 11:44:30 redacted kernel: [327310.699948]  </TASK>
>
>
>
> This is about 10 minutes after reboot:
>
> Jun 26 12:45:13 redacted kernel: [  373.274344] receive_cb_reply: Got
> unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000016b8f89c xid
> b4752784
> Jun 26 12:49:06 redacted kernel: [  605.568257] INFO: task nfsd:887
> blocked for more than 120 seconds.
> Jun 26 12:49:06 redacted kernel: [  605.574821]       Not tainted
> 5.19.0-1026-gcp #28~22.04.1-Ubuntu
> Jun 26 12:49:06 redacted kernel: [  605.581432] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 26 12:49:06 redacted kernel: [  605.589914] task:nfsd
> state:D stack:    0 pid:  887 ppid:     2 flags:0x00004000
> Jun 26 12:49:06 redacted kernel: [  605.589926] Call Trace:
> Jun 26 12:49:06 redacted kernel: [  605.589931]  <TASK>
> Jun 26 12:49:06 redacted kernel: [  605.589937]  __schedule+0x248/0x5d0
> Jun 26 12:49:06 redacted kernel: [  605.590112]  schedule+0x58/0x100
> Jun 26 12:49:06 redacted kernel: [  605.590117]  schedule_timeout+0x10d/0x140
> Jun 26 12:49:06 redacted kernel: [  605.590125]  __wait_for_common+0x99/0x1b0
> Jun 26 12:49:06 redacted kernel: [  605.590129]  ? usleep_range_state+0xa0/0xa0
> Jun 26 12:49:06 redacted kernel: [  605.590134]  wait_for_completion+0x24/0x40
> Jun 26 12:49:06 redacted kernel: [  605.590138]  __flush_workqueue+0x140/0x3e0
> Jun 26 12:49:06 redacted kernel: [  605.590147]
> nfsd4_probe_callback_sync+0x1a/0x30 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590199]
> nfsd4_destroy_session+0x184/0x230 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590235]
> nfsd4_proc_compound+0x42b/0x770 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590270]
> nfsd_dispatch+0x174/0x270 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590303]
> svc_process_common+0x2a5/0x610 [sunrpc]
> Jun 26 12:49:06 redacted kernel: [  605.590392]  ?
> svc_handle_xprt+0x166/0x350 [sunrpc]
> Jun 26 12:49:06 redacted kernel: [  605.590447]  ? nfsd_svc+0x1a0/0x1a0 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590479]  ?
> nfsd_shutdown_threads+0xa0/0xa0 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590510]  svc_process+0xba/0x110 [sunrpc]
> Jun 26 12:49:06 redacted kernel: [  605.590561]  nfsd+0xd1/0x1a0 [nfsd]
> Jun 26 12:49:06 redacted kernel: [  605.590598]  kthread+0xce/0xf0
> Jun 26 12:49:06 redacted kernel: [  605.590603]  ?
> kthread_complete_and_exit+0x20/0x20
> Jun 26 12:49:06 redacted kernel: [  605.590608]  ret_from_fork+0x1f/0x30
> Jun 26 12:49:06 redacted kernel: [  605.590616]  </TASK>
>
> when things started to go really bad we gt another stacktrace:
> Jun 26 12:53:07 redacted kernel: [  847.232530] INFO: task
> kworker/u8:3:5078 blocked for more than 120 seconds.
> Jun 26 12:53:07 redacted kernel: [  847.239937]       Not tainted
> 5.19.0-1026-gcp #28~22.04.1-Ubuntu
> Jun 26 12:53:08 redacted kernel: [  847.246163] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 26 12:53:08 redacted kernel: [  847.254308] task:kworker/u8:3
> state:D stack:    0 pid: 5078 ppid:     2 flags:0x00004000
> Jun 26 12:53:08 redacted kernel: [  847.254328] Workqueue: nfsd4
> laundromat_main [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254438] Call Trace:
> Jun 26 12:53:08 redacted kernel: [  847.254443]  <TASK>
> Jun 26 12:53:08 redacted kernel: [  847.254449]  __schedule+0x248/0x5d0
> Jun 26 12:53:08 redacted kernel: [  847.254460]  ? available_idle_cpu+0x66/0xa0
> Jun 26 12:53:08 redacted kernel: [  847.254467]  schedule+0x58/0x100
> Jun 26 12:53:08 redacted kernel: [  847.254472]  schedule_timeout+0x10d/0x140
> Jun 26 12:53:08 redacted kernel: [  847.254478]  __wait_for_common+0x99/0x1b0
> Jun 26 12:53:08 redacted kernel: [  847.254482]  ? usleep_range_state+0xa0/0xa0
> Jun 26 12:53:08 redacted kernel: [  847.254485]  wait_for_completion+0x24/0x40
> Jun 26 12:53:08 redacted kernel: [  847.254489]  __flush_workqueue+0x140/0x3e0
> Jun 26 12:53:08 redacted kernel: [  847.254494]
> nfsd4_shutdown_callback+0x4d/0x130 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254525]  ?
> nfsd4_return_all_client_layouts+0xc9/0x160 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254556]  ?
> nfsd4_shutdown_copy+0x70/0xb0 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254600]
> __destroy_client+0x1a1/0x210 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254630]  expire_client+0x57/0x70 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254659]
> nfs4_laundromat+0x26e/0x900 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254690]  ?
> finish_task_switch.isra.0+0x82/0x290
> Jun 26 12:53:08 redacted kernel: [  847.254697]
> laundromat_main+0x19/0x50 [nfsd]
> Jun 26 12:53:08 redacted kernel: [  847.254726]  process_one_work+0x222/0x3c0
> Jun 26 12:53:08 redacted kernel: [  847.254731]  worker_thread+0x50/0x3f0
> Jun 26 12:53:08 redacted kernel: [  847.254734]  ? process_one_work+0x3c0/0x3c0
> Jun 26 12:53:08 redacted kernel: [  847.254736]  kthread+0xce/0xf0
> Jun 26 12:53:08 redacted kernel: [  847.254742]  ?
> kthread_complete_and_exit+0x20/0x20
> Jun 26 12:53:08 redacted kernel: [  847.254746]  ret_from_fork+0x1f/0x30
> Jun 26 12:53:08 redacted kernel: [  847.254754]  </TASK>
>
>
> 1 - https://marc.info/?l=linux-nfs&m=168258759703478&w=2
>
> Regards,
> Miha