Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Wang Yugui <wangyugui@e16-tech.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: a dead lock of 'umount.nfs4 /nfs/scratch -l'
Thread-Topic: a dead lock of 'umount.nfs4 /nfs/scratch -l'
Thread-Index: AQHZJZtceQwIxRwSHkuVtnXVMyL6FK6Y9OQAgAGQ/4CAAekjAA==
Date:   Fri, 13 Jan 2023 14:41:28 +0000
Message-ID: <5B9215E4-99FF-4C52-901F-8D909DD165BD@oracle.com>
References: <20230111165945.7605.409509F4@e16-tech.com>
 <20230111173534.82A7.409509F4@e16-tech.com>
 <20230112173046.82E4.409509F4@e16-tech.com>
In-Reply-To: <20230112173046.82E4.409509F4@e16-tech.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Qbxj23ShH7vk3o49Nnp+n6T3yx8zOgPd2dVv0CYE4Kf7fRvDHnzpiHSlAOmf?=
 =?us-ascii?Q?vDy0Mn+M+WmExNmarYICCJZRLIDmFr0LcVAl4GzEf1blZ0ibizLJHveUOA0V?=
 =?us-ascii?Q?z3ewDw+RIKtl+eYrczmTHY6DFy1UH3DtB6cWStrYcisH2GXS/TdRvBOOMtfJ?=
 =?us-ascii?Q?YA/OMLRuXjf4CZuqS5GeDAnPJ5SKLxxjqKxZEhnelOZkYjoTe/Q2t2u0OU3Z?=
 =?us-ascii?Q?/J46UEBbFWzwWxveJFu0tKRwfIxbhwmwZ9vnFttiUgWtT+rLAySnRrk6g87K?=
 =?us-ascii?Q?HofzPNxUCbSNaCv7FjAiw5dfC+NexgyVTvxO+uRfbmZmeE+4yT/AH0OmpA7R?=
 =?us-ascii?Q?b+hw7z1sM+MrDvF1hJ8WBWDF+L5umGHK50bAjmeAAJSxSihzDfaUAAzJRDwE?=
 =?us-ascii?Q?9oIhqR2g1IcGOjKWFPJY4wTVfFC6SluqYooeS6dBs6vxh99aJBd7tTwYK0Vd?=
 =?us-ascii?Q?9bDy5TyyQFliIkIQgHUDL8cHdV5OCK/IL3Q6xcutqRtQ5NfLjoHwmS68xdH3?=
 =?us-ascii?Q?SYBPMeQLRaWicR45CpkSaYSERenAKF8lyHNj7RJdcTd07S9bNItHLOiEaRjZ?=
 =?us-ascii?Q?yGgSRx++XN2xOMtuiZb1lZmvGyE2UMlaUF+ynyDdSM6dyWHfJTa+qEINEy7e?=
 =?us-ascii?Q?52y1GELeaPxqDotigfXQxZkvM2QoJ+qf7OCaNonrpdk/whwRS0UXvywy2qtu?=
 =?us-ascii?Q?aFh0851n+kh4rZDb/WPXrZlgJ7665Y4XgBu+r8Jraz+Qof8bwWwKAKU8edY9?=
 =?us-ascii?Q?5GYm3/zxCasW4cI4W443zUy5yzXp7HbYaNZdZG9b7ycjRhrBXmA67EYgRSpb?=
 =?us-ascii?Q?EGKA51559NfvpyH0DqO26RP6Ua1FvDNTvM6ir+RgHMkOAOCqBAdBcixNR4I8?=
 =?us-ascii?Q?k3D9cethdTvP/HyU/GilPEmVA/ARWgJeuJW1Y8E/70eYE6r1tT+UXE04j3Wh?=
 =?us-ascii?Q?q9RhmRtB/JnUHK23sN3H0aY0Q51QdYLGIRqfMQACUhut7YdHjDn7/KNtgfZa?=
 =?us-ascii?Q?xuWy+sLaUmrdDA3zHKYRFRAFqazx8MsqiwD68/3rXujoCEZfSEKepGA9mnD5?=
 =?us-ascii?Q?vOi/klRz5kkbZD7rEvk2zgvDiphsLESLDcRiy8i2PJvW83uYFnzIUfdnGnsA?=
 =?us-ascii?Q?02nZQQ8D9B5Bqo+DhWFRbI61Did9d1lLKZzfh70cR0l2e5Wl2aUvK0TYURq0?=
 =?us-ascii?Q?7/LcitQkje3NYYZp6Zwc7fdq3z2NR7brtPVdc35JZUe6OmX+LmOWVin3NfMm?=
 =?us-ascii?Q?Av9h1yPzgaBIx352ImAaO3Dsu5hyPT5Ti14d9g2nIU4J2QrQDx14wkfTEULc?=
 =?us-ascii?Q?LZmZZwm7mvGgYa1naG8Z9S2qXm/lHRkNTc+WAX5EckwDGnp5/w/IT/TXEDZu?=
 =?us-ascii?Q?ans35UXSquQenTPqdKckXfrZe/y3JhIRty7FeqVHxwRTKn0nIqj5Rsi5J2FT?=
 =?us-ascii?Q?Lspd8CGRLy86I54AfyAI6Wy9s3mxo89nrAbxksb9+0bXBXO2z5IPnE2YLkBT?=
 =?us-ascii?Q?h/C7K0QP+BXVz33AsqNaPvUzwXMRPm7rQ6OPxheuMOMPp7MPLk123/xsBlj+?=
 =?us-ascii?Q?Do3JRu2v11+7izeCEEtfgQK5q0s2AARfPjSJ+tNNazl+3aoB6JlJJPgjNlZY?=
 =?us-ascii?Q?sA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2CF60F3AF0879D4DADDEED32E466EEBA@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: wzh8FHJoLpdokNspfKmnTShmKpYVVgActyVXKXqJWnlaxRdxFHgEE0qwMy0jtdZH2wkoFLBa0BduGURy8tltLEh2K+qUj4N/UfMj7n9bxgxD8raJK3cR0vXMPYAimK4Cpp7UiV+eiGxs4T7l/ismKneMXRd98O71ZmSwYq/MlUo9PaXobrWos/Skb//PMdMqL4hW5fL3sa/pTbfr5Q22cRH4rQZTou8tu+++6l5ZTe0c0DO8JifjnRqP0I+gp8OcwQu3lxoHyDPGiFoWCPDvV3l659Lmy3Qjt+WAUDiNsqB1GsHWGfVc7v37GVrmrlXjzAJNroSSBBr3wljVaG1/Qf9dKD6ORMmJXjid/f+QLNaPmsAaM0MR21XHOcgtOgpDQSIqt66GppUAjOExVmTGXA58o1BwVaAQ+9jf45QqzwVynlpfWxA0/nR6RrSxyOwcOadKhlDjPHBSt+4wOMWP8vlVZJFbJSdyjva0btRds01FmBUJXfARm/gz7MZNWhINjA4T+Vf/T/PlNrBop9Z0s3OlQI5xKBgFv2LDPOiBcGGVZ96keGuvHfQ5B/FujCHwcZ4k80U7fX95JQd3HNhnC+6PB/pAhfn+juuZ+1LedoMctYR91Lxu3wagsjYZINJMfBATxtzJfL3twvOhR4q9dG1q6BMJnAAEjFodOLgdD0nPLEc0f6bDo7IQYrb4q3viXF5WN1HO4LQzo2PeQGWEG7L/TP40MrlKLAYM5owI3SXVo541TZ9ggaLwd8NHUk6CI1hbV9effL9dMZbKmzQiGCVbppvDYa3whEaU8zz4tWs=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c6f3b76f-0846-4915-0cde-08daf5744171
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Jan 2023 14:41:28.8973
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: QS1TbzV0T6HIZMpvTg2jCOeLIj37FzAh59ZkqwURwasHH7jjeXY0uMx022xzNyxrL3vhq/ltVAlwpJVOmzQH+g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB7273
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.923,Hydra:6.0.562,FMLib:17.11.122.1
 definitions=2023-01-13_06,2023-01-13_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999
 malwarescore=0 mlxscore=0 suspectscore=0 phishscore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2301130096
X-Proofpoint-GUID: ULGhnzGlKJLxLsCiHwydtq0EbBw4n2Uv
X-Proofpoint-ORIG-GUID: ULGhnzGlKJLxLsCiHwydtq0EbBw4n2Uv
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 12, 2023, at 4:30 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>=20
> Hi,
>=20
>> Hi,
>>=20
>>> Hi,
>>>=20
>>> We noticed a dead lock of 'umount.nfs4 /nfs/scratch -l'
>>=20
>> reproducer:
>>=20
>> mount /dev/sda1 /mnt/test/
>> mount /dev/sda2 /mnt/scratch/
>> systemctl restart nfs-server.service
>> mount.nfs4 127.0.0.1:/mnt/test/ /nfs/test/
>> mount.nfs4 127.0.0.1:/mnt/scratch/ /nfs/scratch/
>> systemctl stop nfs-server.service
>> umount -l /nfs/scratch #OK
>> umount -l /nfs/test #dead lock
>>=20
>> Best Regards
>> Wang Yugui (wangyugui@e16-tech.com)
>> 2023/01/11
>>=20
>>> kernel: 6.1.5-rc1
>=20
> This problem happen on kernel 6.2.0-rc3+(upstream) too.

Can you clarify:

- By "deadlock" do you mean the system becomes unresponsive, or that
  just the mount is stuck?

- Can you reproduce in a non-loopback scenario: a separate client and
  server?


> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2023/01/12
>=20
>>>=20
>>> The dmesg output of 'sysrq w'
>>>=20
>>> [13493.955032] sysrq: Show Blocked State
>>> [13493.959997] task:umount.nfs4     state:D stack:0     pid:3542745 ppi=
d:3542744 flags:0x00004000
>>> [13493.969628] Call Trace:
>>> [13493.973003]  <TASK>
>>> [13493.976018]  __schedule+0x2cb/0x880
>>> [13493.980426]  ? __bpf_trace_svc_stats_latency+0x10/0x10 [sunrpc]
>>> [13493.987342]  ? rpc_destroy_wait_queue+0x10/0x10 [sunrpc]
>>> [13493.993637]  schedule+0x50/0xc0
>>> [13493.997697]  rpc_wait_bit_killable+0xd/0x60 [sunrpc]
>>> [13494.003671]  __wait_on_bit+0x75/0x90
>>> [13494.008168]  out_of_line_wait_on_bit+0x91/0xb0
>>> [13494.013547]  ? sched_core_clone_cookie+0x90/0x90
>>> [13494.019101]  __rpc_execute+0x14b/0x490 [sunrpc]
>>> [13494.024603]  ? kmem_cache_alloc+0x41/0x530
>>> [13494.029610]  rpc_execute+0xc5/0x100 [sunrpc]
>>> [13494.034835]  rpc_run_task+0x14b/0x1b0 [sunrpc]
>>> [13494.040252]  rpc_call_sync+0x50/0xa0 [sunrpc]
>>> [13494.045566]  nfs4_proc_destroy_session+0x80/0x100 [nfsv4]
>>> [13494.051926]  nfs4_destroy_session+0x24/0x90 [nfsv4]
>>> [13494.057767]  nfs41_shutdown_client+0xfd/0x120 [nfsv4]
>>> [13494.063774]  nfs4_free_client+0x21/0xb0 [nfsv4]
>>> [13494.069240]  nfs_free_server+0x44/0xb0 [nfs]
>>> [13494.074418]  nfs_kill_super+0x2b/0x40 [nfs]
>>> [13494.079490]  deactivate_locked_super+0x2c/0x70
>>> [13494.084811]  cleanup_mnt+0xb8/0x140
>>> [13494.089147]  task_work_run+0x6a/0xb0
>>> [13494.093587]  exit_to_user_mode_prepare+0x1b9/0x1c0
>>> [13494.099232]  syscall_exit_to_user_mode+0x12/0x30
>>> [13494.104717]  do_syscall_64+0x67/0x80
>>> [13494.109125]  ? syscall_exit_to_user_mode+0x12/0x30
>>> [13494.114799]  ? do_syscall_64+0x67/0x80
>>> [13494.119426]  ? do_syscall_64+0x67/0x80
>>> [13494.124042]  ? do_syscall_64+0x67/0x80
>>> [13494.128649]  ? exc_page_fault+0x64/0x140
>>> [13494.133400]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
>>> [13494.139306] RIP: 0033:0x7fc32f839e9b
>>> [13494.143726] RSP: 002b:00007ffe670f6018 EFLAGS: 00000202 ORIG_RAX: 00=
000000000000a6
>>> [13494.152183] RAX: 0000000000000000 RBX: 000055f4aad71920 RCX: 00007fc=
32f839e9b
>>> [13494.160218] RDX: 0000000000000003 RSI: 0000000000000002 RDI: 000055f=
4aad72600
>>> [13494.168237] RBP: 0000000000000002 R08: 0000000000000007 R09: 000055f=
4aad71010
>>> [13494.176277] R10: 00007fc32fbc0bc0 R11: 0000000000000202 R12: 000055f=
4aad72600
>>> [13494.184313] R13: 00007fc33025f244 R14: 000055f4aad71a30 R15: 000055f=
4aad71b50
>>> [13494.192334]  </TASK>
>>>=20
>>> Best Regards
>>> Wang Yugui (wangyugui@e16-tech.com)
>>> 2023/01/11

--
Chuck Lever