Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
Message-ID: <151d80cb-33dc-4266-a3ba-4b89ea80e49f@oracle.com>
Date:   Sun, 5 Nov 2023 11:27:53 -0800
User-Agent: Mozilla Thunderbird
Content-Language: en-US
From:   dai.ngo@oracle.com
To:     Trond Myklebust <trondmy@hammerspace.com>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: General protection fault in nfs4_setup_sequence caused by delegation
 return task
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dVBNTGtoQ2tPejMrSzBrR0RqdEVHRUlqRWFRTWMxbWthNEZGeEFFZlVRV05X?=
 =?utf-8?B?TEV3cGtxMmc2QXVFWEVmOWhXVm5GU2xoelpRZmNUWW1mbVVad2FRdkxCSFgv?=
 =?utf-8?B?MmJzaUR4QnMzRzhsQ3RxQ3BZR2F5bmJuRVZSaG1pYTY0ZHpCbTNneklYN3FN?=
 =?utf-8?B?dzIrTXVTbEVlRkZxTWVQdC8vVXMzOFlvK2ZQSkQ2MlZqS2VIZTVUMFRQTW9L?=
 =?utf-8?B?aWZXbjk5QWVnS3NFSkY5UW9XMHlBMkFxSnBKK0g0ajdUbDA5RGVXQWg3RWpC?=
 =?utf-8?B?MGR6WHZWZXpmZmJ6VHFYU05TTkZhcjF3amVZMHlCZmNhMC9yUWNyMWd6THM1?=
 =?utf-8?B?aGFBd2lsUFgrS1JJYW9GSXc2Uy9LTXZobnd1R29lc2ZlT29qZ2U3Q25zbGcy?=
 =?utf-8?B?ZnFBak5rd2lQQzNvcUpYaFpQS3FBTzN4L0EwWEFLZlNwdlQ1UFp2bVNsWVN0?=
 =?utf-8?B?aEprNkY1WjNUbHJsWkcxTWRaOE9aS1R1MHIzYzhyTU1Ra0FKbUUwdi9HbUcx?=
 =?utf-8?B?eVF3Ym9GMnRrTDdLUFhrTXlxNHR6SnNVVDN0Vk16eEdFOXFSMjNQam9NN3BT?=
 =?utf-8?B?SGJJVVFTTU1oVTcxUVEyWld1Q0NueUo5VitQZlRkNC81UlpmakZUb0x0R2ll?=
 =?utf-8?B?VFFSTmpkeWVjK0xsM2JXMnBPVC9NMTZrbXJqeGU5djVYeXcrdmxmdnVDNlZv?=
 =?utf-8?B?S3lkL2drL2hUR0NkcmdBQU92cHc1Wmd2MTBkUysrQmZwUkIvSnc3RnoxSEJ2?=
 =?utf-8?B?L1hCeDF2SFNTUTQ1ZldMM1pnOHhmTTFoeGtpSGM2MHErVk9wWXBKUVFKaTRw?=
 =?utf-8?B?YjRMMys2U3RwRkFwNkIxWGJDa3R0OFlzQS83akZ5QnlIYVZWaVVjMTUwb3l0?=
 =?utf-8?B?WXFaa0p6NHRXcDBJdCtrRDhrcWQ3NWNVL3VOT1dmTzdIWnJXSm1yekYvK0tp?=
 =?utf-8?B?cnVwZVJvc05YdUpUdmtqOFFiK2FKaUh0MGxlbnBtTnVXbGJBL2RTQ0VtTzEx?=
 =?utf-8?B?Y1hWenFUMmR3bjVGNEUrM0hkK2QwdWNJTHMxRkM1MndOSVM5ek41V01zcld3?=
 =?utf-8?B?eDBIWmdqWWF1VGlNdDZhTWRHa1I5eFlrWFZUS3FnT0ZPV2QzZDFwNmdkeEc4?=
 =?utf-8?B?NzdTcVNoRmhaVVZ6eDdFTGpuemY5WnlJdmFuZlA0ZmVpMEd5NWFrZmp2cmky?=
 =?utf-8?B?QjhEVWpGVXJkSUxJMWN4bjN3Rm44RzVXQXJBd0xhVTZEcGJhd3pNdW1xdGZT?=
 =?utf-8?B?UlFoaytpa05FTHVMSUtPMHp1Q3o4VExObDg3OGoyMUUrVmttYkdHWGlJZVR4?=
 =?utf-8?B?NkFEd01OajRERFB6TVA5a1ZwTjFLR0NwZ2lNSVBvRGNKY1VVWC8rVU96WGZZ?=
 =?utf-8?B?TFBNdEh2OE5BdTVHQ1BwMzI5WmJTOFptVWpScHVJbk42VXFzZTNUSnhXSFZ5?=
 =?utf-8?B?YTNaUlk4ZUFuY1lXZ0JjSHFBeWJScUpOcllYV1ZaQkdHVTBoVk84dUZQZ1da?=
 =?utf-8?B?dy9KdXpFQ0FSYUE4TUN4YVl2R2dIWmVQV3RlWFFON1MyUERYWE1oR3JSckQw?=
 =?utf-8?B?NEZjdXVEUVh3aHQycDhWNjFlVUk2K0hsNmY0UElEbGhRM2pZd0xpS3c5ZUEr?=
 =?utf-8?B?bGpqL1pkeHpId0E0TXRrVElLNVdJN3VQNUkvSC9zd2NBZHFLcExVRFdIVXpt?=
 =?utf-8?B?cHVNam52VElmRlQrUHhla2hVVWZFZEpLZXU1TUhSTzkrQTN1UENnUHpBWmZP?=
 =?utf-8?B?SzEvUmJ6bTg0cSt5YWlwY3poVEN0bG1abWR6LzZkMkdBenRsNmdyM001a2Jj?=
 =?utf-8?B?MUk4RkVaNjdwakE3QUVpc2N3Y0hCbm1KOXFnQlFRTVVrbnR5RkZkVWJjczV4?=
 =?utf-8?B?Ky9kbm1BZGZFKy9iMzdEcWx4N0djSE1GaDkwbFQ0M3htYnZLZ1A4QXMzNTNY?=
 =?utf-8?B?elBTV3NUUEhqYTA0Zk4wbUExVnJiUVJZWDdKakZLT2t1TEtrcm9WOVNyK2h4?=
 =?utf-8?B?T2JkV3VUNkJmMlFpSUcxek9rRFhQSUtRa1I0S2lUSVArMnErL0o5WEhEQUgx?=
 =?utf-8?B?dkZSb2plRVIxS2FvRnRTS04vUzBrQVFUVkdnRXhPVUhmVkVOaXg4M09naDQy?=
 =?utf-8?Q?et81PAaZPj0JXRzc8UiicvBNT?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: zM2PGVgf6of2EyvH7pWntiLQXoOLIeavqwBCFHGZSsU9I5qkNsi9cTwpP+C3QoZHkaQFfBv64/kGQC4OgH3bzgTt5cJe0LxsiwUZ8tb2QP0S5U49nKgxhLQzK+6YRwkOM4SQjwCeg/fFtgqh0O1CsbP5TbSBWnaLBWzNoBdiSwNxCBGneVLSg15KNLU1PVEY/xC50FpjHEpdVj1zgA4EW8AcYEmR0+7vgcG4ID5+0hsVVfRgluLWktkA4PZQ1fd4yfUdLiOgVKZd0sjzJLFpwnl+oR5aAYYYxrnWkFXqXbbeIDVDBeRIMQZsbJN7L2s+lytvkecY/X2gp5/FfK4sSz+WuKKF+1UE8NrLcD3nsOSwKTcvyQC2IGzUqAuFqZS05vzILqsgOU1RJLYgk+8ZAsTfA1c0bC7vmdrWms2dIifIKHrnpU25OSL8ourYXYgXHc7N0jDPjgPXC+kWOaCvEUZg9EC4mzlduFZ6zPlpEu47MA/L1UTJI1/eoZQkycQxPVyUEKjqqckrbR7iMrl9XDDehyxIxn+GlAlV8BcApmcVRCbK3nU1I1oLMxo2OMA2Smh3hj5c9HQleNe2u6JVBqT/NGP588Kq/4hpHvgmDpI0cBUY1z9WRgwP6c666yDkDrjVIJYnWEYvb10oHk94aFd/9Vf+oznJeC8T4/zZqA+6RAtHIBXnrF1oDpBWEEaULjrCq5ah69ophtKYf9h7t5enzlnisBkof49e6riviGKsfKpkZZ9KIkTK8bJTECL4w2KEQgJh7h4hZxjwFhyMBoBgF46jdKSGPf4vSXGiQJCcsv+9keiRcyhmhcLtYTgv
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5fcb33e8-f75b-4056-6c2c-08dbde354fa9
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Nov 2023 19:27:55.5978
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7ctzpHLI4TTU0Pi3xXtKW9H54iHd2JrBNbk8lpT81U+ZcEK2RjjPWfdXRJHbWEKdrD6ReGnA46RJf6M8Rf6lfg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR10MB4183
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-11-05_18,2023-11-02_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0
 mlxlogscore=982 phishscore=0 spamscore=0 suspectscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2310240000
 definitions=main-2311050169
X-Proofpoint-GUID: RrjgKmX1QOML6fJAx7Ix9uWCWjyiYIB7
X-Proofpoint-ORIG-GUID: RrjgKmX1QOML6fJAx7Ix9uWCWjyiYIB7
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Sun, 05 Nov 2023 11:28:57 -0800 (PST)

Hi Trond,

When unmonting a NFS export, nfs_free_server is called. In nfs_free_server,
we call rpc_shutdown_client(server->client) to kill all pending RPC tasks
and wait for them to terminate before continue on to call nfs_put_client.
In nfs_put_client, if the refcounf is drecemented to 0 then we call
nfs_free_client which calls rpc_shutdown_client(clp->cl_rpcclient) to
kill all pending RPC tasks that use nfs_client->cl_rpcclient to send the
request.

Normally this works fine. However, due to some race conditions, if there are
delegation return RPC tasks have not been executed yet when nfs_free_server
is called then this can cause system to crash with general protection fault.

The conditions that can cause the crash are: (1) there are pending delegation
return tasks called from nfs4_state_manager to return idle delegations and
(2) the nfs_client's au_flavor is either RPC_AUTH_GSS_KRB5I or RPC_AUTH_GSS_KRB5P
and (3) the call to nfs_igrab_and_active, from _nfs4_proc_delegreturn, fails
for any reasons and (4) there is a pending RPC task renewing the lease.

Since the delegation return tasks were called with 'issync = 0' the refcount on
nfs_server were dropped (in nfs_client_return_marked_delegations after RPC task
was submited to the RPC layer) and the nfs_igrab_and_active call fails, these
RPC tasks do not hold any refcount on the nfs_server.

When nfs_free_server is called, rpc_shutdown_client(server->client) fails to
kill these delegation return tasks since these tasks using nfs_client->cl_rpcclient
to send the requests. When nfs_put_client is called, nfs_free_client is not
called because there is a pending lease renew RPC task which uses nfs_client->cl_rpcclient
to send the request and also adds a refcount on the nfs_client. This allows
the delegation return tasks to stay alive and continue on after the nfs_server
was freed.

I've seen the NFS client with 5.4 kernel crashes with this stack trace:

!# 0 [ffffb93b8fbdbd78] nfs4_setup_sequence [nfsv4] at ffffffffc0f27e40 fs/nfs/nfs4proc.c:1041:0
  # 1 [ffffb93b8fbdbdb8] nfs4_delegreturn_prepare [nfsv4] at ffffffffc0f28ad1 fs/nfs/nfs4proc.c:6355:0
  # 2 [ffffb93b8fbdbdd8] rpc_prepare_task [sunrpc] at ffffffffc05e33af net/sunrpc/sched.c:821:0
  # 3 [ffffb93b8fbdbde8] __rpc_execute [sunrpc] at ffffffffc05eb527 net/sunrpc/sched.c:925:0
  # 4 [ffffb93b8fbdbe48] rpc_async_schedule [sunrpc] at ffffffffc05eb8e0 net/sunrpc/sched.c:1013:0
  # 5 [ffffb93b8fbdbe68] process_one_work at ffffffff92ad4289 kernel/workqueue.c:2281:0
  # 6 [ffffb93b8fbdbeb0] worker_thread at ffffffff92ad50cf kernel/workqueue.c:2427:0
  # 7 [ffffb93b8fbdbf10] kthread at ffffffff92adac05 kernel/kthread.c:296:0
  # 8 [ffffb93b8fbdbf58] ret_from_fork at ffffffff93600364 arch/x86/entry/entry_64.S:355:0
         
Where the params of nfs4_setup_sequence:
      client = (struct nfs_client *)0x4d54158ebc6cfc01
      args = (struct nfs4_sequence_args *)0xffff998487f85800
      res = (struct nfs4_sequence_res *)0xffff998487f85830
      task = (struct rpc_task *)0xffff997d41da7d00

The 'client' pointer is invalid since it was extracted from d_data->res.server->nfs_client
and the nfs_server was freed.

I've reviewed the latest kernel 6.6-rc7, even though there are many changes
since 5.4 I could not see any any changes to prevent this scenario to happen
so I believe this problem still exists in 6.6-rc7.

I'd like to get your opinion on this potential issue with the latest kernel
and if the problem still exists then what's the best way to fix it.

Thanks,
-Dai