Received-SPF: pass (google.com: domain of linux-nfs+bounces-698-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Mon, 18 Dec 2023 17:18:01 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: Guillaume Morin <guillaume@morinfr.org>
Cc: linux-nfs@vger.kernel.org
Subject: Re: SUNRPC: crash from svc_alloc_arg()
Message-ID: <ZYDFGQR9HbbjdaoQ@tissot.1015granger.net>
References: <ZYC9rsno8qYggVt9@bender.morinfr.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZYC9rsno8qYggVt9@bender.morinfr.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?2BEyx7ToV3064BLgoqfbTm0KqKpu5UQJtxi/cJQAsMspn8mL+SaswMjEWLlR?=
 =?us-ascii?Q?5o5ChxiCLwvBOR5ILoQC7GcfNkEeFTWM1ixJzPeN4Ma1n9DqEzVcvm+Irc8O?=
 =?us-ascii?Q?H9sqCQb85vc04SJ0C56600DcfWH5eZDKMl7o7sTbnlHq7P8tbU9xTjgDuAs3?=
 =?us-ascii?Q?9nX3fCTmYe4t1yMBxK0+IkmqrM7szZfFU/wj4b54sinOVFr7tSknyOd7XwCk?=
 =?us-ascii?Q?W2//s3m6gDoqkI4HK0ffspee09/u8j9yIWEkCi1u5nLra6Jt72nAfsAp1O7c?=
 =?us-ascii?Q?AU3wCfPCbKe01mLXJbJxbLS1Ks9M+Gx00svShX2j7LhxWa/EMBlySUion+fn?=
 =?us-ascii?Q?DSlleNRl+ryYeyrAHF9qknaKcif65JbRlksV6UirFQegJACRC3q4VXUavWUA?=
 =?us-ascii?Q?Iu2mgUZ5hXiElvd9LeyTX4aVnr7fs/Q9RUnauoVweWpOK00yv3DkFbuK53fc?=
 =?us-ascii?Q?50x3SEeMMHW+Cc3IdgeA+JxFULNZ3juxMd8danG3T2kM0hY20h2MognACeo/?=
 =?us-ascii?Q?38+IygAkKd3k1bo70/JJbZYnHFWtpAkKhCZhlrZSsSdjEVyuPfnzxJEP2imT?=
 =?us-ascii?Q?QCgtze2f9xlPbMOB0rcTBr8Tahctd6w4/mDxpuQUloIK0WqnwvBd0bfB1dyi?=
 =?us-ascii?Q?EDMCwOskzjX0Np85sSDWFXCANkGMa4wxYhzXst19+7dpJHRBB4TtTrM/5E5E?=
 =?us-ascii?Q?UkMSS4ocJYILsP7rlibvDAWlLTsFPSf6i9U5hxqI3BZm8Bqdl4WICq0SpvgB?=
 =?us-ascii?Q?qaim3vBO4kr0qR893sTbNMJo9TXeWddqWbiBTR++qHLTHQgsXknQTfwW58Iu?=
 =?us-ascii?Q?JrhrM1qtVzMBzDAHFeCgbQWSf1D4iZmNmoYNwk2oRPNFpdiD3cRTHpS3Ykdl?=
 =?us-ascii?Q?1Kj37ZFMhIObRUtp6HOPD4ByEJPjDFcWafi2CInoMHGn/XbwDBXKTv6SF4Yl?=
 =?us-ascii?Q?k/NkSnhCMV6y670tHc1gmsRfqkq0a9ZPybCIv3IR+PBeH6zcbIz6lkyo17rL?=
 =?us-ascii?Q?7qpfu+Wpadl8RHmQn6TFVt6N478iWvqHCtXLCbnB5mrYpUTFn7OdDOhsfmD8?=
 =?us-ascii?Q?Jfpb6BiOC4M4qRMCnhOWOeTxD3hAFKXFUClkAYeQd+i6zPw5TTDqLpaXj+tC?=
 =?us-ascii?Q?BfjYhAlwJmuO5whn/oafjRhsQ8iflL5zj/Hg1MGnCINmwlgoFtkAnKperIwH?=
 =?us-ascii?Q?uSLP4GAIx2RW/8efjrR4EVMGmO4ZjZFSOh3KHc2RCE2lXiGZt2axG0JRumXn?=
 =?us-ascii?Q?0UH2ZV3gzftNhgoVYT6pCjjFVvd2f/eiApQ/l1dPbs2HFJWuZwqj0KNw7D27?=
 =?us-ascii?Q?fuSriCJhUi9zyr5IyBjgp3K9LQ7K07apODyqoKwjzWZYVI9fspeNxkJ2zMiV?=
 =?us-ascii?Q?NPhtPAzV5LMeZQG+ZN4ThnevUNiRW3eEXjBGSxt2Yr7F1OmQumSA0PWkEzRB?=
 =?us-ascii?Q?u4ljFX/zXONbRNUcYQdb2Q1LIGvEuj+jA0CWQqMDKoInL0cphQsUH2bM2Gll?=
 =?us-ascii?Q?6uvr6skH0FMPcvLj5qPGqic37Qwf1OlGeyEeqbZppeHR+pejs6IEDd/Vq90V?=
 =?us-ascii?Q?Q35yDD0GIyEiS8tSbVkbYuUZtQaA0BbPi0sAWI/jy9rFKjCKsX1/oq/usD17?=
 =?us-ascii?Q?Cg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	rfplGFs64OHALIL9FtXEDZd4d4JRCTGXDCfJlEXFWawuOWsdnBvktBvm9nRmowpVDU9RjSUVtGLtDLtV/qMqlJLRmaJY9QnLWo8XtPgLw5NyT/S02oFf5vHYQ2PQcsIB3Tlf1fKp18Kb0jwlWz5OYakyollJf0A0GhnPaVL/X9hyHJplTJBbLnMBU0gA5Z7xH/9X+qrnemkKtZTGL4KQirVN7ynwX/3N6MVVtSWY9ixlxAScHS8EmmYLASC9ZuWEOZAb5gKHyVm1r6X1ccE7vFGqD8feZTdldX39ISsNllZ/3dSmtGFSKjxyNhqnGKEmSid39g37vdgm9Yn81NLtdapSXxxpkzaeyk13VXd5gnd+/aYOshGvq48OGaux6BJVWTnCFqtuaqaWThJGpCylr2QBObvdNFHL/3oOHj7YOIEZKNiPfunF00lzlSXQDurYDcVorFla4XYG7zyiJhdVu37M3jp10f7JINRDqzZK1XQRe/nXQgZtHwLFZ4vxddk/vRq6GUuIiLQ/zdDb1Gh1wNEKqMUMHuJi83weYFPgihX0FTESkax6YA9lOrH7vYrk4ZxAyMYDBiUaAJTCw+xaFnOshBH6b62DehTXqDv5K8g=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f4fb5be0-bef0-48b1-2941-08dc0017343f
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Dec 2023 22:18:04.2068
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4z+YsIrfK52W7jdRKZS1+KGcm+ESXRULOedh0LDfofe46/pqNxdyMnrKQXGZxQBEcirTmyZcr5U259p13s1bng==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7283
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-18_15,2023-12-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0
 mlxlogscore=999 malwarescore=0 bulkscore=0 phishscore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312180167
X-Proofpoint-GUID: L5FN6yyVLHR8VJDIa94TBHzPoZky9jlV
X-Proofpoint-ORIG-GUID: L5FN6yyVLHR8VJDIa94TBHzPoZky9jlV

On Mon, Dec 18, 2023 at 10:46:22PM +0100, Guillaume Morin wrote:
> Hello Chuck,
> 
> I believe commit 5f7fc5d "SUNRPC: Resupply rq_pages from node-local memory" in
> Linux 6.5+ is incorrect. It passes unconditionnaly rq_pool->sp_id as the NUMA
> node.
> 
> While the comment in the svc_pool declaration in sunrpc/svc.h says that
> sp_id is also the NUMA node id, it might not be the case if the svc is
> created using svc_create_pooled(). svc_created_pooled() can use the
> per-cpu pool mode therefore in this case sp_id would be the cpu id.
> 
> from __svc_create:
> 	for (i = 0; i < serv->sv_nrpools; i++) {
> 		struct svc_pool *pool = &serv->sv_pools[i];
> 
> 		dprintk("svc: initialising pool %u for %s\n",
> 				i, serv->sv_name);
> 
> 		pool->sp_id = i;
> 
> When using the cpu-mode, this triggers a BUG on my machine:
> BUG: unable to handle page fault for address: 0000000000002088
> 
>  #7 [ffffafa3dc42fc90] asm_exc_page_fault at ffffffffa3e00bc7
>     [exception RIP: __next_zones_zonelist+9]
>     RIP: ffffffffa32fbbc9  RSP: ffffafa3dc42fd48  RFLAGS: 00010286
>     RAX: 0000000000002080  RBX: 0000000000000000  RCX: ffff8ba5f22bafc0
>     RDX: ffff8ba5f22bafc0  RSI: 0000000000000002  RDI: 0000000000002080
>     RBP: ffffafa3dc42fdc0   R8: 0000000000002080   R9: ffff8ba62138c2d8
>     R10: 0000000000000001  R11: 0000000000000000  R12: 0000000000000cc0
>     R13: 0000000000000002  R14: 0000000000000000  R15: 0000000000000001
>     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>  #8 [ffffafa3dc42fd50] __alloc_pages at ffffffffa334c122
>  #9 [ffffafa3dc42fdc8] __alloc_pages_bulk at ffffffffa334c519
> #10 [ffffafa3dc42fe58] svc_alloc_arg at ffffffffc0afc0d7 [sunrpc]
> #11 [ffffafa3dc42fea0] svc_recv at ffffffffc0afe08d [sunrpc]
> #12 [ffffafa3dc42fec8] nfsd at ffffffffc0dec469 [nfsd]
> #13 [ffffafa3dc42fee8] kthread at ffffffffa30e4826
> 
> I believe the fix is to expose svc_pool_map_get_node() and use that in
> the alloc_pages_bulk_array_node() call in svx_xprt.c. Reverting 5f7fc5d
> would obviously work as well.
> 
> The comment in svc.h should probably be updated as well since it's misleading.
> 
> I didn't provide a patch because I wasn't quite sure which approach you would
> prefer but could provide one if that's helpful.

Reverted and applied for v6.7-rc (see my nfsd-fixes branch). Thanks
for the report and analysis!


-- 
Chuck Lever