Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
From:   Qing Huang <qing.huang@oracle.com>
To:     George Kennedy <george.kennedy@oracle.com>,
        "leon@kernel.org" <leon@kernel.org>, "jgg@ziepe.ca" <jgg@ziepe.ca>,
        "sd@queasysnail.net" <sd@queasysnail.net>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>
CC:     George Kennedy <george.kennedy@oracle.com>,
        Tom Hromatka <tom.hromatka@oracle.com>,
        Harshit Mogalapalli <harshit.m.mogalapalli@oracle.com>
Subject: RE: [PATCH v2] mlx5: fix init stage error handling to avoid double
 free of same QP and UAF
Thread-Topic: [PATCH v2] mlx5: fix init stage error handling to avoid double
 free of same QP and UAF
Thread-Index: AQHaBqRwgrXAEMa6sk2OFFhR85ev7bBZoajg
Date:   Wed, 25 Oct 2023 00:15:36 +0000
Message-ID: <SJ0PR10MB54855B65988473FE7642E8F6E9DEA@SJ0PR10MB5485.namprd10.prod.outlook.com>
References: <1698170518-4006-1-git-send-email-george.kennedy@oracle.com>
In-Reply-To: <1698170518-4006-1-git-send-email-george.kennedy@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ZXm0tSX2kZvIaKwWr7/zhA32ihi1XzKlnZKaCZpERgzmcJxoymANx/Uk4KRz?=
 =?us-ascii?Q?b69sOhrOhjuUP8EB6oSwataTks1TEHFOKE7mcKA/1WIfMYeLiQSyK4xitu72?=
 =?us-ascii?Q?t1oHWF/Vxde3Jc+meC1PvHAroUFhYE4XxxYI4wE7eH2u/AqYhy4lyP7qAbWx?=
 =?us-ascii?Q?Z8bdEMuC2rUc/Y78pASWvUxsdnbKHHxUTGxqC2ZWZAkP/a8jocE8a9LhTzrM?=
 =?us-ascii?Q?0snJa7/wM1jrJ8AuSniXFO6/qwk4FTIC6mh8nETclqpHTdbnfsvcvOjzUQQn?=
 =?us-ascii?Q?cdcczr+98+Bi7/fCPsEGK5egpM2AD+UYUx6l/5PYzSaiX9U29O0kSLkhzTfY?=
 =?us-ascii?Q?Zs0ezEipi4mZrI94mApqFYi2EX1JnKAW+OhmmCM78iKG1egi+R7teYhu782P?=
 =?us-ascii?Q?MUazlA8k8Iod4KpvB9sQzVn5YqQk1fy+dAKZpW+GUgoGR7adHAcVHuSuU4fN?=
 =?us-ascii?Q?ldhyZQYHjL6ycT2IytkDdp+RLslsl2QrsP1qskBo/JbgHla1YvJm6BTsAEJ6?=
 =?us-ascii?Q?MJxyVS5Pv4xV83yk5gzSWZpaLYBn040lv5Ahe4pRy/NwKBT6ErNXorQ7EpBp?=
 =?us-ascii?Q?xl6y0pLzJ+SStlv2J9A31qjLA3mf9YGg+K4if/b/DWAqdxy/JLaB2auv3oxG?=
 =?us-ascii?Q?bj18v9RmNSb9aJrSp+4sYaNZDw5/Bz1wI5ChX49+sSUKoIGDAZVl6HKhWcrT?=
 =?us-ascii?Q?8RDiUlHPetj/OeAMqx7pRka0SODAisEzH1sWmuK4lW2n2ESw2XM4wQpFnch1?=
 =?us-ascii?Q?3w792YJCoRQqY30XCm8BpNztmkuSRRPL3nfQc5hjyhHq97hb2CrCT2Wp+LmV?=
 =?us-ascii?Q?UbppzkmpevHT/YLoAJSYkI0KVmm1SvyCaN25Yq6MniL4TT0o75YPyAY88GYb?=
 =?us-ascii?Q?rFB8JGNlWRY55zipNRTa1QgVoNp1dKe7fOk3seb+xKQlaGDp1kxcXUJjauUL?=
 =?us-ascii?Q?TG524MrQHnoNikXEptnHc+LGGf312oem74X8GRtCase2YIRTbBvqe+Pw7Qo1?=
 =?us-ascii?Q?xcKA5Spy1J7VQfeZPUBjxZaatleXpmWSLLYKVedQvLsD+K4aE5KMZyY6M9Kt?=
 =?us-ascii?Q?412cN5HKwQd7OSJSbzD9wYTNKoO5lXOrtdi78r1k4phIwxNi9k0z/+/nA4lG?=
 =?us-ascii?Q?BHZGRxzR09NfCZJvvYLseEhmOqDJPNL6F9XgXm/0QaX5oHzETIzAs//dfnIr?=
 =?us-ascii?Q?ii3zLMVYDwdwYKCA85tfNJ2UOnmpWv3RKz3swT34viG9mfCt5hSmQuJiWAsf?=
 =?us-ascii?Q?TmbGsqFnA/Nh/hM56kSkp2xAFoNofj9omq9l5oNIX/rjIobkKx1kiQ38GgqC?=
 =?us-ascii?Q?HzYFPVrKV8Vsq5mTFGRLL3JDo8g7VuuJTZzIxrpqEopJU3Bk/2y0M6ilN9RI?=
 =?us-ascii?Q?VneUWLxNg1iG9GHrLHpWO+NQUGC4N6Hvw2nppSGsljf4aokGLJ6keCbx5dJl?=
 =?us-ascii?Q?kISLDw1bqsmk6lC1w2u9xYqGEzvlov08dfjNMovPo8yNU39NZOr098tHS/QS?=
 =?us-ascii?Q?7APHStvWxfZCTf6u25J8wKsw9RDKIh42puqjSvFnsEDUnVMXH7nmZ8hpILTU?=
 =?us-ascii?Q?VfcUhinZi0M+4IbjHCj0GE66LLsM+lH0L1G2cj97?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: +uIEdm3Fpns/LyO1adY1Ei6cha027xMtkYOfsuI+Ih9fVvMFhFJvCLqWu+efSXOMhAUKy2lNSY0zP9MzXmLI/2kfAvmtroBBmj72MFT+9NaelTalOWQMT+B7C0qHitwj4EUQ5QIQ0wIU6iN4fD3ZdegrZIQfZrm56N+erwffXDZ1XqP0gdNSKtul+zQbFP4Y5k+7G67IremSUZ+g9ATktTWlMdOr24KMQPB2pivQrQwpchnNhynqNBaLb2OPLeQpXLT4OXTOiPZM6J6gr/Wa3Ge/wqnofdtasJOQhxY4Xpe/qGQj9ngoE+ikIVY+Fy8IPx/ZZeX8P9jvEs/2dcxtpifr8vqHRtH9TW1/F2cFqYpr1FcTthrs3e2wCEEO27NVwzxQnNeqTExDyC7drGuBz1PMlDNy1veGbp/K3ODBk8UO5ttOV1AtRLjHzYp9nwOzKC663kKPo5AdQZ3Z3fr6uv/amijPVk4bWgmcqqgIvo1hgSDDf6n3e2rSG9Bp1zOxIo5DgDP10FkM7C5VdQxmylHj8N2hLxxTKX4GsEAhpFJOp0M8zwgWBXJSNs56Xfjii9cBmhmyEGeKX+Ai7gmbMF68fOD6pqjPmwlMhjUNILQqlZv/vCRHtOnp+yQErgI6c1/HKexAmXGe3eurmxhY5SeiVAN6nhZOnjubK50y4HIDJCG+rzTjC3esYRKCUmGTzH4XWCDK85QiYPwyl30L8yiqZyV9S5CqllLtGhjA10bqB6dc7kVkGOdMFCX6ARD566KigadYfbxgPEKlALJlf9m0Yg0nCAQSfkz3Cq/er6mg9FgInR9XsamKBGaTPm9Wi8lC2Go8+TlMKmx9TPkBjfOU9hyAiJZ/ZGZv3gFWz/Y=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB5485.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 82366ffa-5efa-4f69-510b-08dbd4ef82e9
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Oct 2023 00:15:36.1144
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 9UT4E3Up0c2s01PHcNGbcHQYd4+Tecp7+F6LyzqbvqLnh5GfF/BmzpUx4AsUdb9idDvYSD/fFBj/1Y1BO4Nf/w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6994
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-24_23,2023-10-24_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 phishscore=0
 mlxlogscore=999 mlxscore=0 bulkscore=0 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2310170001
 definitions=main-2310250000
X-Proofpoint-GUID: KXSwC64YSCPJWEvkyxLLOtlsc2a4jSMw
X-Proofpoint-ORIG-GUID: KXSwC64YSCPJWEvkyxLLOtlsc2a4jSMw
X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 24 Oct 2023 17:16:04 -0700 (PDT)


> -----Original Message-----
> From: George Kennedy <george.kennedy@oracle.com>
> Sent: Tuesday, October 24, 2023 11:02 AM
> To: leon@kernel.org; jgg@ziepe.ca; sd@queasysnail.net; linux-
> rdma@vger.kernel.org; linux-kernel@vger.kernel.org; netdev@vger.kernel.or=
g
> Cc: George Kennedy <george.kennedy@oracle.com>; Tom Hromatka
> <tom.hromatka@oracle.com>; Harshit Mogalapalli
> <harshit.m.mogalapalli@oracle.com>
> Subject: [PATCH v2] mlx5: fix init stage error handling to avoid double f=
ree of
> same QP and UAF
>=20
> In the unlikely event that workqueue allocation fails and returns NULL in
> mlx5_mkey_cache_init(), delete the call to
> mlx5r_umr_resource_cleanup() (which frees the QP) in
> mlx5_ib_stage_post_ib_reg_umr_init().  This will avoid attempted double f=
ree of
> the same QP when __mlx5_ib_add() does its cleanup.
>=20


Hi George,

There seems no cleanup function defined for this stage:

        STAGE_CREATE(MLX5_IB_STAGE_POST_IB_REG_UMR,
                     mlx5_ib_stage_post_ib_reg_umr_init,
                     NULL),

Do you know where __mlx5_ib_add() does the double free call after the alloc=
ation failure?

Regards,
Qing

> Syzkaller reported a UAF in ib_destroy_qp_user
>=20
> workqueue: Failed to create a rescuer kthread for wq "mkey_cache": -EINTR
> infiniband mlx5_0: mlx5_mkey_cache_init:981:(pid 1642):
> failed to create work queue
> infiniband mlx5_0: mlx5_ib_stage_post_ib_reg_umr_init:4075:(pid 1642):
> mr cache init failed -12
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> BUG: KASAN: slab-use-after-free in ib_destroy_qp_user
> (drivers/infiniband/core/verbs.c:2073)
> Read of size 8 at addr ffff88810da310a8 by task repro_upstream/1642
>=20
> Call Trace:
> <TASK>
> kasan_report (mm/kasan/report.c:590)
> ib_destroy_qp_user (drivers/infiniband/core/verbs.c:2073)
> mlx5r_umr_resource_cleanup (drivers/infiniband/hw/mlx5/umr.c:198)
> __mlx5_ib_add (drivers/infiniband/hw/mlx5/main.c:4178)
> mlx5r_probe (drivers/infiniband/hw/mlx5/main.c:4402)
> ...
> </TASK>
>=20
> Allocated by task 1642:
> __kmalloc (./include/linux/kasan.h:198 mm/slab_common.c:1026
> mm/slab_common.c:1039)
> create_qp (./include/linux/slab.h:603 ./include/linux/slab.h:720
> ./include/rdma/ib_verbs.h:2795 drivers/infiniband/core/verbs.c:1209)
> ib_create_qp_kernel (drivers/infiniband/core/verbs.c:1347)
> mlx5r_umr_resource_init (drivers/infiniband/hw/mlx5/umr.c:164)
> mlx5_ib_stage_post_ib_reg_umr_init (drivers/infiniband/hw/mlx5/main.c:407=
0)
> __mlx5_ib_add (drivers/infiniband/hw/mlx5/main.c:4168)
> mlx5r_probe (drivers/infiniband/hw/mlx5/main.c:4402)
> ...
>=20
> Freed by task 1642:
> __kmem_cache_free (mm/slub.c:1826 mm/slub.c:3809 mm/slub.c:3822)
> ib_destroy_qp_user (drivers/infiniband/core/verbs.c:2112)
> mlx5r_umr_resource_cleanup (drivers/infiniband/hw/mlx5/umr.c:198)
> mlx5_ib_stage_post_ib_reg_umr_init (drivers/infiniband/hw/mlx5/main.c:407=
6
> drivers/infiniband/hw/mlx5/main.c:4065)
> __mlx5_ib_add (drivers/infiniband/hw/mlx5/main.c:4168)
> mlx5r_probe (drivers/infiniband/hw/mlx5/main.c:4402)
> ...
>=20
> The buggy address belongs to the object at ffff88810da31000 which belongs=
 to
> the cache kmalloc-2k of size 2048 The buggy address is located 168 bytes =
inside
> of freed 2048-byte region [ffff88810da31000, ffff88810da31800)
>=20
> The buggy address belongs to the physical page:
> page:000000003b5e469d refcount:1 mapcount:0 mapping:0000000000000000
> index:0x0 pfn:0x10da30
> head:000000003b5e469d order:3 entire_mapcount:0 nr_pages_mapped:0
> pincount:0
> flags: 0x17ffffc0000840(slab|head|node=3D0|zone=3D2|lastcpupid=3D0x1fffff=
)
> page_type: 0xffffffff()
> raw: 0017ffffc0000840 ffff888100042f00 ffffea0004180800
> dead000000000002
> raw: 0000000000000000 0000000000080008 00000001ffffffff
> 0000000000000000 page dumped because: kasan: bad access detected
>=20
> Memory state around the buggy address:
> ffff88810da30f80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ffff88810da31000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >ffff88810da31080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> 			      ^
> ffff88810da31100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff88810da31180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Disabling lock debugging due to kernel taint
>=20
> Fixes: 04876c12c19e ("RDMA/mlx5: Move init and cleanup of UMR to umr.c")
> Reported-by: syzkaller <syzkaller@googlegroups.com>
> Suggested-by: Leon Romanovsky <leon@kernel.org>
> Signed-off-by: George Kennedy <george.kennedy@oracle.com>
> ---
> v2: went with fix suggested by: Leon Romanovsky <leon@kernel.org>
>=20
>  drivers/infiniband/hw/mlx5/main.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
>=20
> diff --git a/drivers/infiniband/hw/mlx5/main.c
> b/drivers/infiniband/hw/mlx5/main.c
> index 555629b7..5d963ab 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -4071,10 +4071,8 @@ static int
> mlx5_ib_stage_post_ib_reg_umr_init(struct mlx5_ib_dev *dev)
>  		return ret;
>=20
>  	ret =3D mlx5_mkey_cache_init(dev);
> -	if (ret) {
> +	if (ret)
>  		mlx5_ib_warn(dev, "mr cache init failed %d\n", ret);
> -		mlx5r_umr_resource_cleanup(dev);
> -	}
>  	return ret;
>  }
>=20
> --
> 1.8.3.1