Received-SPF: pass (google.com: domain of linux-kernel+bounces-75669-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: Michael Kelley <mhklinux@outlook.com>
To: Will Deacon <will@kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>
CC: "kernel-team@android.com" <kernel-team@android.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>, Christoph Hellwig
	<hch@lst.de>, Marek Szyprowski <m.szyprowski@samsung.com>, Robin Murphy
	<robin.murphy@arm.com>, Petr Tesarik <petr.tesarik1@huawei-partners.com>,
	Dexuan Cui <decui@microsoft.com>, Nicolin Chen <nicolinc@nvidia.com>
Subject: RE: [PATCH v4 5/5] iommu/dma: Force swiotlb_max_mapping_size on an
 untrusted device
Thread-Topic: [PATCH v4 5/5] iommu/dma: Force swiotlb_max_mapping_size on an
 untrusted device
Thread-Index: AQHaZLoRyKgMK9jLMUWPn0GpqcfvobEVbBTg
Date: Wed, 21 Feb 2024 23:39:29 +0000
Message-ID:
 <SN6PR02MB4157828120FB7D3408CEC991D4572@SN6PR02MB4157.namprd02.prod.outlook.com>
References: <20240221113504.7161-1-will@kernel.org>
 <20240221113504.7161-6-will@kernel.org>
In-Reply-To: <20240221113504.7161-6-will@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?BG6xJaHXf78fs/MYqELLopPDnShnyRWDzk0LBv4UYd3dqm9+o6EbTf4etjl3?=
 =?us-ascii?Q?FpvJVdmfDi+Gm6Uk63FkAbqbJSsgj2vmreZ8jJZR3Z9Ry1BpcgLlSRIDeCVi?=
 =?us-ascii?Q?nn6TSpMcpXJgyzf3VhhEERdsPzyCY6KEREv2izcyEW1tBfzm7l+0RLBhJOn9?=
 =?us-ascii?Q?YDMtorliHu5WAop/4w4VC9PUd7RU5HiSpOk/C2vKzWjeboPFPOb98Im6zrI9?=
 =?us-ascii?Q?nnynVTLA0kf+WBBCqGmUU1adijKhlmNDwucal4nvLQCmDAQUVmTQbtcMRYLf?=
 =?us-ascii?Q?e4sXymm7fFduynMRMo1JKR4f7F78EHc0PAiEmMHRbnOXZJHEVay9eTmuIYcG?=
 =?us-ascii?Q?WsZmnSFOZ6uoep3WPsVFzCaK0Ur1/4f5OqcDN0uHnABi9EM9VxUWTHsm6ij6?=
 =?us-ascii?Q?ujvKvGetHUrD+I0J5v5DbWFX/F4o+CJUOV54AyiCGkWD3/4ryDtvTab3dOg7?=
 =?us-ascii?Q?zLey2ZUzL2F/IJ6xkChI6N0AYbbG1b6qv/KC66Pm633v8u6FdeQtdGyOsJNe?=
 =?us-ascii?Q?7d3OFxowlY8NRHuwf7EuVqvaVY8NL5GH4nJI86jfxCP5IXJSrBADByoAXvxj?=
 =?us-ascii?Q?qMFfswjL3uUQG0AFJWSVoYdUwxDkhlEINV9O9ssFwiZJ8v4UQKu+Ntv5O21N?=
 =?us-ascii?Q?ol6Rbu77HeooRPq9N6kc365q1qQGUnyxhszP0Dbk2UNgK7uA5vjBEwTPOa2i?=
 =?us-ascii?Q?Kjk9cHcNKzlaCBQ0gcNIu1JWL3k7jM/Um6nGsbcaCoQVGE1LQwayu455BULl?=
 =?us-ascii?Q?v4Cf6fUkaGFAviD5ZK/hudEAD0dIfCb6nyrU2Kcsg/w5T7yDB4dHMVQyoCVt?=
 =?us-ascii?Q?OfxIfJxFgKfXuzrfuymXRerbQ8ahll9NiVgOSqhyJJy4e46+uf1UA2L+YnJS?=
 =?us-ascii?Q?sRIbM8/LqeCWAFdhCPKH9zxGuVjooK/UELiZNESDEEQo3H3TcrSoUB82DALE?=
 =?us-ascii?Q?u4w/lhr1c7LJeHVPEldeYq2aToPjhF/+siBi+fNanvb3mAj3NL/qoWim0iHV?=
 =?us-ascii?Q?F884AiOk11MdaL9wt96yHQqD4B2B83NGAvaE//FqjMVmUFxBL5SKDNGdELAQ?=
 =?us-ascii?Q?cjkkW+QPWMR6cU8KxIk/HklHoKMDYBwT/9NVLlDPb9uTnthGEc1Qpl+d3WPB?=
 =?us-ascii?Q?VlSlEnfRfcPAJkEJfxMzOmFF2YpREY1mXh+jrssJLu60VobVUpyYZ2jklbLq?=
 =?us-ascii?Q?Rrc3u6TOwtfC6TiVw9qLt9zZOtm1oUe4+vvBBA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR02MB4157.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-Network-Message-Id: 7dc37c86-4795-4f59-c957-08dc33365913
X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Feb 2024 23:39:29.5006
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR02MB9754

From: Will Deacon <will@kernel.org> Sent: Wednesday, February 21, 2024 3:35=
 AM
>=20
> The swiotlb does not support a mapping size > swiotlb_max_mapping_size().
> On the other hand, with a 64KB PAGE_SIZE configuration, it's observed tha=
t
> an NVME device can map a size between 300KB~512KB, which certainly failed
> the swiotlb mappings, though the default pool of swiotlb has many slots:
>     systemd[1]: Started Journal Service.
>  =3D> nvme 0000:00:01.0: swiotlb buffer is full (sz: 327680 bytes), total=
 32768
> (slots), used 32 (slots)
>     note: journal-offline[392] exited with irqs disabled
>     note: journal-offline[392] exited with preempt_count 1
>=20
> Call trace:
> [    3.099918]  swiotlb_tbl_map_single+0x214/0x240
> [    3.099921]  iommu_dma_map_page+0x218/0x328
> [    3.099928]  dma_map_page_attrs+0x2e8/0x3a0
> [    3.101985]  nvme_prep_rq.part.0+0x408/0x878 [nvme]
> [    3.102308]  nvme_queue_rqs+0xc0/0x300 [nvme]
> [    3.102313]  blk_mq_flush_plug_list.part.0+0x57c/0x600
> [    3.102321]  blk_add_rq_to_plug+0x180/0x2a0
> [    3.102323]  blk_mq_submit_bio+0x4c8/0x6b8
> [    3.103463]  __submit_bio+0x44/0x220
> [    3.103468]  submit_bio_noacct_nocheck+0x2b8/0x360
> [    3.103470]  submit_bio_noacct+0x180/0x6c8
> [    3.103471]  submit_bio+0x34/0x130
> [    3.103473]  ext4_bio_write_folio+0x5a4/0x8c8
> [    3.104766]  mpage_submit_folio+0xa0/0x100
> [    3.104769]  mpage_map_and_submit_buffers+0x1a4/0x400
> [    3.104771]  ext4_do_writepages+0x6a0/0xd78
> [    3.105615]  ext4_writepages+0x80/0x118
> [    3.105616]  do_writepages+0x90/0x1e8
> [    3.105619]  filemap_fdatawrite_wbc+0x94/0xe0
> [    3.105622]  __filemap_fdatawrite_range+0x68/0xb8
> [    3.106656]  file_write_and_wait_range+0x84/0x120
> [    3.106658]  ext4_sync_file+0x7c/0x4c0
> [    3.106660]  vfs_fsync_range+0x3c/0xa8
> [    3.106663]  do_fsync+0x44/0xc0
>=20
> Since untrusted devices might go down the swiotlb pathway with dma-iommu,
> these devices should not map a size larger than swiotlb_max_mapping_size.
>=20
> To fix this bug, add iommu_dma_max_mapping_size() for untrusted devices t=
o
> take into account swiotlb_max_mapping_size() v.s. iova_rcache_range() fro=
m
> the iommu_dma_opt_mapping_size().
>=20
> Fixes: 82612d66d51d ("iommu: Allow the dma-iommu api to use bounce buffer=
s")
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> Link: https://lore.kernel.org/all/ee51a3a5c32cf885b18f6416171802669f4a718=
a.1707851466.git.nicolinc@nvidia.com/
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>  drivers/iommu/dma-iommu.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
>=20
> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
> index 50ccc4f1ef81..7d1a20da6d94 100644
> --- a/drivers/iommu/dma-iommu.c
> +++ b/drivers/iommu/dma-iommu.c
> @@ -1706,6 +1706,13 @@ static size_t
> iommu_dma_opt_mapping_size(void)
>  	return iova_rcache_range();
>  }
>=20
> +static size_t iommu_dma_max_mapping_size(struct device *dev)
> +{
> +	if (is_swiotlb_active(dev) && dev_is_untrusted(dev))
> +		return swiotlb_max_mapping_size(dev);
> +	return SIZE_MAX;
> +}
> +

In this [1] email, Nicolin had a version of this function that incorporated
the IOMMU granule.  For granules bigger than 4K, I think that's needed
so that when IOMMU code sets the swiotlb alloc_align_mask to the
IOMMU granule - 1, the larger offset plus the size won't exceed the
max number of slots.  swiotlb_max_mapping_size() by itself may
return a value that's too big when alloc_align_mask is used.

Michael

[1] https://lore.kernel.org/linux-iommu/SN6PR02MB415727E61B5295C259CCB268D4=
512@SN6PR02MB4157.namprd02.prod.outlook.com/T/#m14dd359c5a4dd13e6cb0c35cf94=
f8d746257ae48


>  static const struct dma_map_ops iommu_dma_ops =3D {
>  	.flags			=3D DMA_F_PCI_P2PDMA_SUPPORTED,
>  	.alloc			=3D iommu_dma_alloc,
> @@ -1728,6 +1735,7 @@ static const struct dma_map_ops iommu_dma_ops =3D {
>  	.unmap_resource		=3D iommu_dma_unmap_resource,
>  	.get_merge_boundary	=3D iommu_dma_get_merge_boundary,
>  	.opt_mapping_size	=3D iommu_dma_opt_mapping_size,
> +	.max_mapping_size       =3D iommu_dma_max_mapping_size,
>  };
>=20
>  /*
> --
> 2.44.0.rc0.258.g7320e95886-goog