Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp625862pxb; Thu, 2 Sep 2021 11:21:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzVMcm1z6iDcOQJwgBoGIuTcXbrgw4RVk5cVA5G5AuWB/qsEKkt6TjC7u64mg/P/WLFHwja X-Received: by 2002:a02:2243:: with SMTP id o64mr4022669jao.40.1630606886769; Thu, 02 Sep 2021 11:21:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630606886; cv=none; d=google.com; s=arc-20160816; b=NqHLqiGThqUpewcpdNL7EQeAp1p+f+ZF4+EgPvsu+sFrM1mgcB4Tk+YvFY8DvocL9u +YErYensy1xXrzPGDIAkZb/4CBiHjYgygM2+onuyi2yUwWWIbQW8KXmSAY0Y1XU0nYUv LtgBaRSa008t4zfwEI5K5e7NBkEmRyo+pEHqQNjYanmk0PSV5moXPwf8IiY74p/Dbums RzCbQC8eDEtV2lunpVk+Wn9iu3otwtB9Djm0ncigEq7cewlWGvUOZkF8TeTkFENZpCNr 8YJy9DobOtFAxLxQBPv+iwNnjXIRr361GEZf1a8BN4bmwVQtYW4fuFKOGgxschYedLDj lu3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:subject:cc:to:from:date:references:in-reply-to :message-id:mime-version:user-agent:dkim-signature:dkim-signature; bh=rWeCfc9VwEpivgH3ElxeF4AP9Em0hx7fi+pEfHiMVP4=; b=U25LChXo9jO98v+IJ1nIKAD93XMyV6fCwngCGGnQXn8TvUUYr5+Zaj8YuX3eU5zt83 /97GWcgIKaUiyEtAb6meHGjPmELxNTmBXE798RYmwM5rjxNLcoS9oeMCRatQlPbHPW8K cjmwL3uXIx4UBPIiQDRNXXj+DyUuwSohnQehnQPeC+x8X4YsOth5hW5K/TFSv9RyRKFV avq3EzdtR6TgJAzDP4/uQQvBVwcJ9Mjw4Yz+3r1cG4der71F4xkwpYhMxnLJcQOSv+2w wjznlSIza06rnECj5p4aDI3jaEPXxmQtRIeuadhjpox8zQnUNFtxSAaQ6lSfQmzPClp9 KdeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@svenpeter.dev header.s=fm2 header.b=AuInL9tN; dkim=pass header.i=@messagingengine.com header.s=fm3 header.b=G2ACLst9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=svenpeter.dev Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f8si2777915ilu.107.2021.09.02.11.21.13; Thu, 02 Sep 2021 11:21:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@svenpeter.dev header.s=fm2 header.b=AuInL9tN; dkim=pass header.i=@messagingengine.com header.s=fm3 header.b=G2ACLst9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=svenpeter.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346920AbhIBSUh (ORCPT + 99 others); Thu, 2 Sep 2021 14:20:37 -0400 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:33169 "EHLO out5-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233296AbhIBSUf (ORCPT ); Thu, 2 Sep 2021 14:20:35 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 5AB1B5C01ED; Thu, 2 Sep 2021 14:19:36 -0400 (EDT) Received: from imap21 ([10.202.2.71]) by compute1.internal (MEProxy); Thu, 02 Sep 2021 14:19:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=mime-version:message-id:in-reply-to:references:date:from:to :cc:subject:content-type; s=fm2; bh=rWeCfc9VwEpivgH3ElxeF4AP9Em0 hx7fi+pEfHiMVP4=; b=AuInL9tN1I1ovcXm4oG4VDeb5jtx8G5LKfAsBPLTi6Lf dnZAs/tBmxQBlex49291A2gqYLEcepIl5T7llHDUrBvJkRRHWXY/SfqfFloLxwUM fIWQ7LmoGv4Oc+uxNQH5TsXifPiY0sCO/gAzY3z6DSKJYEsExMZXbII69AoqY22l Bm4dJHFQn+YRHScbkKXtHyqBVhE1P083J6RZhUYPNFyud+JSzRNV++MHS+iEk6ci PoERh7wq7dUbh0ygvb39am+89kbKIdeFfolYAB6G/oh3vd6oYLY4aAM2PBRWZE0S iMm1hfgO+/YQjr5E8Ih5NkA0H4Gza0XUDW5QzQizVw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=rWeCfc 9VwEpivgH3ElxeF4AP9Em0hx7fi+pEfHiMVP4=; b=G2ACLst9n6EmcBeuB8t9Cn UBicpUedbICpaGir/glrW/F8YtEdsTQdUGbkupVVEgEt0yxShv1Zx/tW5wZK944B IzbJZxfsh6km8yo1bOllE/K/D9CXaqHmRIoWOcwMybF7ZNwCJ4VbyKpPkQw32AwD /mpI5FdyNtBjj8tlCMVR4SeT1+DnkjkWVWyhuVSWnbfRfHbtYhSlYcaLPzkVgEI9 0fm7ydZz79dAdthQ8FdjXtgz5r9JVZCM9SXVoDXdj7VzrfdMR/smQne6fQ6A9qRA oEZXqavnx4GHO8CDDelMs03dJf7ZngDiWXGHoPDH8ZX56gyWWE2qSEIjkf5Xqofw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvhedguddvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderreejnecuhfhrohhmpedfufhv vghnucfrvghtvghrfdcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrf grthhtvghrnhepheejgfdttdefleefteejieefudeuheelkedtgedtjeehieetueelheeu hfegheegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epshhvvghnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 8810451C0060; Thu, 2 Sep 2021 14:19:34 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-1126-g6962059b07-fm-20210901.001-g6962059b Mime-Version: 1.0 Message-Id: In-Reply-To: References: <20210828153642.19396-1-sven@svenpeter.dev> <20210828153642.19396-4-sven@svenpeter.dev> Date: Thu, 02 Sep 2021 20:19:14 +0200 From: "Sven Peter" To: "Alyssa Rosenzweig" Cc: "Sven Peter" , "Joerg Roedel" , "Will Deacon" , "Robin Murphy" , "Arnd Bergmann" , "Mohamed Mediouni" , "Alexander Graf" , "Hector Martin" , linux-kernel@vger.kernel.org Subject: =?UTF-8?Q?Re:_[PATCH_v2_3/8]_iommu/dma:_Disable_get=5Fsgtable_for_granul?= =?UTF-8?Q?e_>_PAGE=5FSIZE?= Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Sep 1, 2021, at 23:10, Alyssa Rosenzweig wrote: > > My biggest issue is that I do not understand how this function is supposed > > to be used correctly. It would work fine as-is if it only ever gets passed buffers > > allocated by the coherent API but there's not way to check or guarantee that. > > There may also be callers making assumptions that no longer hold when > > iovad->granule > PAGE_SIZE. > > > > Regarding your case: I'm not convinced the function is meant to be used there. > > If I understand it correctly, your code first allocates memory with dma_alloc_coherent > > (which possibly creates a sgt internally and then maps it with iommu_map_sg), > > then coerces that back into a sgt with dma_get_sgtable, and then maps that sgt to > > another iommu domain with dma_map_sg while assuming that the result will be contiguous > > in IOVA space. It'll work out because dma_alloc_coherent is the very thing > > meant to allocate pages that can be mapped into kernel and device VA space > > as a single contiguous block and because both of your IOMMUs are different > > instances of the same HW block. Anything allocated by dma_alloc_coherent for the > > first IOMMU will have the right shape that will allow it to be mapped as > > a single contiguous block for the second IOMMU. > > > > What could be done in your case is to instead use the IOMMU API, > > allocate the pages yourself (while ensuring the sgt your create is made up > > of blocks with size and physaddr aligned to max(domain_a->granule, domain_b->granule)) > > and then just use iommu_map_sg for both domains which actually comes with the > > guarantee that the result will be a single contiguous block in IOVA space and > > doesn't required the sgt roundtrip. > > In principle I agree. I am getting the sense this function can't be used > correctly in general, and yet is the function that's meant to be used. > If my interpretation of prior LKML discussion holds, the problems are > far deeper than my code or indeed page size problems... Right, which makes reasoning about this function and its behavior if the IOMMU pages size is unexpected very hard for me. I'm not opposed to just keeping this function as-is when there's a mismatch between PAGE_SIZE and the IOMMU page size (and it will probably work that way) but I'd like to be sure that won't introduce unexpected behavior. > > If the right way to handle this is with the IOMMU and IOVA APIs, I really wish > that dance were wrapped up in a safe helper function instead of open > coding it in every driver that does cross device sharing. > > We might even call that helper... hmm... dma_map_sg.... *ducks* > There might be another way to do this correctly. I'm likely just a little bit biased because I've spent the past weeks wrapping my head around the IOMMU and DMA APIs and when all you have is a hammer everything looks like a nail. But dma_map_sg operates at the DMA API level and at that point the dma-ops for two different devices could be vastly different. In the worst case one of them could be behind an IOMMU that can easily map non-contiguous pages while the other one is directly connected to the bus and can't even access >4G pages without swiotlb support. It's really only possible to guarantee that it will map N buffers to <= N DMA-addressable buffers (possibly by using an IOMMU or swiotlb internally) at that point. On the IOMMU API level you have much more information available about the actual hardware and can prepare the buffers in a way that makes both devices happy. That's why iommu_map_sgtable combined with iovad->granule aligned sgt entries can actually guarantee to map the entire list to a single contiguous IOVA block. Sven