Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 31 May 2022 13:21:52 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Robin Murphy <robin.murphy@arm.com>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>,
        Christoph Hellwig <hch@infradead.org>,
        Will Deacon <will@kernel.org>, Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 01/12] iommu/vt-d: Use iommu_get_domain_for_dev() in
 debugfs
Message-ID: <20220531162152.GH1343366@nvidia.com>
References: <20220527145910.GQ1343366@nvidia.com>
 <eda4d688-257b-d12a-56c0-0f9d3a10ef8c@linux.intel.com>
 <20220530121412.GX1343366@nvidia.com>
 <42623a73-c288-1c0d-7021-93caff4ffb6f@linux.intel.com>
 <20220531131052.GD1343366@nvidia.com>
 <60318d83-e22f-f922-436f-6c31bce24d59@linux.intel.com>
 <20220531145301.GE1343366@nvidia.com>
 <a7d6d830-cb06-e0d7-0688-028f9af900e5@arm.com>
 <20220531151332.GF1343366@nvidia.com>
 <b66a2e3b-9adc-5150-fe00-d68b141b1c28@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <b66a2e3b-9adc-5150-fe00-d68b141b1c28@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?jTBnr6V63ebUXlCPav7XYSUAZ+ROmgd/U/GnfwPVF3glC9VeuNIYDSlBgGGC?=
 =?us-ascii?Q?h2OOHJTdo/6BdSc+EzO0jqkGofjfHrE9AQnRu/F7x9LO9mmsnkU0rv28H4oK?=
 =?us-ascii?Q?31tGd28msInTAs15kmfSHYVhs8De52AZuN0TtiTweiUyvxAiVhWgUkrijPri?=
 =?us-ascii?Q?AN/FoTw89Ra4gRAtVp7D36wenQ/0L0MvntiiQ+vZe00QcuQt3Fkaf2XzqbLL?=
 =?us-ascii?Q?sDKhXA3bx7IaGxOnfblD6HdapAW+M+NEp5N9xH53hiagLGDyluVUpm7bKOEh?=
 =?us-ascii?Q?vAb4ugl0zOl2fKcfflKnFRfUX+lmr8kaQHcJ2qaMtC9853w31j8K1Pq/CV70?=
 =?us-ascii?Q?vRuZJdfg4U/+o1W16kKMQJZ/HYz+gq9tA5RVvUZ2a2S5weoxAUpAvCFimZl3?=
 =?us-ascii?Q?Xi0/igLF0hQCEb2MDnSdNuQsLLVYgIKhxC1rYe51oL4+Zv1FmGDx1eWSZLgS?=
 =?us-ascii?Q?/fwyxe0KICaDYGhOhYYZ9hMtF8sz9/JwVbl8OplQVcblMKNFlR6xwH/pXK+M?=
 =?us-ascii?Q?ql7yDQN0cCTBiKcS1DUid8verMJEJf7vFEbicjr0I/akKxTQRd4nhUy4pVeB?=
 =?us-ascii?Q?FuH6Dm26hXZdqnW+0ydU25kglZ1oxFkThHBcwlW8O6U5PhTQI21P3JZ4Vnfk?=
 =?us-ascii?Q?gjHHzAiMgxDr+wZhBBGBK9TkEsr2oY6sVMmyFEr2NGchKHvQDAgZJkx5mg1k?=
 =?us-ascii?Q?3hMSCHZpAzF4wiOQs5UzF8dLCyc6XfxiQP3AhXHUWvqrFlRPoCh8c4Y2oR8c?=
 =?us-ascii?Q?x+BZfx+1uM1BlinwXBPheqsyrK7whfLHUHCAoooQ4NyeQj6tQpXz3Z0ROqbK?=
 =?us-ascii?Q?kG3OEv0O9Fi2bh42BswEdp/yFPEua4e5qP+ZtxU4q8Lpg32jJjLK8d6NyfW1?=
 =?us-ascii?Q?SwgDAuR8w9mbDkAN0K5Lo+PlOFxucjtNAf3TE5t1UL2yCMRQtkMMotMfQZpj?=
 =?us-ascii?Q?Y/YdMo+FEZ0BmcsPEvJkbL8R4QIEY7vsM22HFqmYx+93YRqdlXJ4Ad2qN/jj?=
 =?us-ascii?Q?OSz3dxlehmfxk4yftfOxMFPCM5ERt6yrs7OZwQlX3lgf+z0Rzlmx1S1r9Ni7?=
 =?us-ascii?Q?yto22D2tqwNNBCWZ03baUxrSmfG/8VmqIngH47vkTe2ICpLasIjnPLfOPSVf?=
 =?us-ascii?Q?WaQLaQKRm/W8EnxWHO1wEKcEhroI8p3qgE2rK5LRpsS/trqx9VpZzN+5BzXp?=
 =?us-ascii?Q?+xRP5ORWi/XnjSH5ynKDFhQdDqYA+aDMOTo1IzGq/lKYFnTNg8YKVsrJSS+o?=
 =?us-ascii?Q?lhrMeLwtrA1ZPZ2874eHK9BhTZIBUVFmJ+Z5dyD8qqIgImiZ47LZLfvAwa9W?=
 =?us-ascii?Q?MyR60C+J5Pp0sJVvBaYgVPLRcu6kqcjMLUyxkSqE9IH5NR3p34x6cCOzI6fX?=
 =?us-ascii?Q?DXNFXCnV+DMQ61UhroTjjYGFf2aAhznl7Ah1tHjgxZ8VAW44ztSfyKiXX0vs?=
 =?us-ascii?Q?ViqabfhNLQWfOdWhiO8pXGmBrLrNbwULLQc/dosml+7jn3fJyb/j3I5XtMrU?=
 =?us-ascii?Q?x4eSbILUvh1Cco08L9b0jjW50foiOaj3Vm8x4xvN33DshWO0VNXmeopufn+B?=
 =?us-ascii?Q?K+8Hf4sj3hy3RBSVyOYeF0L7GbRIJtGhw6sq1gXCWXbddOoATRZQWwnlwFci?=
 =?us-ascii?Q?qGO8bHshAbplE4up/kEGQrXJaV6gpIPWTOWXUp+Mzx2/tObPyJznCMY7Htyc?=
 =?us-ascii?Q?DRK7KsriqeJ9XC2FOlFoIKzmRrCPB+3V9XfDlW+QE17W5vWTwwGujcMkcAdL?=
 =?us-ascii?Q?fNN8QEP7Qg=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 76a80500-6486-437a-d82b-08da4321acd6
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2022 16:21:54.0339
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tonsp2Jz6JFFb8YtTBFsdsanesotTBZdiaJxXt3l81rCjM5ijaaAFoBOHXTXURp0
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2647
X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, May 31, 2022 at 05:01:46PM +0100, Robin Murphy wrote:

> The DMA API doesn't need locking, partly since it can trust itself not to do
> stupid things, and mostly because it's DMA API performance that's
> fundamentally incompatible with serialisation anyway. Why do you think we
> have a complicated per-CPU IOVA caching mechanism, if not to support big
> multi-queue devices with multiple CPU threads mapping/unmapping in different
> parts of the same DMA domain concurrently?

Well, per-CPU is a form of locking.

So what are the actual locking rules here? We can call map/unmap
concurrently but not if ... ?

IOVA overlaps?

And we expect the iommu driver to be unable to free page table levels
that have IOVA boundaries in them?

> The simpler drivers already serialise on a per-domain lock internally, while
> the more performance-focused ones implement lock-free atomic pagetable
> management in a similar style to CPU arch code; either way it should work
> fine as-is.

The mm has page table locks at every level and generally expects them
to be held for a lot of manipulations. There are some lockless cases,
but it is not as aggressive as this sounds.

> The difference with debugfs is that it's a completely orthogonal
> side-channel - an iommu_domain user like VFIO or iommu-dma can make sure its
> *own* API usage is sane, but can't be aware of the user triggering some
> driver-internal introspection of that domain in a manner that could race
> more harmfully. 

The mm solution to this problem is to RCU free the page table
levels. This way something like debugfs can read a page table under
RCU completely safely, though incoherently, and there is no
performance cost on the map/unmap fast path side.

Today struct page has a rcu_head that can be used to rcu free it, so
it costs nothing.

Jason