Date:   Fri, 10 Mar 2023 11:25:04 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Robin Murphy <robin.murphy@arm.com>
Cc:     Jean-Philippe Brucker <jean-philippe@linaro.org>,
        Nicolin Chen <nicolinc@nvidia.com>, will@kernel.org,
        eric.auger@redhat.com, kevin.tian@intel.com,
        baolu.lu@linux.intel.com, joro@8bytes.org,
        shameerali.kolothum.thodi@huawei.com,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org, yi.l.liu@intel.com
Subject: Re: [PATCH v1 02/14] iommufd: Add nesting related data structures
 for ARM SMMUv3
Message-ID: <ZAtL0Dx0QRbXOTfZ@nvidia.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <364cfbe5b228ab178093db2de13fa3accf7a6120.1678348754.git.nicolinc@nvidia.com>
 <20230309134217.GA1673607@myrica>
 <ZAnx0lUkw02cVTi+@nvidia.com>
 <20230309182659.GA1710571@myrica>
 <ZApJGwPjHhlDwTDV@nvidia.com>
 <82f5b94b-01fe-5c99-608c-f7d124247b7c@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <82f5b94b-01fe-5c99-608c-f7d124247b7c@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pSQucLOmu+eCQaz5opisn0QWPsOh2xHRjVQsyh7ZkvDAIbA8ew0nD+BDijTy?=
 =?us-ascii?Q?p6MhcJXrrT0LTXgeSv613WbqxNaMUWdx+Yh99JDFKUEIlsdhkuIYJBpC0GzE?=
 =?us-ascii?Q?FvkDcCX9I6PzaOU9lwXQmvPJnAFXftgs8Cu1rCMGFH39mwYzGkBe1cajZjlP?=
 =?us-ascii?Q?z1/bjnm/Z6WUd0kwEkdRYEHF/uJ8PDG01Yk9XHEjRiTbhtv21dWXPe9pxxa1?=
 =?us-ascii?Q?gRxjkcvR3PV5To2yH/rHTENzvVeXue9onv+Qok4A+Wsm4Q/Rj/WVJgdHNwQE?=
 =?us-ascii?Q?65Wa/s+STkPypcEdKrzNmwKEgs/QzRwlKBsFA3D3u6zIN0CMwMoIL1KRnpa5?=
 =?us-ascii?Q?qabSLuKknOub6vZYqNMOgdvySSeLFjb+N+t6ret2u/VVoPHBWhMZdri5jhD0?=
 =?us-ascii?Q?QmrNfr4LMqOWTfu44+IIWQVTPg0peeNM1H2hizwOCnZL+Q8TtN6riAl7Tc1i?=
 =?us-ascii?Q?HTkY51LAwt1ptZ3mYJ5MDHgX/n8WoBywWQhDrw5lrY6XYcN7RanL3uQCxSRs?=
 =?us-ascii?Q?9CqReccSHmuuXYl3iW04/8t97BiBEPp3T79lho4LryV2HYgQOfJ8L7xAEZhk?=
 =?us-ascii?Q?/8gRIiwm77uwxLE5zXQprHmUIOGpnpOp/4jbnOyP6kPjbZWciMvDFvCzPvLX?=
 =?us-ascii?Q?USAUpXreUNPZDlNoeiJi0ZGXWEN6gmk3vqU3ogR16lts1gRqpnCfgq+F7KuU?=
 =?us-ascii?Q?OrL5TC7wBTZEP5GesaUcTr4HyYRHk4Yalz8AA1tY/+/fvPuAm11DRbf716Lx?=
 =?us-ascii?Q?YFZZbr4OOdIt2ssxtG+uTrBa6O1q8HYlAJX0duAnYi7RN7UCiWGbvEOyVIL4?=
 =?us-ascii?Q?b4mIw2laLg8T9JisGZH31ngNM6s6PjEm0IYbnH/VP+3XeQpwjSf4E+k//GlU?=
 =?us-ascii?Q?zNagHr6u2H5Fgi7QIjDgq8SLaq/A9MQtQjp7y32UNeLIERkiPDfrnmsgz5oO?=
 =?us-ascii?Q?eMWqPoGJh9PaNowxxcvGP+fXx5XyBjaFPFmjQ/BZGq/CfcNWc+Frc22jW7mo?=
 =?us-ascii?Q?Gpjb8OlZLpO4Lo+Tp5qtKlUAvGBD/8DeTWrFvKeWB3+1/NmpJWBEBLQhFg8S?=
 =?us-ascii?Q?1O9tLz2amj9kzRW5GUNqWF/GWG1nl5nxOs/PHNPGpUB0n5sXoPrF0lXzhbnw?=
 =?us-ascii?Q?7EiUyVjr5TKtrokBnLaBUBna8hbSfeZ+AIJq6j2XJaGtI4/J5rIQEayaOOBV?=
 =?us-ascii?Q?jlpUULoiZN719PVt7IVkkELWgHEblGyIxnuonnazCrlVFvb2CnoOva6VAhmP?=
 =?us-ascii?Q?2oscKOyyvUtI7Q7sbeIHlllcvc8nNGMdwTzauW052qbOJhWbTaXZdWDMxhiM?=
 =?us-ascii?Q?7aEvJSEHAq/ExsmQqjm/7EBNWaxDgLHD31vEsZ5hjZ6c4kUoE2OpZUOEQw6H?=
 =?us-ascii?Q?J0artBmisNuOpb4GRnV0giXPo/2E9IsJf+dyUO9bA5BsUlw6AIIKK7iqdc9g?=
 =?us-ascii?Q?tiytYu4EXyWhk5aw9AkSPWMw4avir6neVMSI39tWBZg8LGXu3/1uOTfXnOJ1?=
 =?us-ascii?Q?2Xv82PyJRhHjcX6p2Q87tkBXh7JI8E4GGS8F88IFcP7Hbor/2de0wspioeQC?=
 =?us-ascii?Q?kBG4EYOrBnyb9offm4ja4jRfmGP5hFjYussHmTnp?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 26bc74cb-2570-45de-fde7-08db217ba13d
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2023 15:25:07.4351
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: loDTmUxYYQu9g+uTGUn7W7L0WolCaGltx8+JenVv8kzm5x5Ayt7TVE/Wc77PTPmq
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5760
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Mar 10, 2023 at 02:52:42PM +0000, Robin Murphy wrote:
> On 2023-03-09 21:01, Jason Gunthorpe wrote:
> > > For a lot of SMMUv3 implementations that have a single queue and for
> > > other architectures, we can do better than hardware emulation.
> > 
> > How is using a SW emulated virtio formatted queue better than using a
> > SW emulated SMMUv3 ECMDQ?
> 
> Since it's not been said, the really big thing is that virtio explicitly
> informs the host whenever the guest maps something. Emulating SMMUv3 means
> the host has to chase all the pagetable pointers in guest memory and trap
> writes such that it has visibility of invalid->valid transitions and can
> update the physical shadow pagetable correspondingly.

Sorry, I mean in the context of future virtio-iommu that is providing
nested translation.

eg why would anyone want to use virtio to provide SMMUv3 based HW
accelerated nesting?

Jean suggested that the invalidation flow for virtio-iommu could be
faster because it is in kernel, but I'm saying that we could also make
the SMMUv3 invalidation in-kernel with the same basic technique. (and
actively wondering if we should put more focus on that)

I understand the appeal of the virtio scheme with its current
map/unmap interface.

I could also see some appeal of a simple virtio-iommu SVA that could
point map a CPU page table as an option. The guest already has to know
how to manage these anyhow so it is nicely general.

If iommufd could provide a general cross-driver API to set exactly
that scenario up then VMM code could also be general. That seems
prettty interesting.

But if the plan is to expose more detailed stuff like the CD or GCR3
PASID tables as something the guest has to manipulate and then a bunch
of special invalidation to support that, and VMM code to back it, then
I'm questioning the whole point. We lost the generality.

Just use the normal HW accelerated SMMUv3 nesting model instead.

If virtio-iommu SVA is really important for ARM then I'd suggest
SMMUv3 should gain a new HW capability to allowed the CD table to be
in hypervisor memory so it works consistently for virtio-iommu SVA.

Jason