Date:   Mon, 20 Mar 2023 19:19:34 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     Robin Murphy <robin.murphy@arm.com>, will@kernel.org,
        eric.auger@redhat.com, kevin.tian@intel.com,
        baolu.lu@linux.intel.com, joro@8bytes.org,
        shameerali.kolothum.thodi@huawei.com, jean-philippe@linaro.org,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZBjb9sKkAddFmtOU@nvidia.com>
References: <ZAn7uC9UweiNdGkJ@nvidia.com>
 <ZAqv87fjbdynVaHA@Asurada-Nvidia>
 <ZAtYphmOuEqQ1BiC@nvidia.com>
 <ZAxsgjuuG3ziqs9E@Asurada-Nvidia>
 <ZBhZiCOQHAxEaamy@nvidia.com>
 <ZBiCEAqP1OU5EvoG@Asurada-Nvidia>
 <ZBiEE+QLjPpxBjkQ@nvidia.com>
 <ZBiRAVROma5/Os2S@Asurada-Nvidia>
 <ZBip4sbxUQo8RjGv@nvidia.com>
 <ZBjOooJZcUf182ES@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZBjOooJZcUf182ES@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?8cQKnIyDEZW2nVnSTJDSiuWKcNRpgQJFxS8a6czB5r10mekoWFHS/SbmYHcc?=
 =?us-ascii?Q?SykMHO+hFnHJYjwYzrgoqEF3KYfNt/hEtGV43vQfsGPSsKKWQlFN98/4Nxql?=
 =?us-ascii?Q?hxdeXo3rOQj1OdgNjmABOOhBRpw4Wiif+TvdDq6YnpXcl57WfJCBSA2ioK4Z?=
 =?us-ascii?Q?gQ9hdUouO+MoGed7wwn/4dgJoGNlDSoHN2qHdvcIhXZhAgUwhgVK8jvJFI/u?=
 =?us-ascii?Q?zJjxCcR1t3Fh4deNcsVKkQNOS3uDIUYRENA8jhqaC2ttKfGARGA9DuugCLGD?=
 =?us-ascii?Q?4Hjuc3Pf6vuRwrGpoqWr1oJW1tGQ4XA4oJsJF0q8NbcWPYboVtTiJZ4qLD0i?=
 =?us-ascii?Q?4cjzy/ON2DMmvZrTgZ7mw1dTNgN31ipNrZbvOBVfVpmZOaMcvZyrtn+FQOWF?=
 =?us-ascii?Q?gNL/6xQPmhJ+MbS5OJlCG9zW9sRsQtc8k4B5QPOCAVP1sYcN+nf8gQ70F/6u?=
 =?us-ascii?Q?R351wGQf4UxpuXMe8jSwyMTDDJ9lsiz9OeQPkQlhMvjKTgIUOok/kvMBqaB8?=
 =?us-ascii?Q?81d0m0J9kYttWwTszapyDXL7pXnWUbMImDWsvBauAhGXz78frUi3EFycTLl8?=
 =?us-ascii?Q?beqbE//kjqU5BrwROdD5tjdsJm6jzOljvnM94Wmzvq6+dssgHAU23qxB4+XZ?=
 =?us-ascii?Q?5g+KqWRjBjduuTx1X9c+9Vjy6wmy/1e3cAC2aFWLSKGqwjxdVEdCdKiZSblU?=
 =?us-ascii?Q?xDVLHYIhY1Pw4MLwOXmApaDcSkE+LO3KUx3bazbavv2Fff80RdY2VD029/Qg?=
 =?us-ascii?Q?UNv/ivK7I5HU2RoDWW9+8L/mx1kzsZqU/ol12HQNSfO0dEurxh0SvZYTKZxp?=
 =?us-ascii?Q?2EhpkYTh6AP8KhudwZ6VMqzYXhM4fg6ohjmleCX1hSG+dytRUfpZM2JUouJ3?=
 =?us-ascii?Q?AP8c42e4fC3xkw1L2TN6ZK2qHEHuUDs86vowA8E97mlpN/L8SOZzkJzPBt08?=
 =?us-ascii?Q?CoKd5z2N12TKSY2Lp68EQsKQK6r38JKnK/HZZi6APEStfFUFF1jr6587yjEV?=
 =?us-ascii?Q?X9CzKQOLbrNv5H3hOzSGIUlT25WUkHBIK8B/DrcIG6ivFFSloK/JaRTDikLe?=
 =?us-ascii?Q?amNmLVOLSX/0dpbHG2H2TrxTcfMXqoK++nMMeiLiuNcIq1tDw4k1vRQTrLki?=
 =?us-ascii?Q?Ru+vnMHS3lgXgoZ7HFt+9p8M4s+q3ASVPIt8g5R7ZhzZI3UwqnkNC71R225E?=
 =?us-ascii?Q?4AldN3aWfH6+asq5306tXoVmJGP6i/JAMwxp9uZOzuLln9/QG+IF54W2N8Qd?=
 =?us-ascii?Q?4z62uGJ8ls+Pmkc/4+AoZHl3p+VFu8zASpFIaHqmJaZ/GAaNQU6MkcVTSrf8?=
 =?us-ascii?Q?stC679Xk8VaxB3j6VbEJtcMhzEVhzdrgqNDdsqknvxD/uhKMHY6xvH+6ysJl?=
 =?us-ascii?Q?PM6PNSVpYhOFSSdknXNr6XejUzknBNL58Sos/WE1Z8iHipqWffZBkzP1w3Da?=
 =?us-ascii?Q?lDgSQ3sC18xFtneBR8VAZ7h5El4GHq6PICJ1IznFWxK1wYmNCC+hW4MD+sKE?=
 =?us-ascii?Q?2FGgXb+iN+rZcqvzv91WRIjswKboHXhbE36uDTJI3Qxgbf/mXXOUyRRK+39n?=
 =?us-ascii?Q?9v+zIIhk0XF1LaTsmnS4pauQdwbqXdjmcJ4ak7fx?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e748c4d3-7967-4821-281d-08db2991305e
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2023 22:19:36.3093
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: l2oB70yffUWAe1QMnSHEi516OTq1n+TyK5+4EfXgNqATikptggvJkV4LyybfMZ5v
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6161
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 20, 2023 at 02:22:42PM -0700, Nicolin Chen wrote:
> > > What QEMU does is trapping an SMMU_CMD_CFGI_STE command to send
> > > the host an HWPT alloc ioctl. The former one is based on an SID
> > > or a device, while the latter one is based on ASID.
> > > 
> > > So the correct way should be for QEMU to maintain an ASID-based
> > > list, corresponding to the s1ctxptr from STEs, and only send an
> > > alloc ioctl upon a new s1ctxptr/ASID. Meanwhile, at every trap
> > > of SMMU_CMD_CFGI_STE, it calls a separate ioctl to tie a vSID to
> > > a dev_id (and pSID accordingly).
> > 
> > It is not ASID, it just s1ctxptr's - de-duplicate them.
> 
> SMMU has "ASID" too. And it's one per CD table. It can be also
> seen as one per iommu_domain.

Yes and no, the ASID in ARM is per CDE not per CD table. It is
associated with each TTB0/1 pointer and is effectively the handle for
the IOPTEs.

Every iommu_domain that has a TTB0/1 (ie represents IOPTEs) should
have an ASID.

The "nested" iommu_domains don't represent IOPTEs and don't have ASIDs.

The nested domains are just "shadow STEs".

> > Do something about SMMUv3 not being able to interwork iommu_domains
> > across instances
> 
> I don't follow this one. Device instances?

There is some code that makes sure each iommu_domain is hooked to only
one smmu driver instance, IIRC.
 
> IIUIC, the ioctl for the link of vSID/dev_id should happen at
> the stage when boot boots, while the HWPT alloc ioctl happens
> at CFGI_STE.

Yes
 
> > > What could be a good prototype of the ioctl? Would it be a VFIO
> > > device one or IOMMUFD one?
> > 
> > If we load the vSID table it should be a iommufd one, linked to the
> > ARM SMMUv3 driver and probably take in a pointer to an array of
> > vSID/dev_id pairs. Maybe an add/remove type of operation.
> 
> Will try some solution.

It is only necessary if you want to do batching

For non-batching the SID invalidation should be done differently with
a device_id input instead. That is a bit tricky to organize as you
want iommufd to get back a 'struct device *' from the ID.

Jason