From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Robin Murphy <robin.murphy@arm.com>
CC:     Nicolin Chen <nicolinc@nvidia.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Thread-Topic: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Thread-Index: AQHZUnWe1bQRwtfhAUSI/T7BqHBa167yh7YAgAALsACADCcbEA==
Date:   Fri, 17 Mar 2023 09:24:16 +0000
Message-ID: <BN9PR11MB527665B62EFF58F07DA214338CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <aa327f9ea61e5a4771c13e53639e33955b9acde3.1678348754.git.nicolinc@nvidia.com>
 <1467e666-1b6c-c285-3f79-f8e8b088718b@arm.com> <ZAn7uC9UweiNdGkJ@nvidia.com>
In-Reply-To: <ZAn7uC9UweiNdGkJ@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lHu8Bo3mJNUkrJmnPyn3v0MfeHm9aw2pGjkqxfe1zLo2AGEan5M85jF0NJXR?=
 =?us-ascii?Q?Ug6eI1LtQBD+El5eUDptsfkfnbeMVQB7En9OxSib6wUvPYR1kk+pc5n1JPbm?=
 =?us-ascii?Q?cskFuJgd1Dh6wwG/OlItTcaBp8BHe1KUvRrEAEbbmH6PUPwGWdMlUH3HlBr8?=
 =?us-ascii?Q?9jQTVb6DTVaR7nZ47DMqyrO0ku9iK8eh4zJu00azD6Jok76Az4/sbDehhZQ9?=
 =?us-ascii?Q?irfDg1hmr9sdNcMn2KF1vyn7KrwupY8bG8J3B55okUvGXdbaczS6OMqUiDRC?=
 =?us-ascii?Q?fGoXVEpcJui/Kp+h6fMe1AoWBwdjY5KYochjgExEXEK0zVbh1qMpch930/az?=
 =?us-ascii?Q?W6mt1KjPVhCI+9gJvDwgEf1DnA6cnBNMGv4VQiaD2meafmiObMBQRL0zdz64?=
 =?us-ascii?Q?myrqxySm8/tO2eBJdAGqhtR3rJGpImjPPCbgm/MDmwTnCva0s5rtIqrhwYXZ?=
 =?us-ascii?Q?TZhASd7iv5S4ZrUWSgu+lcUFD3m4nWD8HoPmlQWFDjmMQKTBpx04hqhj2i4C?=
 =?us-ascii?Q?5LgieAotUmKEBo3kCxotU0bSBzSG6tKNxJf0rdpyaV/+v6LBubn2uUfYwsdW?=
 =?us-ascii?Q?VA5teOAsBBpTYTsbrDNglgEoTlPcprqGDqebvcW21gGPOBqqW5EAxgsGjl1n?=
 =?us-ascii?Q?rzygnP7+c6ZkUu6lLVmQiSq3LQkf9u5soQ2NwkaEmAl1Eh1LeYXMFfcLSQMj?=
 =?us-ascii?Q?AHCdAIoieSEl8QX3CwphzayGInMORMHWfVzaZ787J+EICtJ0ZSRSXOKUVb5q?=
 =?us-ascii?Q?WAK10cf4aZAG0BamyZI1SrphAITLbp4zrBNj10SK9iPg7z4tf1LK0feGDOPT?=
 =?us-ascii?Q?CqOWT97RwagzJEJtBSuWWH0IroymGpv4x1zDgSIaj0nZGB/ZGBInt+mSPsLn?=
 =?us-ascii?Q?1amsO9h86MA2jElpybO+v5RPSd6ekS9brkH05FEZrEdAgdCDeSRE9hD+D7sW?=
 =?us-ascii?Q?caPW+vleFSlvDjBJ6pHJbisC/jn2k7zeszY/UPxQOVhSM6XqufHsQjXOfwbR?=
 =?us-ascii?Q?3U5jiVIO4JqZUfTprm5HkrvigNfCKeDkFSsBNxIwO9JcmH4SIiq7/lbxwLLG?=
 =?us-ascii?Q?QpGy6MNSDgwMGrMhGcxC1d2LQyz+KSx1QP902Ugv56bRG40IL/1Q+g+Ll26Q?=
 =?us-ascii?Q?y4PLblWTPNWZLxdqwsyVIjoxwWfWXXEI4v7/doh4ZoWb/9N+jTmgrmU8kryP?=
 =?us-ascii?Q?m1KzZ8zG8ecOXsij9P1Q0YU2LE0JCFZca/HtTVjI76504nbq0ddNtpRxtgHg?=
 =?us-ascii?Q?od6vcegsEDHWiVtjXhEki1wb/xuuqIG5OZm/uAa+C0gXB7tTYju+InAcLshY?=
 =?us-ascii?Q?5beCPXrBnF7uNYXjoFHJHqjLllMH/5fxXwzR693ZE3+n03j0swi5TnYpxf0n?=
 =?us-ascii?Q?n2rKWdpMQM3X82aEQaa/BSJOZumAobKLRhZezXL6tBV3pbDjcDdQd4jwH3YZ?=
 =?us-ascii?Q?fgRUjm4nmBOMd4b7ORCKz/FiQ7G1iBA7qYqHD8Q6hKHRV8TMHpS6ANqEZowK?=
 =?us-ascii?Q?I8FkXxgGgkT9/elOTjSNYgRri5qKHiVrsU4zJ6Y0YwD9kPwEmbCNaHYu4nk4?=
 =?us-ascii?Q?lJyBOCJZlXdXabpF+CfZhG/5VW6/3sldS8YjJIS7?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f894f262-c3eb-462b-f5b4-08db26c96177
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2023 09:24:16.7965
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 2/s43VlZrrevhtILss8Tty3Om694bLdR2e40b4w5WTltBjib3yFeKBNK7Vm4Wvgwp5UfAnoeZKorKNNkd0ZXvw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6775
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, March 9, 2023 11:31 PM
>=20
> > Also, perhaps I've overlooked something obvious, but what's the
> procedure
> > for reflecting illegal commands back to userspace? Some of the things w=
e're
> > silently ignoring here would be expected to raise CERROR_ILL. Same goes
> for
> > all the other fault events which may occur due to invalid S1 config, co=
me to
> > think of it.
>=20
> Perhaps the ioctl should fail and the userpace viommu should inject
> this CERROR_ILL?
>=20
> But I'm also wondering if we are making a mistake to not just have the
> kernel driver to expose a SW work queue in its native format and the
> ioctl is only just 'read the queue'. Then it could (asynchronously!)
> push back answers, real or emulated, as well, including all error
> indications.
>=20
> I think we got down this synchronous one-ioctl-per-invalidation path
> because that was what the original generic stuff wanted to do. Is it
> what we really want? Kevin what is your perspective?
>=20

That's an interesting idea. I think the original synchronous model
also matches how intel-iommu driver works today. In most time
it does synchronous one-invalidation at one time.=20

Another problem is how to map invalidation scope in native descriptor
format to affected devices.

VT-d allows per-DID invalidation. This needs extra information to map
vDID to affected devices in the kernel.

It also allows a global invalidation type which invalidate all vDIDs. This
might be easy by simply looping every device bound to the iommufd_ctx.