Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Nicolin Chen <nicolinc@nvidia.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>
CC:     Jason Gunthorpe <jgg@nvidia.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: RE: [PATCH v6 2/6] iommufd: Add IOMMU_HWPT_INVALIDATE
Thread-Topic: [PATCH v6 2/6] iommufd: Add IOMMU_HWPT_INVALIDATE
Thread-Index: AQHaGVcJ4nFLu64FV02Qq9BjVpwSvbCC3axQgAAHEwCAAAD1wIAAl++AgACaJQCAACuAgIAEgqsQgAXeCQCAAMsJ8IABGqOAgAABpoCAA1VUAIAAEHmAgAADm8A=
Date:   Fri, 1 Dec 2023 05:19:13 +0000
Message-ID: <BN9PR11MB52761168800700D7131D601D8C81A@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <BN9PR11MB527659462CCB7280055858D98CB4A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZVuZOYFzAaCuJjXZ@Asurada-Nvidia>
 <BN9PR11MB5276C8EACE2C300A646EA8A18CBBA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZVw/BXxgGCuCZCA6@Asurada-Nvidia>
 <BN9PR11MB52761A9B48A25E89BEECE6308CB8A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZWTzoBTDDEWAKMs9@Asurada-Nvidia>
 <BN9PR11MB5276FD60A0EDF8E3F231FCC88CBCA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZWaLCSAMIOXTlghk@Asurada-Nvidia> <20231129005715.GS436702@nvidia.com>
 <b5f86fde-eaec-47fc-8b4f-36adb0e9e1a1@intel.com>
 <ZWlmD1KDUyR3qzdy@Asurada-Nvidia>
In-Reply-To: <ZWlmD1KDUyR3qzdy@Asurada-Nvidia>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?8+nZQNjGDS1N74aBa6xBP+h01e0swZiTF6H8hwoVjLFnabsKr4ZQyKz6s/3b?=
 =?us-ascii?Q?DM0ck//RZHdMxqXxeOnLtamZOdwOzUCHP0vdIOKiLj6PeCNnYr8xB7NnJiNQ?=
 =?us-ascii?Q?xw9GT5MQ5NsAS8AlpIaeBcm4iFCKlH2ZWaGhBJO8VZqlQsCW1w2I87dLR0ky?=
 =?us-ascii?Q?cybdJ68s/EnAmEcGBlHXgFRB7+dFdSnyrepOnlPra9PCnCy7iwQCvdoKyXC5?=
 =?us-ascii?Q?abj/RxjN2M78+fypqzG/Iq8/83UZzB+fYYguFo1fye4jJrz02a7On2owWzNj?=
 =?us-ascii?Q?Zc+gXc+QQnF8MuQs/8iJDpFXU8yD4rB53qNakLpcIlDT39PGi1pFnBc5IMTO?=
 =?us-ascii?Q?Zc6Cksi/tDFsDDAkS5Z3aW1puWjMwnOiSN3pHZdyq+iPSO9hqpKrdzWVz/fP?=
 =?us-ascii?Q?xxwzKfHrAnZVl97Gup75fu41vghnH5iDBaiBcLHhzuqbZHCGgcptrNQzGYKg?=
 =?us-ascii?Q?muSw5VwXgX/A74a4bfWdOBnNvs/Obrys/5sQ6bYeJJVIDsSmew5N5Ip6niGH?=
 =?us-ascii?Q?jDUtgpsW4XdvMpya5Mu46r7lpjIamJcVSnRahb1TvIXvcznzf+MzgBxfq8Ag?=
 =?us-ascii?Q?MDRQT4c2XytwrK+LQjg8jA+ceuz2Qzjge39JuUpt+GkfkapY6xPeF7EeSx6R?=
 =?us-ascii?Q?7op2JVF7hK0LU/TtUDhDwe/9jhii3mfTo9nw9rGyNSd1JACHS64DVJlNV42K?=
 =?us-ascii?Q?RKjp4VGiTDiTb9NFXux9MnqlRQBXUhKR/eNU+3BBfoXlPpI2D+MgmufUAgqc?=
 =?us-ascii?Q?L9gQOMnpyNt09SZsNVefbHR2nRP219asANtTYNyFCavo7YlDO6TD1zCHpYKv?=
 =?us-ascii?Q?oo9WC9BjqNncMgkVyMBfw+WKLV6QIf4hboOxy3S+qjjE11XUyQeHbR8L/PR0?=
 =?us-ascii?Q?SzhQLfVMkNXCUhPKV7TArW8wYSc2TkxVPTErvGZ3fyndjwsvnSxu5ti/x58p?=
 =?us-ascii?Q?sA6NePXy5IAhyNsi9PMhNTfdOy2YeKUVNxG19Eeqvy3PV2WSe1+/uK0YGq1X?=
 =?us-ascii?Q?vaFVXtineYbZiawEMdG49m8LnF17lJO7ULguLE0tX7dZF/VDYbJdeia4IUcg?=
 =?us-ascii?Q?L5sre7LI45Zz01ykKsOdHW+SYiGUYHYyNP/pEPRUFWvU6yL/q7vKqz5HnD52?=
 =?us-ascii?Q?upyrgM7zexSOOhFu9IxFSlyU9qKV2r2Xrv40agbcl9+NO+iyzuujyfK7Oh01?=
 =?us-ascii?Q?EOsFpbSG/vzgk4pJdzHwdPQk6Fa4Zea02BsBwh+9R+hKkwrSgn6SDY0QCLtM?=
 =?us-ascii?Q?xNAi+L+MXjH55CNFfOSHLxXCAgI7EOjzli4j5y3epiZ66VYW2cWRG16t//wv?=
 =?us-ascii?Q?ZcUezfPDzVb/li2vhDv1LMKZkNQAbE5Mf4eMPQgTiwA8VwbuH9NYT5TWzm8Y?=
 =?us-ascii?Q?CEPUfcGQ/s8YxF28o+REplmGrWwAJIXseZGjBh2tqdnnDcK9IsI7ULOLHRLA?=
 =?us-ascii?Q?LQRirWBXq/wTHxYKcTmhTS4Wb2icJg+oykErpvksmgP3UCjolbGqD484Nuyg?=
 =?us-ascii?Q?JXfJwhZDOVc36qwXhUrz7iqQjTseg5kEUXhFtkh5aPGafK4s7d0N88xTS3yk?=
 =?us-ascii?Q?M0Kw4IvTS/N8j/PhoE2dyQ2y0TwxyVh93tTm6brj?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 20cfe8c0-df61-474b-8e9b-08dbf22d0ee7
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Dec 2023 05:19:13.9966
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: c7h4ZTrsLRFZLG7fQ4P5/I362kURiJfuLlPnE2VYpc85F9rrMImKE00PT1mU2xmOwe36OPiYSY+FrS+FmU0Uww==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5182
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 30 Nov 2023 21:19:30 -0800 (PST)

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, December 1, 2023 12:50 PM
>=20
> On Fri, Dec 01, 2023 at 11:51:26AM +0800, Yi Liu wrote:
> > On 2023/11/29 08:57, Jason Gunthorpe wrote:
> > > On Tue, Nov 28, 2023 at 04:51:21PM -0800, Nicolin Chen wrote:
> > > > > > I also thought about making this out_driver_error_code per HW.
> > > > > > Yet, an error can be either per array or per entry/quest. The
> > > > > > array-related error should be reported in the array structure
> > > > > > that is a core uAPI, v.s. the per-HW entry structure. Though
> > > > > > we could still report an array error in the entry structure
> > > > > > at the first entry (or indexed by "array->entry_num")?
> > > > > >
> > > > >
> > > > > why would there be an array error? array is just a software
> > > > > entity containing actual HW invalidation cmds. If there is
> > > > > any error with the array itself it should be reported via
> > > > > ioctl errno.
> > > >
> > > > User array reading is a software operation, but kernel array
> > > > reading is a hardware operation that can raise an error when
> > > > the memory location to the array is incorrect or so.
> > >
> > > Well, we shouldn't get into a situation like that.. By the time the H=
W
> > > got the address it should be valid.
> > >
> > > > With that being said, I think errno (-EIO) could do the job,
> > > > as you suggested too.
> > >
> > > Do we have any idea what HW failures can be generated by the
> commands
> > > this will execture? IIRC I don't remember seeing any smmu specific
> > > codes related to invalid invalidation? Everything is a valid input?
> > >
> > > Can vt-d fail single commands? What about AMD?
> >
> > Intel VT-d side, after each invalidation request, there is a wait
> > descriptor which either provide an interrupt or an address for the
> > hw to notify software the request before the wait descriptor has been
> > completed. While, if there is error happened on the invalidation reques=
t,
> > a flag (IQE, ICE, ITE) would be set in the Fault Status Register, and s=
ome
> > detailed information would be recorded in the Invalidation Queue Error
> > Record Register. So an invalidation request may be failed with some err=
or
> > reported. If no error, will return completion via the wait descriptor. =
Is
> > this what you mean by "fail a single command"?
>=20
> I see the current VT-d series marking those as "REVISIT". How
> will it report an error to the user space from those register?
>=20
> Are they global status registers so that it might be difficult
> to direct the error to the nested domain for an event fd?
>=20

They are global registers but invalidation queue is also the global
resource. intel-iommu driver polls the status register after queueing
new invalidation descriptors. The submission is serialized.

If the error is related to a descriptor itself (e.g. format issue) then
the head register points to the problematic descriptor so software
can direct it to the related domain.

If the error is related to device tlb invalidation (e.g. timeout) there
is no way to associate the error with a specific descriptor by current
spec. But intel-iommu driver batches descriptors per domain so
we can still direct the error to the nested domain.

But I don't see the need of doing it via eventfd.

The poll semantics in intel-iommu driver is essentially a sync model.
vt-d spec does allow software to optionally enable notification upon
those errors but it's not used so far.

With that I still prefer to having driver-specific error code defined
in the entry. If ARM is an event-driven model then we can define
that field at least in vtd specific data structure.

btw given vtd doesn't use native format in uAPI it doesn't make
sense to forward descriptor formatting errors back to userspace.
Those, if happen, are driver's own problem. intel-iommu driver
should verify the uAPI structure and return -EINVAL or proper
errno to userspace purely in software.

With that Yi please just define error codes for device tlb related
errors for vtd.

Thanks
Kevin