Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Date:   Thu, 30 Nov 2023 20:45:23 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: Re: [PATCH v6 2/6] iommufd: Add IOMMU_HWPT_INVALIDATE
Message-ID: <20231201004523.GJ1389974@nvidia.com>
References: <BN9PR11MB52761A9B48A25E89BEECE6308CB8A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZWTzoBTDDEWAKMs9@Asurada-Nvidia>
 <BN9PR11MB5276FD60A0EDF8E3F231FCC88CBCA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZWaLCSAMIOXTlghk@Asurada-Nvidia>
 <20231129005715.GS436702@nvidia.com>
 <ZWaPM4p7yjJ0sEKk@Asurada-Nvidia>
 <20231129195804.GF436702@nvidia.com>
 <ZWe2PvatTkkyNCY5@Asurada-Nvidia>
 <20231130000816.GB1389974@nvidia.com>
 <ZWjzcEAAg8ptVH4A@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZWjzcEAAg8ptVH4A@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?chAu73OmQUzKoXjfB+rfLk4N6MIIpiiYW+kbImqoXPtPyTKkYGLJXVjn/rsE?=
 =?us-ascii?Q?taI5B2U98Zsmd7624J3aKHT/pOJWCmIORVunDxeoWUZkrWSgYQvHGviGjS29?=
 =?us-ascii?Q?/hTCGrzu+Zf339H1euK7qSXUcrdmCWgg4ruldP66/ylpsOOlNo6zfLXKzm6T?=
 =?us-ascii?Q?gCBLoVdeZCKPR61yXtIddfw5o3q8TVIK8iW4L0/vNmhJa9tug6TD5Vz33PX/?=
 =?us-ascii?Q?IITMW3LMN0Bm0RHE77Bf7xURbTZlodpt2/IwTYS2dwmgLjyIJfMp9Rdc7LFA?=
 =?us-ascii?Q?wD2ExEZsftU5Bfuz4OlYVAbQKSlvnIKXy/fVvwzyxoHCUO1QK58aDSWkw9j3?=
 =?us-ascii?Q?sdfLPNd44pT1ehlUYRO5UQcHid/c4EqGna4Z3POlXRjtQA9DmTz9TocU8XL4?=
 =?us-ascii?Q?hNdkrBfrNrgKJaY7BRuaptYN1CosAG9Mkaeix/2tEzqhQMShU4qxNUen4jOk?=
 =?us-ascii?Q?3fCi3KqZOtyTua1+5wOsL3HVZMYu8HcFbLypuWIsoGtgobTmwPi0zBJphMOg?=
 =?us-ascii?Q?HnErN581x8phHfPeLXKHv3JRslMHg5M81VwiAtClUmu2fWui1vMZAiWacAkg?=
 =?us-ascii?Q?frBEPosgA9HT0PYQli8i1bZtBoJdqdKTH5fDRJYcsv7c0WaAN1UYecslhHdJ?=
 =?us-ascii?Q?b67RShGmTAQdz+kyc9NbpB/vGaz+Nq86IvwsVWGl/OsPzOTwB1PrkK1cgkbz?=
 =?us-ascii?Q?Hu0yqFbaUQcch2MexSfMRXbFsoW8WKb28utt9C/X3purG9QeIzofCb6YYI3E?=
 =?us-ascii?Q?AhRGmGkf8CxKZT4KPOaWqSyv1eAHAsr6Ug1+f3TAQKYmt8FdFCqMghSHdEne?=
 =?us-ascii?Q?wdJBcB+a60nJaTwOf9j2kuDWLA/y9UxcHyCMXWyYlh7oIRP2367Azf/d8BrY?=
 =?us-ascii?Q?Z+QkT4UThSuBc1AbUe4CqZ3MdpMbZOebFFfN8hh0YY6sw93mwB74oTocgWdi?=
 =?us-ascii?Q?ZR7R/8xpSWX1Q6TTmtaw4x1PgLpZJI86d2surzy5xFPoCO0QPLy3nzWDqIOP?=
 =?us-ascii?Q?YmbP9/It9FwvwLcZYzMumQNWA7/Ncf5nbeJuTWe6y6DgUK2HMEWhdhDiximl?=
 =?us-ascii?Q?uM/mzl0Lw3E/G5EGfVhKb6dON68ZqwlfaU9yfB4TyaEox+qVx1B24/E1z+ZX?=
 =?us-ascii?Q?/G32dy0w7/rbzFu/TnZ2NVHeX8wTnWQAVBfRCz4kaD8mvsDUtOsBlW5HX9v4?=
 =?us-ascii?Q?Av6XxGfNTwa4pwxjSXRpPBX16neDdLQ92Zh03lU0KnT177omtmVYb8OBEeVK?=
 =?us-ascii?Q?i9PHgkKf8Mdj6HyDHauIvYQddiHr+dHCuanRTFDyKLzQONRry0kVhhR6H9gk?=
 =?us-ascii?Q?ANYJ3ahSNhm3to/JdXcAUBP2YHz2OPz0ltTpfAN3vHFZksSXXLI3pjX6rtfb?=
 =?us-ascii?Q?xYXRB4CTemVbNDJV+9Wh144eDExpYtMNgduNWylqxrp/R4mE1G9Xv6AbIlJU?=
 =?us-ascii?Q?OjxFQdUZCxitbmMsVrTmf+X9xu27LHYXNk00NfKr6Lfd7yfj80IzxSWS3i2p?=
 =?us-ascii?Q?anCqkol+UstqnDqB2ogp4Y77rzwB3GqIHgzsc3JEgw0cD3fscjy7rg0TtFLI?=
 =?us-ascii?Q?4jciWWNGKQTnkTBCtzFTHqHi0tgMcv2V8Ww91HVM?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b9193dd1-c593-451f-a729-08dbf206ce56
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 00:45:25.0390
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cNAm1m3Topjrt+n4jbb9hYXLYNK0dwVra0CNjGIOu9+9HjmMK7ebdnF5VwdC3g2D
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4320
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 30 Nov 2023 16:45:36 -0800 (PST)

On Thu, Nov 30, 2023 at 12:41:20PM -0800, Nicolin Chen wrote:

> > So userspace would have to read the event FD
> > before returning to be correct?
> > 
> > Maybe the kernel can somehow return a flag to indicate the event fd
> > has data in it?
> > 
> > If yes then all errors would flow through the event fd?
> 
> I think it'd be nicer to return an immediate error to stop guest
> CMDQ to raise a fault there accordingly, similar to returning a
> -EIO for a bad STE in your SMMU part-3 series.
> 
> If the "return a flag" is an errno of the ioctl, it could work by
> reading from a separate memory that belongs to the event fd. Yet,
> in this case, an eventfd signal (assuming there is one to trigger
> VMM's fault handler) becomes unnecessary, since the invalidation
> ioctl is already handling it?

My concern is how does all this fit together and do we push the right
things to the right places in the right order when an error occurs.

I did not study the spec carefully to see what exactly is supposed to
happen here, and I don't see things in Linux that make me think it
particularly cares..

ie Linux doesn't seem like it will know that an async event was even
triggered while processing the sync to generate an EIO. It looks like
it just gets ETIMEDOUT? Presumably we should be checking the event
queue to detect a pushed error?

It is worth understanding if the spec has language that requires
certain order so we can try to follow it.

Jason