Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Date:   Mon, 4 Dec 2023 10:48:50 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: Re: [PATCH v6 2/6] iommufd: Add IOMMU_HWPT_INVALIDATE
Message-ID: <20231204144850.GC1493156@nvidia.com>
References: <20231129195804.GF436702@nvidia.com>
 <ZWe2PvatTkkyNCY5@Asurada-Nvidia>
 <20231130000816.GB1389974@nvidia.com>
 <ZWjzcEAAg8ptVH4A@Asurada-Nvidia>
 <20231201004523.GJ1389974@nvidia.com>
 <ZWlhLk3JVwX0hRt/@Asurada-Nvidia>
 <20231201125538.GK1389974@nvidia.com>
 <ZWo6z59tnmS8F2V7@Asurada-Nvidia>
 <20231201204340.GA1493156@nvidia.com>
 <ZWpaTD9dVge+suyv@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZWpaTD9dVge+suyv@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gFilKasyDk9XSCS9hNh/9M8U+aHkewivnDXs8zRrU12/lNa5vlz2IyxgvN0X?=
 =?us-ascii?Q?wYfpLkzUtt2B+HNnAIkOgwctcR93jUIeb+fFVsQyQSH8zfjK2R4QsYXjUAnL?=
 =?us-ascii?Q?5nCGRD/nAn04q3tz+ee6v29BniWTfWI09LVn3/ZAxx6GNHsimYPfBg8FjVaT?=
 =?us-ascii?Q?5h/m6MN3Hs793Dgo36V/+2EHIjAs6Q6wp+fIXoYtggMUxxTm9Ogg0I49MY9x?=
 =?us-ascii?Q?OALTWx1J5nYpBcD9YY3CEoqKdPI2udYYMEna/GgpVWNnixzoJ7LCdK2Wqnha?=
 =?us-ascii?Q?CtzE/XYqXUh76OEGX9SQrP2nRnefq0/7H0JSRIMHL8FKxusm3mPE+1uZM4TT?=
 =?us-ascii?Q?YaGsbDdkt4ynFt/4i6d1iVEn91Kc4ur1rrm9+PfL3Am41KQ6E+i6d63jN3Mb?=
 =?us-ascii?Q?mK+9MX7d/AuBqgoqB5suO2EkzQW0r3Gw0nrZxj72DN0fUN9cPbH2RM4oEXt/?=
 =?us-ascii?Q?PjB6Rk6WtfewSa41EHIFCo0letvbFoQGR3No0AWyi/+xADKXMQNTtVXb14Qd?=
 =?us-ascii?Q?iQNpWzTVkmsTLlfw9BFpFXM/hlOuBnnWAqQeS7lSlvxmsQ86qByBisBWWGrB?=
 =?us-ascii?Q?I7fmiJNj2Hh0QN31cPV16F4m976NnqM+Uv/A0PFhIy6RGZLuLA1fdabEvqZe?=
 =?us-ascii?Q?Kqn9/4/UEWg8gH3s436QtRBu5+aHxriigpklDjd5g91BIod1GvXnMp16CEMM?=
 =?us-ascii?Q?NL9K9DIxcAINcC9XukLdXgTUDhtFfn2PjW4jS2jJlygfK04EcXDHy7lG5kRu?=
 =?us-ascii?Q?+Yc0AEz54UBmGXHW+Ekl9VZZGHlSVHmUz/LOlfuhGqBdgYPtzRw3Zr6w+sjm?=
 =?us-ascii?Q?6hDf4gXSFdHVMGOZVnBb3sSKIedXQaCmIRUA3N9eGW4vhiqRqn/i2V8NrIse?=
 =?us-ascii?Q?7L/JeZSysprGRiEEzYlh+/CHJwAFoOQetzS6+RjYoCl1MsFaRGiTc+FBDQTR?=
 =?us-ascii?Q?6kKEvovowQGrPzwc3APh9pK2Pak1K/9jNlkc0VUIrtRok+WONWpqwxapRyF7?=
 =?us-ascii?Q?gZmif0nYBZX3BUbOfT7n61+63iFpdw07MjFd2wtBpQutCA5FZT+sdtmfQBkM?=
 =?us-ascii?Q?4WhlEOljNhK8j1hLTubtilnG5IhE7aKi4/heNHelWCP60Vvvgs/7QC+WhHKW?=
 =?us-ascii?Q?5Q1qgr6kY1BzYeoxyaOQuMg6CjdPh4D8GRkmWZcC1ogSWuRnJ9YAiUcg9BDr?=
 =?us-ascii?Q?Pzx3/m8zzqZs5X2GLCoK+aEonm1KSNjhq9EuerAXWu5cLaXubAD5X4lyZQYk?=
 =?us-ascii?Q?k3VuPp/hjrT547qLloQfWOR497iq3aLgbBswr5Q7MgHvm3Da62Lp0fMatcEs?=
 =?us-ascii?Q?qeyeFVv0gQg7nA0sDWCOIYdpXM8muajeAzZFqfrSBAIlDAk5Mq2T2WlLelf6?=
 =?us-ascii?Q?GJUMozrSWe3sNyn52tZWl4QF1LAk0HBljOC43vNAEBE/quGZ3YFq5p/OX8ct?=
 =?us-ascii?Q?xz9l2ZHNcj3uabYGf7B807F1pwdl+l0iA9IDQRLkOhCXfqz9ClLfHOSTu6OI?=
 =?us-ascii?Q?d7BSv81zxrNR0P1+EJP+nSFV/SEY+6p6dmdNDAz3vd+UjZoaYDwQVLloOEWx?=
 =?us-ascii?Q?uuNYzdgU10wBHWQ5kHjbBGBSLnC9XqyCa1q3AQlk?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4b58f5af-d922-4d54-00ff-08dbf4d82123
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 14:48:51.2336
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SQ9R9k9YEolhksVFhps0Dyy8YfKmvbzjt5Nrx+TUefBtig2+Hc1DZGimYNqBAuzK
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB8559
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:49:07 -0800 (PST)

On Fri, Dec 01, 2023 at 02:12:28PM -0800, Nicolin Chen wrote:
> > Why is timeout linked to these two? Or rather, it doesn't have to be
> > linked like that. Any gerror is effectively synchronous because it
> > halts the queue and allows SW time to inspect which command failed and
> > record the gerror flags. So each and every command can get an error
> > indication.
> > 
> > Restarting the queue is done by putting sync in there to effectively
> > nop the failed command and we hope for the best and let it rip.
> 
> I see that SMMU driver only restarts the queue when dealing with
> CERROR_ILL. So only CERROR_ABT or CERROR_ATC_INV would result in
> -ETIMEOUT.

I'm not sure that is the best thing to do. ABT is basically the
machine caught fire, so sure there is no recovery for that.

But ATC_INV could be recovered and should ideally be canceled then
forwarded to the VM.
 
> > > As you remarked that we can't block the global CMDQ, so we have
> > > to let a real CERROR_ILL go. Yet, we can make sure commands to
> > > be fully sanitized before being issued, as we should immediately
> > > reject faulty commands anyway, for errors such as unsupported op
> > > codes, unzero-ed reserved fields, and unlinked vSIDs. This can
> > > at least largely reduce the probability of a real CERROR_ILL.
> > 
> > I'm more a little more concerend with ATC_INV as a malfunctioning
> > device can trigger this..
> 
> How about making sure that the invalidate handler always issues
> one CMD_ATC_INV at a time, so each arm_smmu_cmdq_issue_cmdlist()
> call has a chance to timeout? Then, we can simply know which one
> in the user array fails.

That sounds slow

> > > So, combining these two, we can still have a basic synchronous
> > > way by returning an errno to the invalidate ioctl? I see Kevin
> > > replied something similar too.
> > 
> > It isn't enough information, you don't know which gerror bits to set
> > and you don't know what cons index to stick to indicate the error
> > triggering command with just a simple errno.
> >
> > It does need to return a bunch of data to get it all right.
> 
> The array structure returns req_num to indicate the index. This
> works, even if the command consumption stops in the middle:
>  * @req_num: Input the number of cache invalidation requests in the array.
>  *           Output the number of requests successfully handled by kernel.
> 
> So we only need an error code of CERROR_ABT/ILL/ATC_INV.

Yes

> Or am I missing some point here?

It sounds Ok, we just have to understand what userspace should be
doing and how much of this the kernel should implement.

It seems to me that the error code should return the gerror and the
req_num should indicate the halted cons. The vmm should relay both
into the virtual registers.

Jason