Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Date:   Mon, 11 Dec 2023 09:20:41 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Yi Liu <yi.l.liu@intel.com>
Cc:     "Giani, Dhaval" <Dhaval.Giani@amd.com>,
        Vasant Hegde <vasant.hegde@amd.com>,
        Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        joro@8bytes.org, alex.williamson@redhat.com, kevin.tian@intel.com,
        robin.murphy@arm.com, baolu.lu@linux.intel.com, cohuck@redhat.com,
        eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org,
        mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com,
        yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com,
        shameerali.kolothum.thodi@huawei.com, lulu@redhat.com,
        iommu@lists.linux.dev, linux-kernel@vger.kernel.org,
        linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com,
        joao.m.martins@oracle.com, xin.zeng@intel.com, yan.y.zhao@intel.com
Subject: Re: [PATCH v6 0/6] iommufd: Add nesting infrastructure (part 2/2)
Message-ID: <20231211132041.GE2944114@nvidia.com>
References: <20231117130717.19875-1-yi.l.liu@intel.com>
 <20231209014726.GA2945299@nvidia.com>
 <77ac47d0-2ef0-41fa-86c2-091358541465@intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <77ac47d0-2ef0-41fa-86c2-091358541465@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?0BcBTaelb7kFCycCipzLQzRcElUN91/lSyFFkZ+DBZyqcT77dNUkbEbGa9ws?=
 =?us-ascii?Q?Zb7LQsJ00WuEMc266xArwI/o7+xY3v4bf/zTXYAbMsU8Zs1YS76SmoXOHqf8?=
 =?us-ascii?Q?DDIdX9ea3HHjYzYrFrizmXhxJm8Ooor/2+Mxnulx9dMoflhlI5JJKsJzWGSC?=
 =?us-ascii?Q?nLWbQO6V1pPdN/kRk/Gbea+uEmpCXg1tKz5lxlWZodTMx7tsZIhrkwsMRn5P?=
 =?us-ascii?Q?L6VmxG42CCy/lKnos5GxcFhPr7xEx4V1E/mctIfDsrl5JgmbK1iHsYbSzy1W?=
 =?us-ascii?Q?5Cv8/ciSl6Q5KIlUfbBFVzxuHqDmEnmd2RRz0K4LC3rTY7Fzki+imMFp6VtS?=
 =?us-ascii?Q?m4U3PonW5FH57YBw3uOrEgr8cEW7kR73FBHsQtq1SbQUlO1l3LBkYPu8RLcA?=
 =?us-ascii?Q?KeoZZPTAv1C2ohwLxTJBRjURIll68MDfwINvQXdV3cb8K5rMX+l3jMgde+Y6?=
 =?us-ascii?Q?GUjFTJs51GGTRU1V6CIFXBkFB2BJu/MOKHLZMWdp1UmNAhWCJNGY0IoXCn59?=
 =?us-ascii?Q?8OAcLnozEsS3O5J9nmvAWUvNKfaG3iOLXmOi57TF8lwebFUpkmcT/1w2HHeq?=
 =?us-ascii?Q?Mpsk6JzD6MTV8HKuSashM9PFxI8Ob2xL0pFAhkYjv9be2LgvuC9BioeADMLM?=
 =?us-ascii?Q?qSwVc1H6o5U1TDTvqbWn792Z2TX7CErJBwc3ZmtDNGyhE0C+dPTmS5gF2Tzi?=
 =?us-ascii?Q?4XekRBpXSXkLBgSWww0grG+4GyNpNCD7DgUXpM2FxtHZF4vXvl8DTXE+5Xb7?=
 =?us-ascii?Q?acj9mGzcXZgU+6PSBbqPjuiEv5zNhy026jdAj/ifdwgX+ApBVOkQTd9e7HtK?=
 =?us-ascii?Q?OFP45Dd7DkHGBxjGhNajdsS5dNaPHS80x8At90WwrKnQwHlVsh40JPxCCe5Y?=
 =?us-ascii?Q?cQWDqQ5YKCXo1QfestCqaq5egdUYMQCkvT2fKYo/UNJsbXRzkbQu14dLHuVC?=
 =?us-ascii?Q?UN0Y5VBFKCyeBpDxqcp4Y05zA0d1QPdbAJYqQBNNetyzQeCPB32xpnH5uGdb?=
 =?us-ascii?Q?UBUbT88WOA20L4p0kEYjS1LTIgRpdg04um5Ah46kNP6xNKkM1Npx3cCork10?=
 =?us-ascii?Q?heos1uI6h73PhzM65c7zvwsW55WqpFiaqhpgDADF5kb2I0I+CzvX/6Pxk+06?=
 =?us-ascii?Q?WHuXjBscm20l9ji7H55VyxSmNZs71zImrtTSUgFcHmGNq9XD3i8OBNYF/jtW?=
 =?us-ascii?Q?tsZsYIIUbljry2rL9M5kNtVsvUnJFWHXEUewMwqs/CgbIAO/BsduzLF+IYsh?=
 =?us-ascii?Q?XsMY9CIuWllZpSDJMlORIBTUb7B14jaILhwZ96Sc0TWVg+Z53S4c9RsO+ZMy?=
 =?us-ascii?Q?2gZNNPPtsqkopqwdi3fm0dsXawJZtn6wwd80SEEDeP+VNDKQNdq/ywAnETxe?=
 =?us-ascii?Q?sMLm3t9g/Cxu7ciOvEHitncg9sOzzKsjDXNZVoajvGTdlPkoZm/4ny/7LviX?=
 =?us-ascii?Q?nl69g9y2n6ycw8jlY70o2XSWXb4Y963UpgJ5RHxFdq6I+v+sSJ/7Kbw4vQNJ?=
 =?us-ascii?Q?abXkH/vtM9MRc6J7KYopva7fmu8PRP1T8luIwPF9uSVvH+g70b0XELW031R8?=
 =?us-ascii?Q?GpYqJxffyBibHfg4tTR7hph34K2dC7cik0TyUQ0G?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d2d80785-df60-43fe-fd54-08dbfa4bfa07
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Dec 2023 13:20:42.8520
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6USC606W1KUQG6muXesMZiz0WErARjrBK6U2Iy+mpyRkoxXXe0ke2lv1GxnJ9n9N
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4178
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 05:21:13 -0800 (PST)

On Mon, Dec 11, 2023 at 08:35:09PM +0800, Yi Liu wrote:

> > What iommufd object should receive the IOTLB invalidation command list:
> >   Intel: The Nesting domain. The command list has to be broken up per
> >          (vDomain-ID,PASID) and that batch delivered to the single
> > 	nesting domain. Kernel ignores vDomain-ID/PASID and just
> > 	invalidates whatever the nesting domain is actually attached to
> 
> this is what we are doing in current series[1]. is it?

I think so

> > So.. In short.. Invalidation is a PITA. The idea is the same but
> > annoying little details interfere with actually having a compltely
> > common API here. IMHO the uAPI in this series is fine. It will support
> > Intel invalidation and non-ATC invalidation on AMD/ARM. It should be
> > setup to allow that the target domain object can be any HWPT.
> 
> This HWPT is still nested domain. Is it? But it can represent a guest I/O
> page table (VT-d), guest CD table (ARM), guest CR3 Table (AMD, it seems to
> be a set of guest CR3 table pointers). May ARM and AMD guys keep me honest
> here.

I was thinking ARM would not want to use a nested domain because
really the invalidation is global to the entire nesting parent.

But, there is an issue with that - the nesting parent could be
attached to multiple iommu instances but we only want to invalidate a
single instance. 

However, specifying the nesting domain instead, and restricting the
nesting domain to be single-instance would provide the kernel enough
information without providing weird new APIs. So maybe it is best to
just make everything use the nesting domain even if it is somewhat
semantically weird on arm.

> The Intel guest I/O page table case may be the simplest as userspace only
> needs to provide the HWPT ID and the affected ranges for invalidating. As
> mentioned above, kernel will find out the attached device/pasid and
> invalidating cache with the device/pasid. For ARM and AMD case, extra
> information is needed. Am I getting you correct?

Yes
 
> > Thus next steps:
> >   - Respin this and lets focus on Intel only (this will be tough for
> >     the holidays, but if it is available I will try)
> 
> I've respinned the iommufd cache invalidation part with the change to
> report error_code/error_data per invalidation entry. yet still busy on
> making Intel VTd part to report the error_code. Besides, I didn't see
> other respin needed for Intel VT-d invalidation. If I missed thing, please
> do let me know.:)

Lets see

Jason