Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 23 Sep 2021 10:30:37 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Message-ID: <20210923133037.GP964074@nvidia.com>
References: <20210921174438.GW327412@nvidia.com>
 <BN9PR11MB543362CEBDAD02DA9F06D8ED8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210922140911.GT327412@nvidia.com>
 <BN9PR11MB5433A47FFA0A8C51643AA33C8CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923120653.GK964074@nvidia.com>
 <BN9PR11MB543309C4D55D628278B95E008CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923123118.GN964074@nvidia.com>
 <BN9PR11MB5433F297E3FA20DDC05020E18CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923130135.GO964074@nvidia.com>
 <BN9PR11MB5433B300CF82CB09326A00978CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5433B300CF82CB09326A00978CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wCkCUca3sLbUcHdzR7oBaowf3WmrqfaZbUXOJiG9ncoWpMAZxDEbCtKT4trG?=
 =?us-ascii?Q?UviRYRdTytvqt8M98qvsxjtoS+nbOkgtwZWjIdbFyJn3FZ2/dm3UdZdlVGyH?=
 =?us-ascii?Q?v4lXIpmjseixGPjob4KU+RODHwdYAHmEP4LsNoVbBs+WBZgGnG01zc157x8j?=
 =?us-ascii?Q?tIksbZgppQAJzdTaVU9DS+0Ktd6uJkZxsVcv1wKLcA1NKhmtWKvM84CJ270N?=
 =?us-ascii?Q?4HIssX2kwhtAwOrrksJUT5EKEnC1RG8LtMnUgQae6VTYsP1uXJy3NvOIlYi3?=
 =?us-ascii?Q?+CYc+byTVQjmhp68Hw1UzqYZFnpLO1vIO6ibfodMZr7jSOkA+brU3hzeOCFH?=
 =?us-ascii?Q?uJ5MBWGnh16Ypwmz2zwknyjZwDxD9DUJ+O6kgg4x7NyQRrSL1acrz80PgJ3q?=
 =?us-ascii?Q?MAH2qRb6wc1DPi3ZvTAWxrJaepa7/hVuz7v2NxmEnOsJsjMltvYN0YPjYWmk?=
 =?us-ascii?Q?iY8SBU+a2NVCvYvWeQCs6OZ+ZmlLtoUh4tNDVlwUQcnieRt7stP7GkyBn5bs?=
 =?us-ascii?Q?uYpepKcrQjtDjVdKunTQo7vW5EKM6XC9menlF/aqeJdTTWmYij3ld2aIQXeW?=
 =?us-ascii?Q?a5KpJ/fL4tkeSKfAgUhXzHVeRmTB++Tm0xWmYDpcskibk4n+/SPNNqXCXuui?=
 =?us-ascii?Q?TgkGHxV69a4TZGvPduBAG9h8QgcvWpzGBb7N0BvcpWWEr92leSkHe6ppqt51?=
 =?us-ascii?Q?evF0c+cl5mDwWLsn6ybSWgC4vx/otg4mCQp6BTdL8uHdE/3lDNfh6dgWvx/d?=
 =?us-ascii?Q?1+HATaTPdVjHSGzR+S1FXfaOs75avUGvY6q9uH+RgDFpg/ewUYY/+/KU7fg0?=
 =?us-ascii?Q?/BDVcNjrjBe+SosQbA47DQ07S/xOApG/qkYS9Fqh0kDf9cEKsRf1L+38ND8/?=
 =?us-ascii?Q?ESUputlipeBxT0BKp76fuTbqOgGCxJRnqhepqoz+aOUzoVuFtI2K+MgHVIQp?=
 =?us-ascii?Q?HBXZ3VpYl6CAB1cruDNM7385IkeFKYXPW6zmbLTWH7aRm1IThLEnW0DS8Opn?=
 =?us-ascii?Q?kRxwuTi2z1CkbzjjEjjP2ttpPhejkkNprAKszzC77RiR21xAE94uwFk6Gib2?=
 =?us-ascii?Q?7RR9Pkh9mg6poJnE/1ZeG2r9qHahBtJgEHluoOPBVggjMyRrww9pYqF3VXWy?=
 =?us-ascii?Q?n3SecFJnGLNk/m4TWeiAFhkZuU9qDtQyTapNdQV0xclHpUupO5Eb9vGmdI+V?=
 =?us-ascii?Q?l3fQVYUhxKL4+B3LxAfLiUZV1a7bo0x1h8107sNcIE9+Y3Ydk4WidcigYGMk?=
 =?us-ascii?Q?nkw6T5v9vg1XQlnbMoA7lcWkUtLvQActKROJwzWP4LMtVnpjnpxfQ2BWPrxa?=
 =?us-ascii?Q?fphgbGvmAiV08uC4Gu4xVOvd?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: aa65b4e5-74cf-48bc-1b39-08d97e96553f
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2021 13:30:39.2080
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wQbnYpZx0OXREUaESwRWAmc3ky3M4OYEi9awp6oJnwBiBLIJXYU10YXksJuC8evi
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5046
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 23, 2021 at 01:20:55PM +0000, Tian, Kevin wrote:

> > > this is not a flow for mdev. It's also required for pdev on Intel platform,
> > > because the pasid table is in HPA space thus must be managed by host
> > > kernel. Even no translation we still need the user to provide the pasid info.
> > 
> > There should be no mandatory vPASID stuff in most of these flows, that
> > is just a special thing ENQCMD virtualization needs. If userspace
> > isn't doing ENQCMD virtualization it shouldn't need to touch this
> > stuff.
> 
> No. for one, we also support SVA w/o using ENQCMD. For two, the key
> is that the PASID table cannot be delegated to the userspace like ARM
> or AMD. This implies that for any pasid that the userspace wants to
> enable, it must be configured via the kernel.

Yes, configured through the kernel, but the simplified flow should
have the kernel handle everything and just emit a PASID for userspace
to use.


> just for a short summary of PASID model from previous design RFC:
> 
> for arm/amd:
> 	- pasid space delegated to userspace
> 	- pasid table delegated to userspace
> 	- just one call to bind pasid_table() then pasids are fully managed by user
> 
> for intel:
> 	- pasid table is always managed by kernel
> 	- for pdev,
> 		- pasid space is delegated to userspace
> 		- attach_ioasid(dev, ioasid, pasid) so the kernel can setup the pasid entry
> 	- for mdev,
> 		- pasid space is managed by userspace
> 		- attach_ioasid(dev, ioasid, vpasid). vfio converts vpasid to ppasid. iommufd setups the ppasid entry
> 		- additional a contract to kvm for setup CPU pasid translation if enqcmd is used
> 	- to unify pdev/mdev, just always call it vpasid in attach_ioasid(). let underlying driver to figure out whether vpasid should be translated.

All cases should support a kernel owned ioas associated with a
PASID. This is the universal basic API that all PASID supporting
IOMMUs need to implement.

I should not need to write generic users space that has to know how to
setup architecture specific nested userspace page tables just to use
PASID!

All of the above is qemu accelerated vIOMMU stuff. It is a good idea
to keep the two areas seperate as it greatly informs what is general
code and what is HW specific code.

Jason