Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 19 Jun 2023 09:37:05 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>
Subject: Re: [PATCH v2 00/11] iommufd: Add nesting infrastructure
Message-ID: <ZJBL8QLLBiwRsUSI@nvidia.com>
References: <20230511143844.22693-1-yi.l.liu@intel.com>
 <BN9PR11MB5276DAF0A11809CF8433EE338C7C9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZGdiS2m8jcd5OOt5@nvidia.com>
 <BN9PR11MB5276A74B2DA86C79908A420B8C419@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZH9AGWf1yRDu/86q@nvidia.com>
 <BN9PR11MB52763C7B838B04D3200322FD8C58A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52763C7B838B04D3200322FD8C58A@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JQVVaojLw80TopkgGkBHyAH7XPaIaTTED+A6OgjTz/TYgtQxIFDiTdfirfVv?=
 =?us-ascii?Q?N/gF/LjM2CSxi1F7BNe/cKwpehVwU6ZgxZ9VX5iVzSGLl0NN0YbzFPKyyI8u?=
 =?us-ascii?Q?d/6M1sGy6UJtMvZnYE3VykYqgSFd/dZWiB+rym1uLuHJU+rGrYLQIXZPXAyj?=
 =?us-ascii?Q?Ypp5Jn7yqLUhJijepPU98CUCuyWVvkthh9fzmEE7Xsh838Uj9u8TqT3L67D8?=
 =?us-ascii?Q?rVJkydq2ucUSovanf8OAoog1JlS+9EWtYTiCPiWeR8X66gCp3WqDad0cxPgD?=
 =?us-ascii?Q?7H6P/lMNSK5Rgui9s0tc2jTQexPrLy0H3AMNPoN2hlWP89bnEEt6crKZvFtW?=
 =?us-ascii?Q?vuugwdfPLg1iONth7bei0UeICIXTgszuz7exEka1KuhQb4TIMgd8o+JhYf7r?=
 =?us-ascii?Q?HqIxYmJJyrKGJnmySdk0SsmhnilwH0jjOAs3f2kLBYpSBTrCf58/ec7Sq7Tl?=
 =?us-ascii?Q?6iIv9N5ruzz+dN8evUD9jkRu5kqTZVhVNIAt4c3adEZEASuMthBDv2bD6FPd?=
 =?us-ascii?Q?wiFQjEJIJiT9fYT0Us4qGmsFTTq4kmUOE/MVgw2WgpVLO/6DIgqdK9qoxNNu?=
 =?us-ascii?Q?s37tOSz8s6126ngjnizEcNTGNP3dERr1rhOdOUrIEkFRCjU+u7EDlo3fkpGF?=
 =?us-ascii?Q?oh1eE4+HAMbHHKFDh5y7xLBVvIkjisu4DaRPJcEiLBcp1lDDfow5Klm9k5Ru?=
 =?us-ascii?Q?aHIQQc84degoEVcxUVoDmh/5u338S4C7EuHI9V05boiys+bsBoxHtpblx/FU?=
 =?us-ascii?Q?03Xd8bwGxjxpL5ptSn2SbyNJ+lSIoRdOMO98St5F9ThJeiDpBjgOkaYhChGa?=
 =?us-ascii?Q?avd/SZhPgtbibWBcBb055UewR3vT5UTCXted6Ob37TqZpMJ2SqcJuRhFTpWg?=
 =?us-ascii?Q?T4hAs6Y3Us01CoGCQaK5ezMq1I2y5VGmCJwwVGRJXsfA+Hyg6ZEmC8F5nM5f?=
 =?us-ascii?Q?1+NuWcP3CPMZgGn5EIRWol1UhFUIrzBy4bff0GBHGHjUX5nhee4YVAce2Jly?=
 =?us-ascii?Q?FxWdFN6EtJPlSYv8+2l+aKYtfT0cYxSTB4UpGea8dWEmaM7jehUQ78yZxWGV?=
 =?us-ascii?Q?BtZOVwvQLVZLRl9eJu1zjDFMHEKHebW3FSJjW0XoM8kaMATAT75qfUqeX7KN?=
 =?us-ascii?Q?5UqPNeSED/CUFwOOPunEumNXqlktEGNbupkBg3FAoZJd7Q+y0ae73W8xJjdt?=
 =?us-ascii?Q?WwYydYWbR5ebge8fHL3MJeE5WAdO6iRY1hTxkpJ3Mz7IOdJX9v+xDwwf6JlM?=
 =?us-ascii?Q?E+lgGQL9tRhjEky4pFvDTwWVCSZqIalrDcnrGQBvnmiPzq2qbov2vf0iL+5U?=
 =?us-ascii?Q?aCHzAB3I/UMC6vIO2g5++JCp7jX/C4Lm2d8eyG17rhaVVPnUUn8/p45EF+0l?=
 =?us-ascii?Q?6prHze0rhZLJsy0di22m7CGlG8v0DlJWDJ9bD+aUwuXSjYs4Edb7Mrzg18LQ?=
 =?us-ascii?Q?sgxglF9tXavNYMQZg/wNLzupt+lQvA8OOe4HQune5fLjuZyTxbQHCPySgb/2?=
 =?us-ascii?Q?rd7m1XTn5psghhrDI9JmDPPH7Jkxn+paautJ0gK5qMB+v3jih9KuOc7rp3FA?=
 =?us-ascii?Q?AmXgf7afxT9F/adCOrk=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7d43756f-656e-471f-d57a-08db70c1e4b6
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jun 2023 12:37:07.2616
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LoB2dtMKg8jljQYMvE7ro/qmlPatHE7ybzT1rhJ01XQiCJ+ZLtNr97VjoDSEz4cS
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8833
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jun 16, 2023 at 02:43:13AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Tuesday, June 6, 2023 10:18 PM
> > 
> > > > > It's not necessarily to add reserved regions to the IOAS of the parent
> > > > > hwpt since the device doesn't access that address space after it's
> > > > > attached to stage-1. The parent is used only for address translation
> > > > > in the iommu side.
> > > >
> > > > But if we don't put them in the IOAS of the parent there is no way for
> > > > userspace to learn what they are to forward to the VM ?
> > >
> > > emmm I wonder whether that is the right interface to report
> > > per-device reserved regions.
> > 
> > The iommu driver needs to report different reserved regions for the S1
> > and S2 iommu_domains, 
> 
> I can see the difference between RID and RID+PASID, but not sure whether
> it's a actual requirement regarding to attached domain.

No, it isn't RID or RID+PASID here

The S2 has a different set of reserved regsions than the S1 because
the S2's IOVA does not appear on the bus.

So the S2's reserved regions are entirely an artifact of how the IOMMU
HW itself works when nesting.

We can probably get by with some documented slightly messy rules that
the reserved_regions only applies to directly RID attached domains. S2
and PASID attachments always have no reserved spaces.

> When talking about RID-based nesting alone, ARM needs to add reserved
> regions to the parent IOAS as identity is a valid S1 mode in nesting.

No, definately not. The S2 has no reserved regions because it is an
internal IOVA, and we should not abuse that.

Reflecting the requirements for an identity map is something all iommu
HW needs to handle, we should figure out how to do that properly.

> But for Intel RID nesting excludes identity (which becomes a direct
> attach to S2) so the reserved regions apply to S1 instead of the parent IOAS.

IIRC all the HW models will assign their S2's as a RID attached "S1"
during boot time to emulate "no translation"?

They all need to learn what the allowed identiy mapping is so that the
VMM can construct a compatible guest address space, independently of
any IOAS restrictions.

Jason