Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 6 Jun 2023 11:18:01 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>
Subject: Re: [PATCH v2 00/11] iommufd: Add nesting infrastructure
Message-ID: <ZH9AGWf1yRDu/86q@nvidia.com>
References: <20230511143844.22693-1-yi.l.liu@intel.com>
 <BN9PR11MB5276DAF0A11809CF8433EE338C7C9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZGdiS2m8jcd5OOt5@nvidia.com>
 <BN9PR11MB5276A74B2DA86C79908A420B8C419@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276A74B2DA86C79908A420B8C419@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?X+slWf+MhBLuEQWZP2enae4XI6kFSCmqwukYVhl+cQeZ+OyF8LLcyhopmqvR?=
 =?us-ascii?Q?blYO0LeUC8Sr/ez/rgd1yH4EhilVOtAUqQvZKthWf00PUaeSVysYD0mejio2?=
 =?us-ascii?Q?ktsWOoVyul7FwK3AZHb4leAfBMoP0z5mtJQqHY0Cou1zwCzMlTfkuAZFlejw?=
 =?us-ascii?Q?vjXGf9ipsrFJqSPFxZ1Tki315NRftdkdozYf3JcD1QeDJf0U7DGZzFdoyYwy?=
 =?us-ascii?Q?YvX97HXFyNGHjXzUo6VIe7LE5bCenNVDGaSfqu5KlWQnfulHhacaQGAloo9t?=
 =?us-ascii?Q?sWZAwD7o28w1iP3DygtlM4J03FZZnbh++75aG1+xJnFHDVu9nErB0wWrovwo?=
 =?us-ascii?Q?u5t0S/dn79SVaeNnZLxOH7XWAhKsa7L+zePY7Mnl0IY2j3v2dlXWiNgulyQ2?=
 =?us-ascii?Q?tEiw89i9uqTKeUS1z+MPTLaBMs+YWXL4TvhKtjf5wPpMXNuLIlYzDhzIVYqN?=
 =?us-ascii?Q?WmC6IIIuq8Jy9QrgLqs8hJASkR/qU9uSVS1GEMLCvP5B2vAXEPXaOmL5U+mA?=
 =?us-ascii?Q?uCv1xjsdGIirN3HmIZBKnaBly3PkzqPnM0X4jkcgru4H1b+59SyiIGYvBEJL?=
 =?us-ascii?Q?0fp/Ap/hGLzAWddU0GoMipX5nw221HwbaRxazk+wAgb57gJX9W/QnKJfqDjW?=
 =?us-ascii?Q?wbfXQBZIBeD3yOGUwep1frxREwqLDKz4IVdpEqhn4ZSHPxeVklXprPg0EfXN?=
 =?us-ascii?Q?xgo9lvTYJWTby+9I+rTBSzJTcbtbLJjGnUPUQc0fFSoU8ObmohPkk1yEhMf1?=
 =?us-ascii?Q?o51TE2hnWvM21GGP0b1TFwKFN/ImdRuAiYAY4v0LO4eg0qKKP9xks6LBERG8?=
 =?us-ascii?Q?zuGrMl4iMJvEOvJpCsdeYP860XFpa7pVkSFp6VcMey4+DGjjNSYsp2wt1g21?=
 =?us-ascii?Q?8WZMNUSgnjM4nDEy6smqT706Sekr/iC9tbaCZrPdbDsESxZ/HDxmRX+qGvGN?=
 =?us-ascii?Q?wcF6qb3dms2cQJhKxCUYe5Wba81PWRNlyXiTlJP7xXeq7Hwh+m7KC162ilFU?=
 =?us-ascii?Q?rUROWyDZ2covFvw8RTX5IXiJDrYGFsHUwxhj6yO84+t20pXlSJ3aIbtFruu1?=
 =?us-ascii?Q?78xfHLtPgQwoRdmKZmKTDUtUUX/z5NlPeDiC/tQaj94OV6uFVssQd9DPCm76?=
 =?us-ascii?Q?vesywcwOIQNYhO5Go1LSYUCJtrDmXtX3CiluXCI0HH3OnfFyEbxD6Ud0R/b6?=
 =?us-ascii?Q?fk2APQF6U9UKQThSR+FDFlsQlU8DnATmgSBRFop011GQMoQ4exRL7fkhngl/?=
 =?us-ascii?Q?ulELCuwoWXZ30Ik4qM4PaOWtX6nUrtKrHeDUsVWvI3Bt3rSUCnlKV30G8XqT?=
 =?us-ascii?Q?vRgXZPJ0oc138QgH/HptU/p8ZTBpwdSEagG9ZuvAP5oy8/Wx5Mlw2//ZcUhY?=
 =?us-ascii?Q?GOq7hygy/Ju912XphP9VNUcnkMKMFg12/8nUcOWDQYt7jcNO2KFrMyxbnC3h?=
 =?us-ascii?Q?c5rK9gG7kiTPAYob70vrWb1Nt1OugKA9CyuQbdX/vd7Md3IrgpQKfUsrM3KY?=
 =?us-ascii?Q?LNuGEYU/xUwbuTaEAWqukko+4Bw1H7/F6WWrwElGkTl/TO/+dBABJ3+RwZbb?=
 =?us-ascii?Q?IuW8YC+muf+kmdfrwSlXeWXj1MImCVeoZUPWF5q0?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 745bd347-d31e-442d-c45b-08db6698d69c
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2023 14:18:02.6605
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VKoVnrAZFsrZEBNrgBuVXl5LlbDDIXozVtp1phsvCy4P5Oq9+o5XAKCFBeqZvcoM
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB9161
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, May 24, 2023 at 03:48:43AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Friday, May 19, 2023 7:50 PM
> > 
> > On Fri, May 19, 2023 at 09:56:04AM +0000, Tian, Kevin wrote:
> > > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > > Sent: Thursday, May 11, 2023 10:39 PM
> > > >
> > > > Lu Baolu (2):
> > > >   iommu: Add new iommu op to create domains owned by userspace
> > > >   iommu: Add nested domain support
> > > >
> > > > Nicolin Chen (5):
> > > >   iommufd/hw_pagetable: Do not populate user-managed hw_pagetables
> > > >   iommufd/selftest: Add domain_alloc_user() support in iommu mock
> > > >   iommufd/selftest: Add coverage for IOMMU_HWPT_ALLOC with user
> > data
> > > >   iommufd/selftest: Add IOMMU_TEST_OP_MD_CHECK_IOTLB test op
> > > >   iommufd/selftest: Add coverage for IOMMU_HWPT_INVALIDATE ioctl
> > > >
> > > > Yi Liu (4):
> > > >   iommufd/hw_pagetable: Use domain_alloc_user op for domain
> > allocation
> > > >   iommufd: Pass parent hwpt and user_data to
> > > >     iommufd_hw_pagetable_alloc()
> > > >   iommufd: IOMMU_HWPT_ALLOC allocation with user data
> > > >   iommufd: Add IOMMU_HWPT_INVALIDATE
> > > >
> > >
> > > I didn't see any change in iommufd_hw_pagetable_attach() to handle
> > > stage-1 hwpt differently.
> > >
> > > In concept whatever reserved regions existing on a device should be
> > > directly reflected on the hwpt which the device is attached to.
> > >
> > > So with nesting presumably the reserved regions of the device have
> > > been reported to the userspace and it's user's responsibility to avoid
> > > allocating IOVA from those reserved regions in stage-1 hwpt.
> > 
> > Presumably
> > 
> > > It's not necessarily to add reserved regions to the IOAS of the parent
> > > hwpt since the device doesn't access that address space after it's
> > > attached to stage-1. The parent is used only for address translation
> > > in the iommu side.
> > 
> > But if we don't put them in the IOAS of the parent there is no way for
> > userspace to learn what they are to forward to the VM ?
> 
> emmm I wonder whether that is the right interface to report
> per-device reserved regions.

The iommu driver needs to report different reserved regions for the S1
and S2 iommu_domains, and the IOAS should only get the reserved
regions for the S2.

Currently the API has no way to report per-domain reserved regions and
that is possibly OK for now. The S2 really doesn't have reserved
regions beyond the domain aperture.

So an ioctl to directly query the reserved regions for a dev_id makes
sense.

> > Since we expect the parent IOAS to be usable in an identity mode I
> > think they should be added, at least I can't see a reason not to add
> > them.
> 
> this is a good point.

But it mixes things

The S2 doesn't have reserved ranges restrictions, we always have some
model of a S1, even for identity mode, that would carry the reserved
ranges.

> With that it makes more sense to make it a vendor specific choice.

It isn't vendor specific, the ranges come from the domain that is
attached to the IOAS, and we simply don't import ranges for a S2
domain.

Jason