Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>
Subject: RE: [PATCH v2 00/11] iommufd: Add nesting infrastructure
Thread-Topic: [PATCH v2 00/11] iommufd: Add nesting infrastructure
Thread-Index: AQHZhBZTNX7dDuLMFEel2CdlG13MGq9hZYMwgAAiQoCAB0Y6EA==
Date:   Wed, 24 May 2023 03:48:43 +0000
Message-ID: <BN9PR11MB5276A74B2DA86C79908A420B8C419@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230511143844.22693-1-yi.l.liu@intel.com>
 <BN9PR11MB5276DAF0A11809CF8433EE338C7C9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZGdiS2m8jcd5OOt5@nvidia.com>
In-Reply-To: <ZGdiS2m8jcd5OOt5@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?IFsHWdXuuGR/f0zinyV2U8c62mwk8UrnEZXHWxkQS20KI5hEdg9vTqhos0kH?=
 =?us-ascii?Q?R65ESAs7y8Retg//6gE881ORPc+st9xl4JO6C/1wRoWm2RuWojhcnHpoi3nq?=
 =?us-ascii?Q?SWupqn6fFcedh2D6GAH2IixSIXmY+6E+DnOIQcV7eZwtsGj5o057VJdjxZn6?=
 =?us-ascii?Q?2YM2N1aFpYQvD8bY8rLB8eYkb+tOv9GkGd0TZOVV4BIaEH2fWEGYMfZcHkEA?=
 =?us-ascii?Q?Vu5fLk0oCSpt9vKQ2hTBvuRox5ZLa1yXse7ONbf+HyA69gzTko0Wqyko0Wlm?=
 =?us-ascii?Q?fe5fZ3YOVYVlbh9jN+nO3LA7kKfapyyhDi+gwGJ882U8qHkMYnH/NXz9fQor?=
 =?us-ascii?Q?O3BGnjdIhaHSvN8XqeKeOXVu/eycm75VMRQg65iM3yHK/cCfMRuuLRNpFD9o?=
 =?us-ascii?Q?qnaJqejKHgulDKXH52/UINTs8BiAWB+BTQHalN/Trf/yhEFmJN3SOA1nGgxn?=
 =?us-ascii?Q?5w35lMv1rpHn036QzMgB0Z/Z63kxdVTidEfK3RmKACzSokn9mIjcqNC9bUYh?=
 =?us-ascii?Q?oBzuDr3dC9KcVeiw8gw7AbT14+O3ENF03KzZPGnzz4tvxea89E9FnptNb5So?=
 =?us-ascii?Q?ybO9hsnwL/FsrjyUFCaJQ8puyT/WR+L37Tv0HtRWOMRgdpuJe469Ye8R9dei?=
 =?us-ascii?Q?PpWt7VjPAyNgUGuaacu89qbJ6fLIogXSnkLKuXi7Fl8OMcdzHElaMYJugLyP?=
 =?us-ascii?Q?5YDyWHyRcCAtmzs8B6ROSzIRdBUwvzlBaCKgFY6HTddUHeZbLUQZ7/2GMudY?=
 =?us-ascii?Q?g3a1d2x3u4pbbdltsxwASpY4i16b4z2Uhj3H+Lp/Vjpa1OSsn2R0pi6h7wGh?=
 =?us-ascii?Q?1AC8F4pAWpf10Y2mPcNkVcQTdqk10EHDC1Q6ZCYh/l6jAauBcE3RX16pLGQR?=
 =?us-ascii?Q?Wo8sawlMm/J9w6ZUWxOo4OGfo8uB+p0ynPDZG3z2rzg7q/oMT9o6wDTH+Kdi?=
 =?us-ascii?Q?z+ELf0QwA2moaZoE0CfEHmAGOMU/mivqBNHnlf8Pvc1YJUWsjvBo6UrYFAhE?=
 =?us-ascii?Q?sFVs17o1hWtioR9IxRU20xJYrxtcrscoG4n3NMMVcfN9mFjyiOL+DcP3uaWG?=
 =?us-ascii?Q?qbaSoYrDhpkOLQWbj+oLgd221NUsQJx9WJfdoyepnPRyVFKarhYEtm/jPdVC?=
 =?us-ascii?Q?KpOA5mXwC+azauxwIvEUih7qOHfInm0APDCXI3GIzaug++ccNf7Q1FV8BhJ9?=
 =?us-ascii?Q?J4rEqDxFQGqTEWaYP57dVpOtsSK5RQKO0Unmk6MXxoPTvX0SLvr5Z3UVQtEF?=
 =?us-ascii?Q?08UYTbpF6Mx7RXnf19k69kl8Ziicr+l1oLmlCuAy1gTMr71m0TKH+8To1+Ms?=
 =?us-ascii?Q?h7y/T2ktNhD6azeTSiTbQ36BYbRx/pOC8Y+D3URqsr4AeCZ9T75cq+AqHaqI?=
 =?us-ascii?Q?2VuH0yzH47ujkx1mr3mgQ1QFrJTZg6GDAhZinXTkCf55F+ir6GQxTEeCWNiG?=
 =?us-ascii?Q?ZZjaN8DhTYy0TyU9gi2kUWsLrkryLRXW+n8UPUpGz90IzICi2a+Ipk1C1InK?=
 =?us-ascii?Q?D8Hk2lBONAL0apSi6llscX/6Q747gmYwGvAe0D6q4Nzzwy2Dit75clpXjGlV?=
 =?us-ascii?Q?quPMIbAiGgnj3RIMJNaUjqrmWboRTPI9b7qB12rg?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 540ada17-a46a-40ab-d077-08db5c09c54d
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 May 2023 03:48:43.7184
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: zpeHv13YwXIxkIO26Qno7IjzvXDPR0j1C+uqb7BnRFBq+LQUGtjxHwAddyGe63MPWK1eKBnYmMcc+J8jRThT0Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR11MB8488
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, May 19, 2023 7:50 PM
>=20
> On Fri, May 19, 2023 at 09:56:04AM +0000, Tian, Kevin wrote:
> > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > Sent: Thursday, May 11, 2023 10:39 PM
> > >
> > > Lu Baolu (2):
> > >   iommu: Add new iommu op to create domains owned by userspace
> > >   iommu: Add nested domain support
> > >
> > > Nicolin Chen (5):
> > >   iommufd/hw_pagetable: Do not populate user-managed hw_pagetables
> > >   iommufd/selftest: Add domain_alloc_user() support in iommu mock
> > >   iommufd/selftest: Add coverage for IOMMU_HWPT_ALLOC with user
> data
> > >   iommufd/selftest: Add IOMMU_TEST_OP_MD_CHECK_IOTLB test op
> > >   iommufd/selftest: Add coverage for IOMMU_HWPT_INVALIDATE ioctl
> > >
> > > Yi Liu (4):
> > >   iommufd/hw_pagetable: Use domain_alloc_user op for domain
> allocation
> > >   iommufd: Pass parent hwpt and user_data to
> > >     iommufd_hw_pagetable_alloc()
> > >   iommufd: IOMMU_HWPT_ALLOC allocation with user data
> > >   iommufd: Add IOMMU_HWPT_INVALIDATE
> > >
> >
> > I didn't see any change in iommufd_hw_pagetable_attach() to handle
> > stage-1 hwpt differently.
> >
> > In concept whatever reserved regions existing on a device should be
> > directly reflected on the hwpt which the device is attached to.
> >
> > So with nesting presumably the reserved regions of the device have
> > been reported to the userspace and it's user's responsibility to avoid
> > allocating IOVA from those reserved regions in stage-1 hwpt.
>=20
> Presumably
>=20
> > It's not necessarily to add reserved regions to the IOAS of the parent
> > hwpt since the device doesn't access that address space after it's
> > attached to stage-1. The parent is used only for address translation
> > in the iommu side.
>=20
> But if we don't put them in the IOAS of the parent there is no way for
> userspace to learn what they are to forward to the VM ?

emmm I wonder whether that is the right interface to report
per-device reserved regions.

e.g. does it imply that all devices will be reported to the guest with
the exact same set of reserved regions merged in the parent IOAS?

it works but looks unclear in concept. By definition the list of
reserved regions on a device should be static/fixed instead of
being dynamic upon which IOAS this device is attached to and
how many other devices are sharing the same IOAS...

IOAS_IOVA_RANGES kind of follows what vfio type1 provides today

IMHO probably we should have DEVICE_IOVA_RANGES in the first
place instead of doing it via IOAS_IOVA_RANGES which is then
described as being dynamic upon the list of currently attached devices.

>=20
> Since we expect the parent IOAS to be usable in an identity mode I
> think they should be added, at least I can't see a reason not to add
> them.

this is a good point.

for SMMU this sounds a must-have as identity mode is configured
in CD with nested translation always enabled. It is out of the host
awareness hence reserved regions must be added to the parent IOAS.

for VT-d identity must be configured explicitly and the hardware
doesn't support stage-1 identity in nested mode. It essentially means
not using nested translation and the user just explicitly attaches
the associated RID or {RID, PASID} to the parent IOAS then get
reserved regions covered already.

With that it makes more sense to make it a vendor specific choice.=20
Probably can have a flag bit when creating nested hwpt to mark
that identity mode might be used in this nested configuration
then iommufd should add device reserved regions to the parent
IOAS?

>=20
> Which is definately complicating some parts of this..
>=20
> Jason