Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>
Subject: RE: [PATCH v2 00/11] iommufd: Add nesting infrastructure
Thread-Topic: [PATCH v2 00/11] iommufd: Add nesting infrastructure
Thread-Index: AQHZhBZTNX7dDuLMFEel2CdlG13MGq9hZYMwgAAiQoCAB0Y6EIAVLTqAgA7xydCABWBTgIAA1LOwgADAhgCAARBZAA==
Date:   Wed, 21 Jun 2023 06:02:21 +0000
Message-ID: <BN9PR11MB5276B852A32F53BE8EAA1A7D8C5DA@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230511143844.22693-1-yi.l.liu@intel.com>
 <BN9PR11MB5276DAF0A11809CF8433EE338C7C9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZGdiS2m8jcd5OOt5@nvidia.com>
 <BN9PR11MB5276A74B2DA86C79908A420B8C419@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZH9AGWf1yRDu/86q@nvidia.com>
 <BN9PR11MB52763C7B838B04D3200322FD8C58A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZJBL8QLLBiwRsUSI@nvidia.com>
 <BN9PR11MB527663567ECB8AD52D3170818C5CA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZJGf3sgLKr9HLZuE@nvidia.com>
In-Reply-To: <ZJGf3sgLKr9HLZuE@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?QrcfOR3CEbxcDQbsFJmf8mFve7Yo9UMIupjXH92MbJOTcfAGztYOdnhwIzYx?=
 =?us-ascii?Q?57jxA4WD6CPExV099ihik4MWUKkC+ytcbWrHPCGX/KmPW0NxspNn64nn7MsK?=
 =?us-ascii?Q?4PaewIJE/aie0g/DvTS2ebJkmWR/iMp6GVoVRNjsgwGoOS1rGvT+Wc3NJApm?=
 =?us-ascii?Q?VuNdRb8BSNfUn3ZicZXFlc2kbOISdiig6ZwsIXaj2iKGTZcMyr78E4eLhUnL?=
 =?us-ascii?Q?PNJpAXaUn1XgGL6PJybaGosACPAd6IVFZ+V4Vv1Xyv+Syb9iIL7yCmH+eWNW?=
 =?us-ascii?Q?eazr09PSA89CC2xKV5gOmjUdTVWHPRnHNyXjpMCNSTIG46ZQDHjJOM5yo46/?=
 =?us-ascii?Q?7yGPmGG9Pqx1N81Z27FBfHLq5U6wjZV72PxgF9crO4al4Jt6gYBY836Kd5Ly?=
 =?us-ascii?Q?afx9zDzEcl/da7uTcN/F1/JaW/cUGmzrC8Z0bspiAMWUNM56fjPg4Rvf7bkg?=
 =?us-ascii?Q?2gccdQdahuNSYS5A19bTdlRv31dPod+3cyWPogywPCcF8iVunCqi2HEpEzl/?=
 =?us-ascii?Q?qQ1lYZgM9SNg0YJzBBEWwDEQAdpfZWJUDhOkgTL2LkEJ744fH/Lu1rzOB8N0?=
 =?us-ascii?Q?LLnncGIDDz0Rg+qwI3Je/JCW4fwJr8BuQ84JauFiHX8vLXwzFn5xAHjiGpME?=
 =?us-ascii?Q?qiyNVztn6f4iyixkoAyrCmcRFjaoUEH1rXFi8htpYQc59Gi1vKVAptG129hv?=
 =?us-ascii?Q?y0CPYgcDBZ3GC74Uc0AETdRsLdLYgWjRPFRYPFxQ6igiTbyY1sv2OBL+YhNk?=
 =?us-ascii?Q?E5DjpN+1pip1AcKSBPKNuk8ydRao80X6e2KLA482k01O2M7eNk3dvRx4M3IG?=
 =?us-ascii?Q?ept9Djk2YfCtoQsTSG0r0e+T8tHOnLttfsPhDc3r8b/kzdB4I/MBBuzG05Mv?=
 =?us-ascii?Q?HYpX8WlPnJEmYKjvkWr9GyI7u6ddzfn5ZVocSM2Z5GHqyHOl78/DswRPCJ8e?=
 =?us-ascii?Q?yAUPzTMNRpBhA8HDMuD51aZAtUuAhBzlOdmRADq2tciHuqnjtHD/6mU7s84M?=
 =?us-ascii?Q?QQNuZUdGzs/MICevFE1r38U2dwFSc5K7L7xQ/63PhSklJpsC/Mjrj1Opt14g?=
 =?us-ascii?Q?SKx7x3wHbg95hGLyCSTyDb5Jy6j4ikuGYPsCBJlLNgg6WcZBN/jPWzkcD6O5?=
 =?us-ascii?Q?msrD2/LSr1JQQgt9o3XV4jW3XNGl8u3f8mMiTvah3llFoBbtcGJvQGQVIdPn?=
 =?us-ascii?Q?BObFRgY3T9q6TntkCaMzKDpGn48DPQRIspMQnuJ1MgAJdgLICNCvZ+ctOY8x?=
 =?us-ascii?Q?owc798aGLlIRhUboPHWn9WvVJrlhvgLzFPlW0SsaJY9HqwFWKPCB8ruPADRB?=
 =?us-ascii?Q?z1MCUqV9m+QjOioOODw0TVxrM2KIw0WpzYMLkdMhHFl7YDFnBqn8jVRS5OnQ?=
 =?us-ascii?Q?fPjs1VhzzEfqG4yLMOWetP5MiTahghEnOsRpJsoMShuMozfKQ+S70pM83V+w?=
 =?us-ascii?Q?hjpGeRaHo19cUOto4bNAM165KbGnsRyzDHdnot5zorUxm8KNPRmR27SrzEDX?=
 =?us-ascii?Q?NZCcB8spCZRVfZ3Y5nr3BPPhkg5VhxfrVOcvgrm5LKuFi99CcG6eOvTiXzll?=
 =?us-ascii?Q?FtqkoD2NqeM3mmZgT2rHyKYfZHGsDVPLnfG/71uJ?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 970d10f8-3bfa-4002-e10c-08db721d13fd
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jun 2023 06:02:21.7913
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: NuulJxxEzCFRZUB8Q++0lwZL245ueO1pEzh3O4VLqoBJ+KOrB1qUzyvwYFYWobiqEqS6Td57HeUPieTq9ayAaQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5865
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, June 20, 2023 8:47 PM
>=20
> On Tue, Jun 20, 2023 at 01:43:42AM +0000, Tian, Kevin wrote:
> > I wonder whether we have argued passed each other.
> >
> > This series adds reserved regions to S2. I challenged the necessity as
> > S2 is not directly accessed by the device.
> >
> > Then you replied that doing so still made sense to support identity
> > S1.
>=20
> I think I said/ment if we attach the "s2" iommu domain as a direct
> attach for identity - eg at boot time, then the IOAS must gain the
> reserved regions. This is our normal protocol.
>=20
> But when we use the "s2" iommu domain as an actual nested S2 then we
> don't gain reserved regions.

Then we're aligned.

Yi/Nicolin, please update this series to not automatically add reserved
regions to S2 in the nesting configuration.

It also implies that the user cannot rely on IOAS_IOVA_RANGES to
learn reserved regions for arranging addresses in S1.

Then we also need a new ioctl to report reserved regions per dev_id.

>=20
> > Intel VT-d supports 4 configurations:
> >   - passthrough (i.e. identity mapped)
> >   - S1 only
> >   - S2 only
> >   - nested
> >
> > 'S2 only' is used when vIOMMU is configured in passthrough.
>=20
> S2 only is modeled as attaching an S2 format iommu domain to the RID,
> and when this is done the IOAS should gain the reserved regions
> because it is no different behavior than attaching any other iommu
> domain to a RID.
>=20
> When the S2 is replaced with a S1 nest then the IOAS should loose
> those reserved regions since it is no longer attached to a RID.

yes

>=20
> > My understanding of ARM SMMU is that from host p.o.v. the CD is the
> > S1 in the nested configuration. 'identity' is one configuration in the =
CD
> > then it's in the business of nesting.
>=20
> I think it is the same. A CD doesn't come into the picture until the
> guest installs a CD pointing STE. Until that time the S2 is being used
> as identity.
>=20
> It sounds like the same basic flow.

After a CD table is installed in a STE I assume the SMMU still allows to
configure an individual CD entry as identity? e.g. while vSVA is enabled
on a device the guest can continue to keep CD#0 as identity when the
default domain of the device is set as 'passthrough'. In this case the
IOAS still needs to gain reserved regions even though S2 is not directly
attached from host p.o.v.

>=20
> > My preference was that ALLOC_HWPT allows vIOMMU to opt whether
> > reserved regions of dev_id should be added to the IOAS of the parent
> > S2 hwpt.
>=20
> Having an API to explicitly load reserved regions of a specific device
> to an IOAS makes some sense to me.
>=20
> Jason