Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Robin Murphy <robin.murphy@arm.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Baolu Lu <baolu.lu@linux.intel.com>,
        "Alexander Duyck" <alexander.duyck@gmail.com>,
        LKML <linux-kernel@vger.kernel.org>,
        linux-pci <linux-pci@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: RE: Question about reserved_regions w/ Intel IOMMU
Thread-Topic: Question about reserved_regions w/ Intel IOMMU
Thread-Index: AQHZmZR0JpzuQIuRD0G85gaP7++jm6+AOJSAgADP/QCAB+LmgIAEOrcwgABAlgCAB47s0A==
Date:   Wed, 21 Jun 2023 08:16:52 +0000
Message-ID: <BN9PR11MB5276B96470F3BF1E7077CC018C5DA@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <CAKgT0UezciLjHacOx372+v8MZkDf22D5Thn82n-07xxKy_0FTQ@mail.gmail.com>
 <CAKgT0UfMeVOz6AOqSvVvzpsedGDiXCNQrjM+4KDv7qJJ1orpsw@mail.gmail.com>
 <a1cff65b-b390-3872-25b5-dd6bbfb3524c@linux.intel.com>
 <b24a6c7b-27fc-41c0-5c82-15696b4a7dc1@arm.com> <ZIiRK2Dzl2/9Jqle@ziepe.ca>
 <BN9PR11MB52765C24405D2475CF3CBEBE8C58A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZIxTmGU4a5dniEY3@nvidia.com>
In-Reply-To: <ZIxTmGU4a5dniEY3@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?hYGCP6zE19s/RQD8oT/gWJYkSES1hoJmAoOwaDlCBJ2+7sDCXOswILrZzzRW?=
 =?us-ascii?Q?AFKBtlyr00LsnIUo7HemXyW1uRaGeGB9qqmolWrbXu8THRKrzABl+NLJXd8p?=
 =?us-ascii?Q?acbrVvk4d+qoSrLm6LNt5x8sbUHQutqplbhUaWjWYfVYHyyHAJhnNo6CTeQ7?=
 =?us-ascii?Q?tlkKsZXwYOIoD53vDPoQ/RP0nVTCPb6fEQkQJXpuLASchakNrc5qHzNZYOuJ?=
 =?us-ascii?Q?mLtnxnXU6NIMwftKjM/0qmdO6vm3eTHBXjcMSsOrkdosbBf2kigyyKSm+N4I?=
 =?us-ascii?Q?t6IQzTRs/oOy5yC8WYuZpJMsvPUypsZcxOzUS5qzMvyAlhs/4Ko/i5HvMJ0b?=
 =?us-ascii?Q?CiZ0u52swTOPQShHLPcK0kdGDeBvnNlfiYNG9tiO2Nj7X7JPoUIjojOQwhZM?=
 =?us-ascii?Q?MLaLfwcmWKonbOeFQv9t1/z4768psxOTQCFR5oirzss7QFyA0g+Z5k4n2TAh?=
 =?us-ascii?Q?RvG3VXXBeJZTNY5pDtRmQ9HI8WBgFHZ4X31d6JJEJH2R4OEyOr1llOgPJaLU?=
 =?us-ascii?Q?K6/G/J8zuwWs7zogPRtnfezwBBqfFjpcn09c4P1kUX4rfl+xhaXuYcxSasUU?=
 =?us-ascii?Q?8032mDXJZc3aOe64eOU75SOOio88EsHiGXZNFVZAFqjX/94qHGIBy9OXLHto?=
 =?us-ascii?Q?hBZTWMLy7B1s9SAh/aIDYYAnLXf2yof/9qa/xHMSFaTPdrDZCYRVp+3G4e4v?=
 =?us-ascii?Q?BIaoy/Pu5Zg+oBzh/6W1/74GDQe49GeLRbRujKDJkL42RqJuikQf8MCTm3eX?=
 =?us-ascii?Q?BJU5BjvilDNompDxnXNl6Bua/WRDnD8+XslLc1JYkEUrb0ghb9SD7nv20T7c?=
 =?us-ascii?Q?yZuNEjF+jy1XmRbZsjomclXyOfus2TOD031BCweS4Ws1o7TRypsumjb93356?=
 =?us-ascii?Q?sB9eMiqpJFpmkSpcxkjz3iUtQYtzlwi0D+EJ39gjjK3IbC2VYsMve3kWBI++?=
 =?us-ascii?Q?Fnw9bbJRRfqcVmOxF9or3eduDzk4UuypX7BHvpkt5KUr+UOf3aA4z17/0gTd?=
 =?us-ascii?Q?+aXFohX/fqLQe7Yo/KPM7YcteVcAbiGndXb+9QyoYKC1oJKmL27XX2WWMUCD?=
 =?us-ascii?Q?ZNSDVrJFdWwpLsDrJW7JAjowYYD6AM6Hfy9cfzWbAQPXdNTc/88YlDzcMSKz?=
 =?us-ascii?Q?SVW/ux7aQusqC7j4aWVSmxcue2PrVIVrw1APWQoKIH0PM7AtnglpWqpWTJsz?=
 =?us-ascii?Q?F5O4YoiME9oG+beIBOLQ1FFDlCOJUBBd0eK8mdxec7GCMcwKaODUeTHaN3iS?=
 =?us-ascii?Q?zTWgQg3CdzaigTKTUOkhGIC8m1yfUc6PesMjlKdNn2f2uhzRnyejBxKsPcpD?=
 =?us-ascii?Q?XE+i+8nNTjWT3LlrGOjn9bOuUdVBvzOvu1rTVQS19x0wDcZzY1BFttluiNC0?=
 =?us-ascii?Q?P5ApWUYxxZv4+HOJhEhTpYUz5U2pF8GaKJ7H+Nm6zdaDOiuIVw+zmpQSQ3Oh?=
 =?us-ascii?Q?XHJQx0CbiiO5kdnHj2R0KN9fZ/haE+PYVlTAJp6IQbHmsY/4lUaQ+BMzO30M?=
 =?us-ascii?Q?D9idvP+HVC2ql7GRID7TwmGvcJYfMJSWy9pKUCzwop33bEG0rRwDGLB+el5W?=
 =?us-ascii?Q?hohroEbeHdg2E7Gw/YMOgxe5ISZeiTUAQnAVxb6E?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: aa90ffe9-5472-4c74-d5ee-08db722fdea8
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jun 2023 08:16:52.7533
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: A04PTd7hjzzHc4979Y9xZTb5XCq76nFFEYndB2/jJx4JqcWeiIz7X5sbdqgtDa02LtOkw7jABa0fsFZlT45NsQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6989
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, June 16, 2023 8:21 PM
>=20
> On Fri, Jun 16, 2023 at 08:39:46AM +0000, Tian, Kevin wrote:
> > +Alex
> >
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Tuesday, June 13, 2023 11:54 PM
> > >
> > > On Thu, Jun 08, 2023 at 04:28:24PM +0100, Robin Murphy wrote:
> > >
> > > > > The iova_reserve_pci_windows() you've seen is for kernel DMA
> interfaces
> > > > > which is not related to peer-to-peer accesses.
> > > >
> > > > Right, in general the IOMMU driver cannot be held responsible for
> > > whatever
> > > > might happen upstream of the IOMMU input.
> > >
> > > The driver yes, but..
> > >
> > > > The DMA layer carves PCI windows out of its IOVA space
> > > > unconditionally because we know that they *might* be problematic,
> > > > and we don't have any specific constraints on our IOVA layout so
> > > > it's no big deal to just sacrifice some space for simplicity.
> > >
> > > This is a problem for everything using UNMANAGED domains. If the
> iommu
> > > API user picks an IOVA it should be able to expect it to work. If the
> > > intereconnect fails to allow it to work then this has to be discovere=
d
> > > otherwise UNAMANGED domains are not usable at all.
> > >
> > > Eg vfio and iommufd are also in trouble on these configurations.
> > >
> >
> > If those PCI windows are problematic e.g. due to ACS they belong to
> > a single iommu group. If a vfio user opens all the devices in that grou=
p
> > then it can discover and reserve those windows in its IOVA space.
>=20
> How? We don't even exclude the single device's BAR if there is no ACS?

I thought the initial vbar value in vfio is copied from physical BAR so
the user may check this value to skip. But it's informal and looks
today Qemu doesn't compose the GPA layout with any information
from there.

>=20
> > The problem is that the user may not open all the devices then
> > currently there is no way for it to know the windows on those
> > unopened devices.
> >
> > Curious why nobody complains about this gap before this thread...
>=20
> Probably because it only matters if you have a real PCIe switch in the
> system, which is pretty rare.
>=20

multi-devices group might not be rare given vfio has spent so many
effort to manage it.

More likely the virtual bios may reserve a big enough hole between
[3GB, 4GB] which happens to cover the physical BARs (if not 64bit)
in the group to avoid conflict, e.g.:

c0000000-febfffff : PCI Bus 0000:00
  fd000000-fdffffff : 0000:00:01.0
    fd000000-fdffffff : bochs-drm
  fe000000-fe01ffff : 0000:00:02.0
  fe020000-fe02ffff : 0000:00:02.0
  fe030000-fe033fff : 0000:00:03.0
    fe030000-fe033fff : virtio-pci-modern
  feb80000-febbffff : 0000:00:03.0
  febd0000-febd0fff : 0000:00:01.0
    febd0000-febd0fff : bochs-drm
  febd1000-febd1fff : 0000:00:03.0
  febd2000-febd2fff : 0000:00:1f.2
    febd2000-febd2fff : ahci
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
  fed00000-fed003ff : PNP0103:00
fed1c000-fed1ffff : Reserved
  fed1f410-fed1f414 : iTCO_wdt.0.auto
fed90000-fed90fff : dmar0
fee00000-fee00fff : Local APIC
feffc000-feffffff : Reserved
fffc0000-ffffffff : Reserved