Received-SPF: pass (google.com: domain of linux-kernel+bounces-198739-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Vidya Sagar <vidyas@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>, Bjorn Helgaas <helgaas@kernel.org>
CC: "corbet@lwn.net" <corbet@lwn.net>, "bhelgaas@google.com"
	<bhelgaas@google.com>, Gal Shalom <galshalom@nvidia.com>, Leon Romanovsky
	<leonro@nvidia.com>, Thierry Reding <treding@nvidia.com>, Jon Hunter
	<jonathanh@nvidia.com>, Masoud Moshref Javadi <mmoshrefjava@nvidia.com>,
	Shahaf Shuler <shahafs@nvidia.com>, Vikram Sethi <vsethi@nvidia.com>, Shanker
 Donthineni <sdonthineni@nvidia.com>, Jiandi An <jan@nvidia.com>, Tushar Dave
	<tdave@nvidia.com>, "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, Krishna Thota
	<kthota@nvidia.com>, Manikanta Maddireddy <mmaddireddy@nvidia.com>,
	"sagar.tv@gmail.com" <sagar.tv@gmail.com>, Joerg Roedel <joro@8bytes.org>,
	Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: RE: [PATCH V3] PCI: Extend ACS configurability
Thread-Topic: [PATCH V3] PCI: Extend ACS configurability
Thread-Index: AQHarNt+aNbsAVmh9EuxtikUDxoII7Gk6fAAgAAEm4CAEMzwAA==
Date: Mon, 3 Jun 2024 07:50:59 +0000
Message-ID:
 <PH8PR12MB667431B8552D271F906F8F4BB8FF2@PH8PR12MB6674.namprd12.prod.outlook.com>
References: <20240523063528.199908-1-vidyas@nvidia.com>
 <20240523145936.GA118272@bhelgaas> <20240523151605.GP20229@nvidia.com>
In-Reply-To: <20240523151605.GP20229@nvidia.com>
Accept-Language: en-US, en-IN
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?HctwYmh1LrQAa19T8KyuBdbATa96hfN3zTh0leHPZfPQ61NXxw7BbrP4GEy9?=
 =?us-ascii?Q?+GrlfVLfMxZJrAtAMdRZ2oqJ9qhMnP5a+7r1ZdArQNO7Mhm2+AAmEFrY2N44?=
 =?us-ascii?Q?5e82PzgqW8xknZ/8xYW/ubRpfUeMxZ434d1Z6kgY0jGdYj1J7FQGi3AN6vyK?=
 =?us-ascii?Q?gL8qbR9YCWAV0OAs3JjpWPqfgjJfIVhzlDPH4CwYS4Or5q48J4On6U1MsQN1?=
 =?us-ascii?Q?IbMzH50bK/h2/bnuS4VR9kXNLGzvhTslN6wo04XkpBmu1g4rKZ8i1SKBgDLX?=
 =?us-ascii?Q?Ulu3UaSjwW3hwJVXOTVh1+l7qVSF81AJGWKZYqOxLT1jxev5pummyyZTyLcW?=
 =?us-ascii?Q?/r609ZoHJHNpbpuUVLL9egADk3ViZMLrsPPJHm5O8H0Oj4rCo4DImZEHyszn?=
 =?us-ascii?Q?GoWGRxoq/YIDdikEVjDnJLxUTBM2CVrZe6QkhVVbdiYOBxzYgvygYdQxgwN3?=
 =?us-ascii?Q?uc0jTjJLoBA1y7lKsxKE6vMwy12/9KdqtqB8sLsO3QwbUzOw5El4wOQJRLyg?=
 =?us-ascii?Q?1C1DnWMGwmHnHhqjGRbJPwlp9W4o6UK0qALGyVASmnt1F79gAlqI0aNlyzN3?=
 =?us-ascii?Q?gLXpKgCzMuwG4oI3cJx3Hc2m5adjShdSbujJjArdNV1MR0ad9tDfh1n0Oo9J?=
 =?us-ascii?Q?YBDX0/gfmDyqJTJ8Ycvpz+gIKoo8VZ00TjOBUWhPhRuZER0OSeV/MIoFSOge?=
 =?us-ascii?Q?K3ihmSNvVLPgcQeoCMY7+CY0d+hbByIWg82KsJkPt1avwjxmEmPVl5dZ8tjI?=
 =?us-ascii?Q?f0FfCqrAMCOcgDyJ8Z6gVK+9c29C8zG+lR6IwlaSwu22GxzNqagrhEp6PrY5?=
 =?us-ascii?Q?Ic2ZGLBpSVGDj90jyFJwyZ8zRcY1TbyIfmhsmHM7HWYDDukzJm206NGoZ7xU?=
 =?us-ascii?Q?16XJbvT/FDeTzR4hd0XW7D/6w26IyCcsgTv3IIN09CwTT4ZuDbz/73URDHoy?=
 =?us-ascii?Q?E/tQQ20Msf4rNntKayHv9qS31ityl5Gx1r2du8xX+A+VnmWaREOd3yFvcpHe?=
 =?us-ascii?Q?Lyf4czjLkUBobdEPqM64+bhsHDrP1a306MWvk9wHYva6mvS5jD3PdVB/peJn?=
 =?us-ascii?Q?anXh/kExFKdCZ6U9Jb5AT5wU9TLN58IdCdgfuuXX04D2tLT6Yv5dvu/tocoK?=
 =?us-ascii?Q?3s/TjWmnNb2l5VkPTp3dv9lXSkgACfeKSq3Poed79L0IW/a0TqQ4xlVpCVcO?=
 =?us-ascii?Q?TVSGnWbx7cXQrKbklg8JdvScujERq0B3sr0ql986/p+5H/QxtKUKIl+NKeJ4?=
 =?us-ascii?Q?vRmGGPfArJ9F/RVE1kTIkDvyjQTsd0WJZLxzBjnJJAWvNV7n/OTYvojncBPJ?=
 =?us-ascii?Q?ODyg2l98HiBTyU16MrEH+gE7vdldDvZ5HEV3bw02jbLbfIziNGv5BDAiKq4l?=
 =?us-ascii?Q?/FphU3EziAGmXHTlXWj5XrJjw7WlaClUq8BvjUjpgHgEvFrk9ozUxj36AsRx?=
 =?us-ascii?Q?zgsG/2ewnUh6uulI+6irPuhQbXehtGCKjn1VM3//3OgNccTNpfZ98ycDB2Ym?=
 =?us-ascii?Q?BZ7D9HbVl/gDYsdNaJnZp40oa0KPtxEeqsp8O1VjAG5yGjI3Q5eM4h92N1IR?=
 =?us-ascii?Q?8VQViJVh5a0Hc+L1qH0=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB6674.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b2d75e41-8784-40ec-3669-08dc83a1e8d8
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jun 2024 07:50:59.8680
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: flK1psFOIiRhrqJ/TxO1PI0epzwrJCywBfU7/sZGCrljGG8i6FssMPFACNeI1DeVhLrMutg1zr3CRABmjgBNzg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4432

Hi Bjorn,
Could you let me know if Jason's reply answers your question?
Please let me know if you are looking for any more information.

Thanks,
Vidya Sagar

> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, May 23, 2024 8:46 PM
> To: Bjorn Helgaas <helgaas@kernel.org>
> Cc: Vidya Sagar <vidyas@nvidia.com>; corbet@lwn.net; bhelgaas@google.com;=
 Gal
> Shalom <galshalom@nvidia.com>; Leon Romanovsky <leonro@nvidia.com>; Thier=
ry
> Reding <treding@nvidia.com>; Jon Hunter <jonathanh@nvidia.com>; Masoud
> Moshref Javadi <mmoshrefjava@nvidia.com>; Shahaf Shuler <shahafs@nvidia.c=
om>;
> Vikram Sethi <vsethi@nvidia.com>; Shanker Donthineni <sdonthineni@nvidia.=
com>;
> Jiandi An <jan@nvidia.com>; Tushar Dave <tdave@nvidia.com>; linux-
> doc@vger.kernel.org; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.=
org;
> Krishna Thota <kthota@nvidia.com>; Manikanta Maddireddy
> <mmaddireddy@nvidia.com>; sagar.tv@gmail.com; Joerg Roedel <joro@8bytes.o=
rg>;
> Will Deacon <will@kernel.org>; Robin Murphy <robin.murphy@arm.com>;
> iommu@lists.linux.dev
> Subject: Re: [PATCH V3] PCI: Extend ACS configurability
>=20
> On Thu, May 23, 2024 at 09:59:36AM -0500, Bjorn Helgaas wrote:
> > [+cc iommu folks]
> >
> > On Thu, May 23, 2024 at 12:05:28PM +0530, Vidya Sagar wrote:
> > > For iommu_groups to form correctly, the ACS settings in the PCIe
> > > fabric need to be setup early in the boot process, either via the
> > > BIOS or via the kernel disable_acs_redir parameter.
> >
> > Can you point to the iommu code that is involved here?  It sounds like
> > the iommu_groups are built at boot time and are immutable after that?
>=20
> They are created when the struct device is plugged in. pci_device_group()=
 does the
> logic.
>=20
> Notably groups can't/don't change if details like ACS change after the gr=
oups are
> setup.
>=20
> There are alot of instructions out there telling people to boot their ser=
vers and then
> manually change the ACS flags with set_pci or something, and these are no=
t good
> instructions since it defeats the VFIO group based security mechanisms.
>=20
> > If we need per-device ACS config that depends on the workload, it
> > seems kind of problematic to only be able to specify this at boot
> > time.  I guess we would need to reboot if we want to run a workload
> > that needs a different config?
>=20
> Basically. The main difference I'd see is if the server is a VM host or r=
unning bare
> metal apps. You can get more efficicenty if you change things for the bar=
e metal case,
> and often bare metal will want to turn the iommu off while a VM host ofte=
n wants
> more of it turned on.
>=20
> > Is this the iommu usage model we want in the long term?
>=20
> There is some path to more dynamic behavior here, but it would require se=
parating
> groups into two components - devices that are together because they are p=
hysically
> sharing translation (aliases and things) from devices that are together b=
ecause they
> share a security boundary (ACS).
>=20
> It is more believable we could dynamically change security group assigmen=
ts for VFIO
> than translation group assignment. I don't know anyone interested in this=
 right now -
> Alex and I have only talked about it as a possibility a while back.
>=20
> FWIW I don't view patch as excluding more dynamisism in the future, but i=
t is the best
> way to work with the current state of affairs, and definitely better than=
 set_pci
> instructions.
>=20
> Thanks,
> Jason