Received-SPF: pass (google.com: domain of linux-kernel+bounces-187702-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Thu, 23 May 2024 12:16:05 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Vidya Sagar <vidyas@nvidia.com>, corbet@lwn.net, bhelgaas@google.com,
	galshalom@nvidia.com, leonro@nvidia.com, treding@nvidia.com,
	jonathanh@nvidia.com, mmoshrefjava@nvidia.com, shahafs@nvidia.com,
	vsethi@nvidia.com, sdonthineni@nvidia.com, jan@nvidia.com,
	tdave@nvidia.com, linux-doc@vger.kernel.org,
	linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
	kthota@nvidia.com, mmaddireddy@nvidia.com, sagar.tv@gmail.com,
	Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>, iommu@lists.linux.dev
Subject: Re: [PATCH V3] PCI: Extend ACS configurability
Message-ID: <20240523151605.GP20229@nvidia.com>
References: <20240523063528.199908-1-vidyas@nvidia.com>
 <20240523145936.GA118272@bhelgaas>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240523145936.GA118272@bhelgaas>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?qk2qnUvO/ZvwuNenEfO6mmgyOGF/a8l4G7236ghsMN0kaH8SOCxdeDipv4mW?=
 =?us-ascii?Q?buRQCpVYOPh1bzdFJJrAkI5zd7aKKZ1uEQA6HpaQIQnvE9xN9IGSA1wfJ733?=
 =?us-ascii?Q?0Fyg5LZrigSWSJLFDOq0KaPTbxWnI+yzUhPF7MwDvF2v3+5E4EDRyOCDq+X7?=
 =?us-ascii?Q?8kc2K+9xLUN3xvUooHEIWAPeN2oznjp27oZQ+TVDQkz4Ivz4OD6GfX2Hjdkq?=
 =?us-ascii?Q?Iq+Y67NVUlcB4DCp+gxK/xOAWyIOgCGerEKtehF7TwPUnIR7bmGq5Vd8LGjI?=
 =?us-ascii?Q?6laEOy89rBjV59scUGcQY3ICadcrdg6NH/43NihGQJdlUdLFD05EF90GpCoH?=
 =?us-ascii?Q?nfffsHvEOgBN2XO3o60rbNujVrNUXQ8QGhkq8K9zlqBQRdVv60jZ2/zUjyjZ?=
 =?us-ascii?Q?swAsRXvv/oaf9xovBSY2cmWJsrELv0o758TtljXlA5/8F1vEDOI8FaxNMDJc?=
 =?us-ascii?Q?neqXeRdkhA0141QAQ0WlwdlemH+INxWV+io5OE6aybviaJ6/73i8BxzE3Kvp?=
 =?us-ascii?Q?tCgVXH853r0nuD0hpz4k0SCjvthspe1/FKqH4KLu6G4w64Xbpgu+WV/4v3lb?=
 =?us-ascii?Q?BXD/lp3xa8k8PJyfoxFYNbLsMhtC0llySWkrj2M3Pqm84VHESaYiU+SScyKH?=
 =?us-ascii?Q?jODK7iBuZy8RkFREX5FEFUboSLlWY0gI37HfT+9UEjQQsp31xlOIcuYzrytj?=
 =?us-ascii?Q?m8/OrjHkRzmfn9sXXkrMwIPMNb0f5EPpfdW0Qw2VJNWlX6I4pTtEK5R4ZZjy?=
 =?us-ascii?Q?YVpwsil8dhMvnVhLKMMLLxcAyuMZAXgR6QWDamTgk96Pe/F69McnDHSqVZpB?=
 =?us-ascii?Q?XrhLv2Ajx17Zi4MGarlPP6AV+SasgCxkyQtujoC7pNF1wnxB0/L4hGXq/HS2?=
 =?us-ascii?Q?YjOOYr+n0Dt01BDCZJcdrDBtfNILzY343U9Ayl+tt+8eMXI3++Yu62zTyIP7?=
 =?us-ascii?Q?5rN1B1VnH6S0Nz2TuujI4g0cQjkoSKErUWfnRpcX2nUpYiYj80ILYNLnQsFN?=
 =?us-ascii?Q?IEBoLgsT8pQdsXH/PFInKSWj5OUXuXpr+SbrKSAjS7xd2GLOq/g3CNX7/kFn?=
 =?us-ascii?Q?5+oXbdDrlG9GQlCGAm63CTpEefqgo+8IML86Uzmm5+nDzD8APcMK9Zv2GiKB?=
 =?us-ascii?Q?A4wAASyuqoyHG+3Tobvn2huuE8QvOVH9asfiYQvX5MjTM/Yq+hCHbk8gvcVW?=
 =?us-ascii?Q?KB98R3AshHexSuOs6GJojLdyu56CMd2s19XOQTTQOG8OR0tdYRbBHHV2pfee?=
 =?us-ascii?Q?P6ogw2Gfgq2BFJvmKw1d/Bo722kgH9HpsRLfUeN6R8rKee0big4NsVnfgu+b?=
 =?us-ascii?Q?LGlCGepTHFCndumlx4tEN3iZooY8QHtT37HDe9Osqk3dYIooXrSh6z0O19GM?=
 =?us-ascii?Q?tcmiGRxWDcNwRW9fC/Z3Sz3eIIzkmtXHijYAGhJCQFvTuKxy+yClnnK0xMxk?=
 =?us-ascii?Q?ApvSgXnVl5EmbWW5Wd8PrYqsVxeIHtqXiaLBNBnNnPJEVIcoviXdS/8C9gUS?=
 =?us-ascii?Q?+TvRyIoTnnqppCSvOw+Z7aS/cNnO95dh+ZhtUdJ/SljOVP66QbkxeQIhsn8i?=
 =?us-ascii?Q?zX/VOGvwKcTpSwYqcdo=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 68a0a0df-bc4e-4b98-dc97-08dc7b3b4599
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2024 15:16:08.1990
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZkuQyvRZTk7emY+speGnHFovBD1jOcDM+B6t+Oeuoangt59sLr3sxt6YCK0cjasT
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8658

On Thu, May 23, 2024 at 09:59:36AM -0500, Bjorn Helgaas wrote:
> [+cc iommu folks]
> 
> On Thu, May 23, 2024 at 12:05:28PM +0530, Vidya Sagar wrote:
> > For iommu_groups to form correctly, the ACS settings in the PCIe fabric
> > need to be setup early in the boot process, either via the BIOS or via
> > the kernel disable_acs_redir parameter.
> 
> Can you point to the iommu code that is involved here?  It sounds like
> the iommu_groups are built at boot time and are immutable after that?

They are created when the struct device is plugged
in. pci_device_group() does the logic.

Notably groups can't/don't change if details like ACS change after the
groups are setup.

There are alot of instructions out there telling people to boot their
servers and then manually change the ACS flags with set_pci or
something, and these are not good instructions since it defeats the
VFIO group based security mechanisms.

> If we need per-device ACS config that depends on the workload, it
> seems kind of problematic to only be able to specify this at boot
> time.  I guess we would need to reboot if we want to run a workload
> that needs a different config?

Basically. The main difference I'd see is if the server is a VM host
or running bare metal apps. You can get more efficicenty if you change
things for the bare metal case, and often bare metal will want to turn
the iommu off while a VM host often wants more of it turned on.

> Is this the iommu usage model we want in the long term?

There is some path to more dynamic behavior here, but it would require
separating groups into two components - devices that are together
because they are physically sharing translation (aliases and things)
from devices that are together because they share a security boundary
(ACS).

It is more believable we could dynamically change security group
assigments for VFIO than translation group assignment. I don't know
anyone interested in this right now - Alex and I have only talked
about it as a possibility a while back.

FWIW I don't view patch as excluding more dynamisism in the future,
but it is the best way to work with the current state of affairs, and
definitely better than set_pci instructions.

Thanks,
Jason