Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Date:   Mon, 2 Oct 2023 11:16:15 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Robin Murphy <robin.murphy@arm.com>
Cc:     joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev,
        baolu.lu@linux.intel.com, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v4 4/7] iommu: Switch __iommu_domain_alloc() to device ops
Message-ID: <20231002141615.GA650249@nvidia.com>
References: <cover.1696253096.git.robin.murphy@arm.com>
 <458dd0ed839541693a49da33239b33cf4c48b8ec.1696253096.git.robin.murphy@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <458dd0ed839541693a49da33239b33cf4c48b8ec.1696253096.git.robin.murphy@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fdMdqQNE/5QVW78UzUQZA6CEZzXOra49ASVm4+s8yn9wqldp0+hcmzM4JWw7?=
 =?us-ascii?Q?EvuOLW7sLzpnCwiQsfUEIcfS0wMaCYcb0j6YPNARcjWVlAh7HoLmw82L05n6?=
 =?us-ascii?Q?PDA3lhtRxYgEWTmSiVbLEOH/garcy6iLTk4OznNhooU+rCwQJwdhBtHmFwts?=
 =?us-ascii?Q?fvMVoenE/IODRbGkwLo6Qp7I9r3hHNwQz+GLcWQCwBqOu5t1fsvjsx4ZxHOo?=
 =?us-ascii?Q?IAfJCK5Lw+aVD8Xt5/cinQlV4yqMMzHKo7vg3+qJoFm8ausIBK6opxjr3F49?=
 =?us-ascii?Q?TgAC5F1tZL+Lq+mXuaBaxh4kyjKQuqByUbv5IbbAV7TkHIFtT5HD3TRFxF8+?=
 =?us-ascii?Q?NhBr/CB7NfguCv2jjHou60kNlh/+ZmrMETreXWSk5U+mN81FBfpIYYiWDhwx?=
 =?us-ascii?Q?4uaHqudSfF+lmy3X2GOm1t5wW3QMeITvdTiHlou1ByeXA1Al9r4qiBr6YkIv?=
 =?us-ascii?Q?YhOdXnv8FoQF3C5gs5PWv+RcdZp5CNpuWSGW9uFVYUB0FKo6uknHAIgrltB4?=
 =?us-ascii?Q?XrMLr2qfdZAKGdtLtTgDD6J0eIVFnDeYX9SuzkKol2Qfq/Pt1ejfRr/QIgik?=
 =?us-ascii?Q?gugq+CyDU8kJuYhKqXmIldh01mucFSUOeeG87itTbhAsnd0FeEuGOkAFCHFR?=
 =?us-ascii?Q?4eFW5jfDDtfAe+EAseISPWGeQUNzqcWz3X/jPAPYrgw6R7SsSp+uotOzB2oD?=
 =?us-ascii?Q?CgBSf+cZw7DmoyJYyNwUUuY+ug+s70yz1fhQ6xIhXJsWUNxw9tL4gYvOhNIW?=
 =?us-ascii?Q?ze9rzmTenptZCHMaoQWcOk3QiHVwlCua9lhXUjZNT4CqacwTwH5+5FCkZ2QR?=
 =?us-ascii?Q?dE1r4KFCUqZBWsg+CJCXOEluhsGcIoYAEQlWDZdTxVjheYeDirzop8F96xXQ?=
 =?us-ascii?Q?55g9kN07cEtWf1whjfHvLSIkeJvG8r1a3ve4QWprlG/ni1BSo3PbCUzA6Dc4?=
 =?us-ascii?Q?sdmp0qTszBxYKxYu6pDimuTKVZX1qW2GYsxmDQp7KIBtVoASS6e85tsEIEGx?=
 =?us-ascii?Q?r3ZfTRbmXW50D93+S3rUpsGVdlsIIJ6sB/hEoJg4X6c+jM1JbgiHJKFSm7GJ?=
 =?us-ascii?Q?KbngyIllwMjFPMaxMlmJcSkw35Bz1mchJ0c4s8NjD47Z/bSVHMt/utchL67n?=
 =?us-ascii?Q?nCe2V2rwezn4HQrgEgO9iV5kwP/9Q1nEQ6EaM2S9H4ShpYUE6Vr3z66Wo3b/?=
 =?us-ascii?Q?pz6GZW2BrwCaOTuxFbSd6wsM263gzvShu8BYZtBVnsjUw5yx7bjN3tAiNWTr?=
 =?us-ascii?Q?isJzTEZcUExXpS2Yhr5LvIoOhWg2hAkehZJEDLf+T2ZwfKs2XlCCNVumdnlJ?=
 =?us-ascii?Q?IWQ8c28YxUsFvF7kBj/4nqbbi/fjgdXclRgG6LO6GutHil2TF4VOU+/eun2y?=
 =?us-ascii?Q?4VJbYnRr+y6Xfn/yLS6mMiI4ALDdxpJPZFeSCIH+FwLVeqJ+ZyYviTuGYkU3?=
 =?us-ascii?Q?urAVCHO0kpTAjyxnCJR3rhS8EsNSKw94tdnVVbhTFMAMn/D+tTY5BN1qqNUN?=
 =?us-ascii?Q?OPrAKSYocKrKnHokpepHy8UTqr1WXmD9jgjt6EjP33OfbdCPKrSLNOg3yJEA?=
 =?us-ascii?Q?enW5kX1a/yuk+/lmRHcHAvknAp3Vzl33VsIK3Dia?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8a6b1768-924c-4e2c-d2db-08dbc3522566
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Oct 2023 14:16:18.6279
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m7O8BbZOX5XmFk7kpFMAFDI+xixQ5TyDPI4gNFl4X80fNOcmwdyJFfB7/G2kdApR
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4958
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 02 Oct 2023 07:16:51 -0700 (PDT)

On Mon, Oct 02, 2023 at 02:49:12PM +0100, Robin Murphy wrote:
> @@ -2120,20 +2120,30 @@ static struct iommu_domain *__iommu_domain_alloc(const struct iommu_ops *ops,
>  	return domain;
>  }
>  
> -static struct iommu_domain *
> -__iommu_group_domain_alloc(struct iommu_group *group, unsigned int type)
> +static int __iommu_domain_alloc_dev(struct device *dev, void *data)
>  {

Why? The point of this design is that drivers are not allowed to
allocate different things for devices in the same group. So we always
force the driver to see only the first device in the group even if we
have a more specific device available in the call chain. 

This patch has undone this design and passed in more specific devs :(

The new code here:

>  struct iommu_domain *iommu_domain_alloc(const struct bus_type *bus)
>  {
> -	if (bus == NULL || bus->iommu_ops == NULL)
> +	struct device *dev = NULL;
> +
> +	/* We always check the whole bus, so the return value isn't useful */
> +	bus_for_each_dev(bus, NULL, &dev, __iommu_domain_alloc_dev);
> +	if (!dev)
>  		return NULL;
> -	return __iommu_domain_alloc(bus->iommu_ops, NULL,
> -				    IOMMU_DOMAIN_UNMANAGED);
> +
> +	return __iommu_domain_alloc(dev, IOMMU_DOMAIN_UNMANAGED);
>  }
>  EXPORT_SYMBOL_GPL(iommu_domain_alloc);

Should just obtain any group for the bus and pass that to
__iommu_group_domain_alloc().

Also, how does the locking work here? Definately can't pass dev
outside the bus_for_each_dev() like this.

If this needs to sweep over arbitary devices that are not the caller's
probe'd device it needs to hold at least the device_lock to prevent
racing with release.

So I'd structure this to find the matching device, lock the
device_lock, get the group refcount, unlock the device_lock then
get the group_mutex, check for non-empty and then call
__iommu_group_domain_alloc()

(there is a missing lockdep annotation in
__iommu_group_domain_alloc(), the group mutex is needed)

Jason