Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 28 Jun 2023 11:36:22 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Zenghui Yu <yuzenghui@huawei.com>
Cc:     Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Kevin Tian <kevin.tian@intel.com>,
        linux-kernel@vger.kernel.org, iommu@lists.linux.dev
Subject: Re: [RESEND PATCH v8 04/11] bus: platform, amba, fsl-mc, PCI: Add
 device DMA ownership management
Message-ID: <ZJxFZkxvPNj74uQe@nvidia.com>
References: <20220418005000.897664-1-baolu.lu@linux.intel.com>
 <20220418005000.897664-5-baolu.lu@linux.intel.com>
 <6472f254-c3c4-8610-4a37-8d9dfdd54ce8@huawei.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <6472f254-c3c4-8610-4a37-8d9dfdd54ce8@huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3PQO15JMjgbxpAUDoVPs1Ljg/dpFHJtH/ej1xGTI3MDfHdO0X93viKRh1AIH?=
 =?us-ascii?Q?GOi0ALfV+SnjZpO4xJ0daVrR+J+eu0icMOMoxato6vt7E9qxH4LrNrIqfG+J?=
 =?us-ascii?Q?IPntQlA7rtMoflOU2zA+wdti7ai5ub5LmSFJL/esYhzcCQOUEm7azM+a57xS?=
 =?us-ascii?Q?9pTGwemWeCpPQvkFUZpiR/PieWaG6oNsDo7uZSMn5n/ib4cCHHv1P9enti4z?=
 =?us-ascii?Q?Jzp0C54+k0nnQSCwdijLDgQw2dVszdorPGF4/x7D7O6nsPrMS3+lSnS8eou+?=
 =?us-ascii?Q?v9P8ebzxT6SvD6WmZeYoCGSC+dH7keu5g63bTrcfN3Zt5xy5Y6gcz3xe/oq4?=
 =?us-ascii?Q?lT1AYLjdlqbqQUHpixzoBSy6ZLk07SNK5mpMHMwcwLz2vWl9LvtCN5+Ui5Vt?=
 =?us-ascii?Q?sAe5CTNS5Cw3k5FXz9dIh+n8laZb+KlcKADM5SXcO4VdkLABCajtGH6tmEbD?=
 =?us-ascii?Q?+ecmN8e/DxR6HCwBKKb1b1xL8IUZ0yWGaM4L55Y2oiWgX6X7bwikYXGeTolw?=
 =?us-ascii?Q?WdlTrypBsd8R2ChcuFLLe6Md6tOaZ/XiLs2hBx+qvAe8axYmNHqtD0nGP760?=
 =?us-ascii?Q?Fw/9lEXWdurbv4i7bdQ3QOCP+rEPyLuJiWsEIpjxJLRvPEiMaO2/x8FkpgMr?=
 =?us-ascii?Q?K7iNJG9jpjIpW36oGidnXQVW6FNdKv9AmZxspX7ur1L7oV7brdH2DUIJ7iGF?=
 =?us-ascii?Q?GmBUchyf4qvHZyWrMCJNuN2x2/M/uvm/jIJomJPiJ89332uh+kPizf40IPSu?=
 =?us-ascii?Q?CqKjxdGNKcr6pdYpsQIoXNvE86XbulVNjyyURpKkmA6x9NXj67UG6okNkVgw?=
 =?us-ascii?Q?OimGBMBBVe5h8FQHgXs/xa9YYi/eDEBUERJ8w+pf+siqg8MWnK7cdEQrbHGy?=
 =?us-ascii?Q?btiYZsUvh4czFzEYthhWCgBuGtPrV9g1upPATXsEd7UFXrFeA5Uo3pZUbR+H?=
 =?us-ascii?Q?h2Ji7EDMXpfgh9HB7DuJVaXPF1WBEfnQmmHRai9pUVQ1Sf4FrfbNLl3Acuif?=
 =?us-ascii?Q?ifxb3pYShxm2c0Hhzgm5clBO49au1jeBI7gvJswdQJ1y9paP+iIr95qSKYzS?=
 =?us-ascii?Q?hR4pU88XlX3SR33Vy/meEpyvF+ISZakr81p4n4adeEcrHDD+SqFOnhA4sABy?=
 =?us-ascii?Q?MZf84tBq2tOukmpZi0a34hWwCqlOHjgmqNp/xP9BviL+mHADhDoXraPFp40Q?=
 =?us-ascii?Q?5X0aZMWw/cFxykFOjx1GkoLFzMxAd0EUuPbMe5lRmD2OcbW6BkA5YlvpwTmB?=
 =?us-ascii?Q?voLfxHSHmJqmTyJ2ipnS9S+oi65xPc+IWbb31YRNW89CJAW1jFQR8WfO2OYy?=
 =?us-ascii?Q?vVsP+qm/X7BdUVD74mMhGJ4w5DX+OKI9NxNa6WQPk4qem4peKMKr4oa9LhSQ?=
 =?us-ascii?Q?71SbX1SJQHzia5euNK+0h50xZ2Nocfybh5tlr+AHzU5mn8UfF1+Cagt0npho?=
 =?us-ascii?Q?d6vywSeNyH6pwfZL61NTtafdxILr9hZX1JGU/NZ77YHBxjWGf2wbK1Qr3/Bi?=
 =?us-ascii?Q?aA4r2K5NIzFNamfGwsBqkTFtvAzJx4AeuHSlaDNMtuHX6zENrviunVMO2es4?=
 =?us-ascii?Q?27hWCekp0y5ktZ/rR1Z4psdTKeSVvFhZG9aWftqG?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bd6800b6-633e-44da-1ab7-08db77e50c3a
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2023 14:36:24.0821
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: stNyoPnhbXiVQiH8y3YRI2qctvAA6W5Xw0/wHMgL+rO+L3FVuZrBHrQQfBsrrgIP
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4483
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Jun 26, 2023 at 09:02:40PM +0800, Zenghui Yu wrote:

> It looks like on device probe, with DEBUG_TEST_DRIVER_REMOVE,
> .dma_configure() will be executed *twice* via the
> really_probe()/re_probe path, and *no* .dma_cleanup() will be executed.
> The resulting dev::iommu_group::owner_cnt is 2, which will confuse the
> later iommu_group_dma_owner_claimed() call from VFIO on guest startup.

Does this work for you?

diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 9c09ca5c4ab68e..7145d9b940b14b 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -693,6 +693,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 
 		device_remove(dev);
 		driver_sysfs_remove(dev);
+		if (dev->bus && dev->bus->dma_cleanup)
+			dev->bus->dma_cleanup(dev);
 		device_unbind_cleanup(dev);
 
 		goto re_probe;