Date:   Mon, 13 Feb 2023 10:49:34 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "will@kernel.org" <will@kernel.org>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
Subject: Re: [PATCH v2 08/10] iommufd/device: Use iommu_group_replace_domain()
Message-ID: <Y+pN/vtO8I/oiMOJ@nvidia.com>
References: <cover.1675802050.git.nicolinc@nvidia.com>
 <4653f009c3dacae8ebf3a4865aaa944aa9c7cc7e.1675802050.git.nicolinc@nvidia.com>
 <BN9PR11MB5276C1807B710CAD3E5820D78CD99@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+Vh479cDD7LX2x/@Asurada-Nvidia>
 <BN9PR11MB5276268D3ED0360913A05C368CDE9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+bc8OrWfw3Fq57n@Asurada-Nvidia>
 <BL1PR11MB5271180A1DE8C739455AD06D8CDD9@BL1PR11MB5271.namprd11.prod.outlook.com>
 <Y+nrTsIz2UHH/1lD@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y+nrTsIz2UHH/1lD@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?lZLF37sdwQhVvAqjPcqlfbRqqGglP0RBjOX9Zb2VqO5XqkdgESpidgCkWi93?=
 =?us-ascii?Q?xKe1TgSj8RMZv4XccvBOGbJOq+YhoLAHz3c/bxL2IeX0vOWd/a/8XY06hetD?=
 =?us-ascii?Q?JidSJJZphosmygemqAjtDXq+kL54/Q9U+MPNEcvlMpx4BBPdtQvF5B6nY/b4?=
 =?us-ascii?Q?VURFNsGkVnm/PtzSYurY+y64hLaD64RfYrXcmFknmbhKj79jYOsp6bFmhJQd?=
 =?us-ascii?Q?oo39JwHSZs3tMhbqPJ/rX4soGdbBjBqEAQWUfMlmxspD+9A4G+lfpAIVk0Ti?=
 =?us-ascii?Q?lSCeXTX+cAG3pLQjsN4jk1vglciSGk6jOi7q1k0aql+kwlBSYAe7Z9kje2cw?=
 =?us-ascii?Q?T9V55EYrgWVsiylUl4HxTLkDtpv3Qc/1PypozzqfoI7PSZZbyRIuy/jY0uek?=
 =?us-ascii?Q?QqXMrYF+S/xEC2sR1ivQzhCe/j70zelMiIdn01Dte9mQzE4CR04FxzXw+VB4?=
 =?us-ascii?Q?DAXWQq9PHtv55huwwdiOBgiifZ/OhjPwfA3RrduVet37d04BoWHZQhxCTuwJ?=
 =?us-ascii?Q?xRXBOUEGm8lPfYs6IpGdSXVEfE+fhkMcHmFLXeATaDQC+pIit1Ib5RyXAKyk?=
 =?us-ascii?Q?ydbCHwClr08lNN0all+o+p+NLAH37x9ePwRDTu2ouAdgpsYXXLFTmosa1YWo?=
 =?us-ascii?Q?04J6gWXj1TfoeMewnAFVBeVHRBkx62rJzIadvDS/B7kC8Z6x/KAi3L5AW7Y+?=
 =?us-ascii?Q?3CezGSBLS8KzEhjmjUiNMaMZzlYWFupdPrqf6jfdpP4u0Z3ntK+7T2Xb9iDh?=
 =?us-ascii?Q?UD84GqK2EHkDrY+tNtUDtVWAZw71DPvMN7QCJk3SIsRwwTri4PXe2DYeMMTi?=
 =?us-ascii?Q?FBnQOs+O5ewZ+KQxDLIT1RQWAxGq3XEhVsg6VSai16XFFWseFs44cBFf81//?=
 =?us-ascii?Q?IErO96kJaxJ2Bj2DNyU45dpDwHkDjLI2PYQ9LaOyCyxv86EOglcm6QEU7KPF?=
 =?us-ascii?Q?98sndLUCetitqfMFHecsxVDURpYepRRGQPfQkmRIZjwukrQ8c5FXXbX9MXcj?=
 =?us-ascii?Q?10hJHpufpgENb1mQudKnwjQKS4WpKuPhZ04M+btMnLo/jSZIO1jbvP79soxK?=
 =?us-ascii?Q?nbwmCr36MZEOY2vzzR67yAWReZjEDVRR9HWwCACnx5eN23MbMsgothMgwTF2?=
 =?us-ascii?Q?OskHUa8E56DUKRqnG+5GFye7rcGCafD0x3KmHsyszxL9qjI5TTGjn0lDSpxc?=
 =?us-ascii?Q?Q5EEuaq80eN0FzKy8Pc2/e6m0qZpdZWfmskuTLLCRQ9hcLJ4Qm0MkP+b7SzL?=
 =?us-ascii?Q?BBoR9OS+1K5dD0voIxcMr7llmkk1E7VtF+UI++P5/FbuJbKzWevDBMXI1CuT?=
 =?us-ascii?Q?p8XU1mQ3LcA2JXLNVOpRy/+Obxq3rITYsd9jIu50MnOWVyt91e8TlktF1QcV?=
 =?us-ascii?Q?Da6HJyYj7aIYq0+Gl/4P7QLO+LpJ66reM/wZcfrvhHDM4OQ/vOfQi1OFRAJX?=
 =?us-ascii?Q?0+wsEvMrIBBtREEbW4WyXjmkdPW5bCFMQ1Z6XPRY5f2puk9ZKtNaZW3NSrO3?=
 =?us-ascii?Q?h4R8xHk3NLZe3ymSfJ95HQRK+No4s09mhZpX59aJYo2ASWHP8nuDhwpK1B9x?=
 =?us-ascii?Q?X5kkmlByPwjpuczwKuQp/xpflSiqELOs+suSSpTX?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9a62ea8a-8882-4a4c-b536-08db0dd18694
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2023 14:49:36.2306
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1JFxGkLetWD/Q0dMHnB5zje1i2WOx7RGts3OWVp1X+kjLAi7MQmt6HZgcMI4mBUp
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6327
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sun, Feb 12, 2023 at 11:48:30PM -0800, Nicolin Chen wrote:

> What about point 1? If dev2 and dev3 are already replaced when
> doing iommu_group_replace_domain() on dev1, their idev objects
> still have the old hwpt/iopt until user space does another two
> IOCTLs on them, right?

We have a complicated model for multi-device groups...

The first device in the group to change domains must move all the
devices in the group

But userspace is still expected to run through and change all the
other devices

So replace should be a NOP if the group is already linked to the right
domain.

This patch needs to make sure that incosistency in the view betwen the
iommu_group and the iommufd model doesn't cause a functional
problem.

Jason