Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 7 Sep 2022 14:33:11 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        Zhangfei Gao <zhangfei.gao@linaro.org>,
        Zhu Tony <tony.zhu@intel.com>, iommu@lists.linux.dev,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v13 09/13] iommu/sva: Refactoring
 iommu_sva_bind/unbind_device()
Message-ID: <YxjV1y/FF0nCI/WO@nvidia.com>
References: <20220906124458.46461-1-baolu.lu@linux.intel.com>
 <20220906124458.46461-10-baolu.lu@linux.intel.com>
 <Yxd2+d/VOjdOgrR2@myrica>
 <682d8922-200d-8c89-7142-83e7b3754b8d@linux.intel.com>
 <YxhqbhMmWLeFS512@myrica>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YxhqbhMmWLeFS512@myrica>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?RqlZptRaD2vk/hRTXkzXA3xQ4iVAg6gcBHAlFypxicoqZ9rpYb6nPMe5Kam8?=
 =?us-ascii?Q?NjHUryyUhDxRBGicS9aGrFjfq+SNpi8ud0XARgYugNxGnHntDOB5bnWm1HY+?=
 =?us-ascii?Q?Ywk4SeOCqXpCsV45NniCsu0z24GufVXdgj4fIEgVD3yRBxM1whP4zdZFYzPc?=
 =?us-ascii?Q?p85+ViTxtuJdpPOG8ZIpwIIXUbIWNQy+edxHosp7ANIktS/BFGbY70QQc4TP?=
 =?us-ascii?Q?MDaNh6ECGgFC506GVqSxYldvvAZKVGZYRUQLAiKT9d2zKmu/0Lit6TMxYarn?=
 =?us-ascii?Q?S/+CnHj/OkiohkB6ZIoX4vbuPLsy2fvxgAUkS2rDtDhpQ8VECVSq64XXJIif?=
 =?us-ascii?Q?EnapsUxRlxDQ6t48XPJjeYAoZNvYricpyGlYx3BYhneuBxnnM95L7xaEBUDG?=
 =?us-ascii?Q?uh2aaP+UctvaQ/jcUxdZ0NtGszUXQKixoe9MtztO0xGrnZaXYZucxmAAnbxH?=
 =?us-ascii?Q?r43cicEtRmIg1CjtxwtmKWAKFrMrhlYUG/emmTfpPKXq+Fm32ErfEvoMP9Vz?=
 =?us-ascii?Q?ITb7qAjV/5gmpRgVuxjDQ+NL4hf6dgarfrJzbyDPNWDQmnUshis4REoCW5I6?=
 =?us-ascii?Q?AVU8wfKB0itjzTCOBCYuHZ8dVSJ+iVvMC37ew6yVaHR7EiGHtX6vDqnuPhwP?=
 =?us-ascii?Q?8Eb8UQqMnlWskqBDDbtl+hUQRJqsfJ2Xgic1S0dhiAxAtirmMozJgNVeKMzi?=
 =?us-ascii?Q?x5B4gOBpUZCqeYiSTBQed92QBboCRmybpH98KlQgTrh0NWJMo+zPg/gpBR+g?=
 =?us-ascii?Q?OLWA713Du4pNkm9QE9Gf2tcInHlgYat274pkhFvWYxNHsOwyqggfc37cwJcx?=
 =?us-ascii?Q?yzs5o5+J3suV/KZwMpKJPM6FHyLSAz3NmN8WSkL0ELokaBsQbLxayo+43OQn?=
 =?us-ascii?Q?MI4ysUBkZLpK1Ig5ptJfFFQE4eVsYkKNpTl6+l9rycOlKF/55rucPLFh8D3b?=
 =?us-ascii?Q?sPGtrpW6A3vGYYrZp0dTNhIf5Q8qBfmfywekosWuLpfMq9/0AxFws7zg5K5A?=
 =?us-ascii?Q?/AT8VPggUcQVQxVSYPc8LJVCOo9nCiYWwTg36Ex+FpkqsO4WD18UEXfpPSMc?=
 =?us-ascii?Q?37H8YMgjAJesaFqKcnbx5SN1P8thjU1yJpTW4ih7iPi4f5DJnDq2vAT+OE7x?=
 =?us-ascii?Q?P+212BbwPMS2mM4s62Cfk3dHeAC0ivQ9KcmqoQyVujBy2gvPhUesMmpH5GrX?=
 =?us-ascii?Q?kVE/Vl4vab3NtuY7GIS5+AcsaPZGO3JMl6H47Jq7gxc/c+dN4Uu8xpXIdWXe?=
 =?us-ascii?Q?sdGpq+NLfdN0N5DzvLXvptR9RawqCQSNojamDVjaHyMUZJNVhNMP6Jj6F3CB?=
 =?us-ascii?Q?oO1N2P8jRXKbQZyYQHqrdewZdWNlt3c/kNZ/nJKxA+tBP/EFejEiRQNkPVkW?=
 =?us-ascii?Q?+Q8JofedvjG9/swfhBc70e7sAClnDzl3eGzsNjKfs0MYR1pCiR1Tjpr25fh8?=
 =?us-ascii?Q?pPPHhgccU8uyS+w2iIk4zVnmCqWj5pUx6+Y2z2tmnGIkFJYjmWn93a1ExjXi?=
 =?us-ascii?Q?Qb2qlz+rbzhAkjuE5JNRQGOJTl/OC1/x9pGt4aTFij2jqPlHYlaBxgSQekR5?=
 =?us-ascii?Q?2o+Y4K6W1/pUMV9l+DW6E6VtdjJa/PZtiySns09K?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c1b79552-cda1-4eee-0fe2-08da90f70c2f
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2022 17:33:16.5203
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0mE42ZkUmFcJi35sj9tWOVfqATs4/Pwz1bTOEs5KUp34vr23nSiY4RsLABof05NF
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5687
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Sep 07, 2022 at 10:54:54AM +0100, Jean-Philippe Brucker wrote:

> Is iommu_domain still going to represent both a device context (whole
> PASID table) and individual address spaces, or are you planning to move
> away from that?  What happens when a driver does:
> 
>   d1 = iommu_domain_alloc()
>   iommu_attach_device(d1)
>   d2 = iommu_sva_bind_device()
>   iommu_detach_device(d1)
> 
> Does detach
> (a) only disable the non-PASID address space?
> (b) disable everything?
> (c) fail because the driver didn't unbind first?

I think it must be (a), considering how everything is defined and the
needs for vIOMMU emulation.

If it is any other option then we have a problem of what to do if the
guest VM asks to change the page table associated with the RID while a
PASID is attached.
 
> I'm asking because the SMMU driver is still using smmu_domain to represent
> all address spaces + the non-PASID one, and using the same type
> "iommu_domain" for the new object makes things unreadable. I think
> internally we'll want to use distinct variable names, something like
> "domain" and "address_space". If (a) is not the direction you're going,
> then it may be worth renaming the API as well.
> 
> I'm also not sure why set_dev_pasid() is a domain_ops of the SVA domain,
> but acts on the parent domain which contains the PASID table. Shouldn't it
> be an IOMMU op like remove_dev_pasid(), or on the parent domain?

There is no "parent domain"

PASID or RID+PASID are completely equal concepts for binding.

If you are thinking "parent domain" because SMMU is storing the PASID
table in the RID's iommu_domain, then I think that is a misplacement
in the SMMU driver...

The PASID table belongs in the iommu driver's per-group data
structure. The iommu domain should only have the actual IOPTEs.

Otherwise everything blows up if you attach an iommu_domain to two
RIDs - the API demands that every RID gets its own PASID mapping, even
if the RID shares iommu_domains. We do not have an API to share PASID
tables.

Thus the PASID table is NOT part of the iommu_domain.

The exception will be for nested translation where we will have a
special ARM iommu_domain that contains the PASID table in userspace
memory. When this domain is attached it will logically claim the RID
and every PASID and thus disable the PASID API for that RID.

Remember also that an UNMANAGED iommu_domain should be attachable to
many PASID's and RID's concurrently.

Jason