Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 8 Sep 2022 13:41:43 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        Zhangfei Gao <zhangfei.gao@linaro.org>,
        Zhu Tony <tony.zhu@intel.com>, iommu@lists.linux.dev,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v13 09/13] iommu/sva: Refactoring
 iommu_sva_bind/unbind_device()
Message-ID: <YxobR/brG9VlxPvX@nvidia.com>
References: <20220906124458.46461-1-baolu.lu@linux.intel.com>
 <20220906124458.46461-10-baolu.lu@linux.intel.com>
 <Yxd2+d/VOjdOgrR2@myrica>
 <682d8922-200d-8c89-7142-83e7b3754b8d@linux.intel.com>
 <YxhqbhMmWLeFS512@myrica>
 <YxjV1y/FF0nCI/WO@nvidia.com>
 <YxoXfCQcD3yC5ppn@myrica>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YxoXfCQcD3yC5ppn@myrica>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?DHsVugHe0+BtMBQNLWeD4ley8yFxzOv9Gv4P8HyC34eereYDoWFpdg/1odXh?=
 =?us-ascii?Q?W0TqWXvS8nt5V9nVQZFNHcoYshLr1OX4qtaheMCQ96vythXcGD2XsKhtwKeD?=
 =?us-ascii?Q?RtAqhY17kxFsUG2tIoSROn1F4Rh7HlPV9fBW5DFud1fVOgTFvbHaWkYCU0x5?=
 =?us-ascii?Q?ECisT4S7nLosbX2LC8EuCBg959QQupfeH+BWrE/GXQaVa0r19PBXT5Yw/pHb?=
 =?us-ascii?Q?971iHo8Wmk/TG/XWQY+yLYNSrTHua48D9NHrgqaql/lonnbpb1U/wecVxh3Z?=
 =?us-ascii?Q?WgxmdfZHpUa0nxVu8tgKhpDpOurHxyNLAxTXwaNyFkLJLf/NvYau3QehPRzw?=
 =?us-ascii?Q?ecsxERL43yXILhjjqtKB28WxX2RxUqWmBlOJLOZmdK5dvuwtp3utirI6UkRb?=
 =?us-ascii?Q?yk2Y4xWWjqUNdgt3zuL6OGugcxFyXnxAvzYghMuJYByqmz2CuiQMY31dJZkz?=
 =?us-ascii?Q?+Kjt0MmDc1tSea7dNSHg1ArxSFfHzCTu5jo8H97ql77dmE7/Q1/GpJsHZ9Aq?=
 =?us-ascii?Q?rPZHrVPcNl37dzyvXu7Wq2fJ7xepFtqlKRUqgJt4UD8peRgsG44SEEpjNqIs?=
 =?us-ascii?Q?KcWTRJtuFW27AYeqKyjV8DYY4AusHGJRof0wAWyvSNyRbxb+rcxL+O97JgT0?=
 =?us-ascii?Q?9COKrKFJWxfI812GgoQrTmEB+rn/omGSI2Sw0VeBUWBG0pCfhXqzMJaah0e2?=
 =?us-ascii?Q?wSraJYhShyjtbMOVJAaHcXPkOnpCEdpCxKsJS95LrnUPbvRvuptuQanTqnVW?=
 =?us-ascii?Q?PV6C5ZzBvL+yryjCJDmSJK83YS48J+3ScRyxLJL5sUzmYGLjWhGHxbGTrFse?=
 =?us-ascii?Q?GL1AayLEfHJg9NecAQh2sGbQR+1h940hBd1tbAM6k/0btMk1F7S0IxR++0pU?=
 =?us-ascii?Q?+fWjBn8GzY6s2Mt+QOF2n7SdfTh2P8YBcvc5MgrvHtWj6v9K3v8THS8NIFT+?=
 =?us-ascii?Q?SUZKBmk2XX3rDrFDExQexPlW1vxqt4gvYHLUKsPsxI/T9cQOwiMuL0Z2DHW9?=
 =?us-ascii?Q?L3ZcxU6ft+PMVeeOKArpSbB1XNmNw8kOVuUYq5PgVODlfKT6Ao6znPy1w3UP?=
 =?us-ascii?Q?maWF1Nj2aOEheXITlP67+vBXQ0Z9wqysfMsrXrY8sPi7XALyvV+s7dDlxWHk?=
 =?us-ascii?Q?UDTXbLUbQ0WZJzptf9bsxeVgIFvCm3jY80vbdO/5j/nIdqDQOw2im4NcDreX?=
 =?us-ascii?Q?EW8CHPjsM8AaZzJEUUibkfL8xSfoOf/XfMQHq2wpDNwt0sf526Yxxuz49eVQ?=
 =?us-ascii?Q?+S6KwlTgStBryZKRJjA3qcX0Kc/07AB3Fjp8qfpAlgNdfPQutDwfSD0DrFmP?=
 =?us-ascii?Q?23ox75Km51HJk84RFUmY31dBJ79J3ZXXjqCKughYRSHxuVcHRhZqrCIc0yZp?=
 =?us-ascii?Q?WxVbT4b+yMPx83pAVArgX87+mOMjB3B0HRuwkwoGv9W+2CW3vEgaFItoS8yg?=
 =?us-ascii?Q?gu3STeQUALCZ/O+fhVE+KqmtvYnwRNmj8CE9VBwLxdFK7GkxZU/8Nzmx98Of?=
 =?us-ascii?Q?rkyPEK69eoQIfRNbmdDP4fNrqk1y1MGK0r29HJOETAoAwEK7bnTMVwHACMIV?=
 =?us-ascii?Q?sSzqopkifSbWyTFV3msZLiai+6+XcZCVPDvZNmFO?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: eb4c9828-f1d7-476e-ee8f-08da91b90389
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2022 16:41:44.2725
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sbyGZ3M/NNxnWa/tumbV75WpaJZNz6ggCrwIyzhyDUeDIXtVaGHruDe9QNv1Oa4s
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5469
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 08, 2022 at 05:25:32PM +0100, Jean-Philippe Brucker wrote:
> On Wed, Sep 07, 2022 at 02:33:11PM -0300, Jason Gunthorpe wrote:
> > On Wed, Sep 07, 2022 at 10:54:54AM +0100, Jean-Philippe Brucker wrote:
> > 
> > > Is iommu_domain still going to represent both a device context (whole
> > > PASID table) and individual address spaces, or are you planning to move
> > > away from that?  What happens when a driver does:
> > > 
> > >   d1 = iommu_domain_alloc()
> > >   iommu_attach_device(d1)
> > >   d2 = iommu_sva_bind_device()
> > >   iommu_detach_device(d1)
> > > 
> > > Does detach
> > > (a) only disable the non-PASID address space?
> > > (b) disable everything?
> > > (c) fail because the driver didn't unbind first?
> > 
> > I think it must be (a), considering how everything is defined and the
> > needs for vIOMMU emulation.
> 
> Yes (a) is probably better. The SMMU driver currently implements (c) to
> ensure that you can't switch device driver without unbinding everything
> first, and we should keep that check somewhere

Yes, the owner stuff is a logical place to put that, when ownership
is all released the PASID table of the group must be empty. Lu?

> > Otherwise everything blows up if you attach an iommu_domain to two
> > RIDs - the API demands that every RID gets its own PASID mapping, even
> > if the RID shares iommu_domains. We do not have an API to share PASID
> > tables.
> 
> Well, we still do since SMMU implements it. Changing the API is fine, but
> someone will need to rework the SMMU driver to align with the new meaning
> of iommu_domain. I can take a stab if no one volunteers but probably not
> before next year.

I think given the current progression it is OK for this series to
leave the SMMU driver with a bit of a different view of how the API
should work. The main target here is the in-kernel drivers and they
won't trigger these cases.

As we extend things into userspace PASID support we may want to block
SMMU PASID support through iommfd/vfio until it is updated.

I've been thinking on and off about this - we likely need to have a
conformance test suite as part of iommufd that is specifically for
checking that the iommu driver meets the API. So it would do things
like exercise scenario (a), attaching domains to all sorts of
combinations, and so on.

We know this from other driver subsystems that drivers can implement
the uAPI incorrectly and it causes a lot of pain for userspace.

Jason