Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 14 Oct 2021 12:06:10 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Message-ID: <20211014150610.GS2744544@nvidia.com>
References: <20210921174438.GW327412@nvidia.com>
 <BN9PR11MB543362CEBDAD02DA9F06D8ED8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210922140911.GT327412@nvidia.com>
 <YVaoamAaqayk1Hja@yekko>
 <20211001122505.GL964074@nvidia.com>
 <YVfeUkW7PWQeYFJQ@yekko>
 <20211002122542.GW964074@nvidia.com>
 <YWPNoknkNW55KQM4@yekko>
 <20211011171748.GA92207@nvidia.com>
 <YWezEY+CJBRY7uLj@yekko>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YWezEY+CJBRY7uLj@yekko>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uI5CnTu30fEyEe3vzb0hJJ9jNsRk3TEV2Ph/P0vjFLqiNF//kNNO2/s6N2+G?=
 =?us-ascii?Q?/fIxWkzdUSlobA/V9hHGsWNgEc0hooFga2qGm4nkxaHvin6B3YWpOLltoUf6?=
 =?us-ascii?Q?3rd1vVtqTEk7+pSNP8ikdz5aUiF+nGUpgq8nVT07DozzaNfmZy/lZG7Ss/1H?=
 =?us-ascii?Q?5Y2FJa8Vl/ryJUYaaiZvSH5/u+moYvRiiPxGnP1PxTfr2gkcRtPy893QZ9FI?=
 =?us-ascii?Q?DaSXk0vDeUBv1TC/5icjA10tU56wLyAbQhR2/oVocnlByLderSJjsd8GkscK?=
 =?us-ascii?Q?yAJGm4ekazG23U6iIJ5WIQmcE4Iji9p+8dfYDam40UKXveVUa6ctuXj5kLgp?=
 =?us-ascii?Q?QT+OpaUmksDxjPwZoj9iBWotLjXuv7yp7UT0D3TDicJ579SF5iLzJAac5feZ?=
 =?us-ascii?Q?GhJq00dimVoTjmXlKLM0H5VHQXJqswJaVWkUZtcvfoQK7FnSLEgJHQ4KZZkJ?=
 =?us-ascii?Q?auHI4p+kWu0PWOOQgvaHkiuieTsYFkG5U+rQ3knvHvrd7yrNkLJIYi0Cd/pO?=
 =?us-ascii?Q?huoJ50zZfn+MFdnncz+neNMkukcS4XZyxowYxq+qKLjep0YOi+QhaLoT/CoY?=
 =?us-ascii?Q?YsvzOSVbrkDW5k/4+/rkwdkFOHvd/3jfWGgo/DuTYH8iyD/RINr0YUQ21G5+?=
 =?us-ascii?Q?Bp2dmG3TgUxGttHougSGzipG4NDmF+Qnja8tSsM+JwttqyqSHknBCb9F4olz?=
 =?us-ascii?Q?mI9nyshVDNdfoLgr0hFIvwHluplT9MryJXm8O3UqiEflRg/giCvVwNt9a+f7?=
 =?us-ascii?Q?zWko7fP73TERrtzZ27s4KRxddzNW9JaEUW6A9PKEHIBWv1RSjgJfQSmJ8rDF?=
 =?us-ascii?Q?0MqPxmgUQR0PPbEQVVUnzIoWjcmE+3quGqp7LtaB1UUtZFlctgyqXWZpCBnu?=
 =?us-ascii?Q?5u0G2d2vsEq/tznTaJ81TZS1bXiZlZZqVA+j4BdczX0X6aSF4qJV6+ButTZR?=
 =?us-ascii?Q?/CxOlx7APWj98GK9NUORVQj9yJZl26bj9p+ylDyWcetngZx+ejwlLiKW7dk3?=
 =?us-ascii?Q?AGth2b+j6J9T7LdB0bGDO2KexPDNoCxPCHsJ4mrNOtq5QhbzVyf5+zfboH0K?=
 =?us-ascii?Q?Vab7zY5BgvMa9N/hdomabOZ9VUcEi/mF2bsiCDnsLtvhR+TbRqNh6VBPbQx2?=
 =?us-ascii?Q?/Z50XSL/UAkAJbSZbRJWYE63hlbWjVRzi0+bsxMAeIYFvcyJfgDzA6Y5JAHp?=
 =?us-ascii?Q?DmU3c0DVM5T+yprujizivKAKIrBBOoR26V2MeyhxFfvf6b7clnlgPphX8l3e?=
 =?us-ascii?Q?Dj77+uhs+PNiJnI0lphIYPsbs8oJt/yZERhE7PLpcWhSY2K+/9HVn9DlD2yZ?=
 =?us-ascii?Q?erjp/9XkuR6i1sRugs6miC+co6SI/IoRJGUMAms0uSa9Og=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3539ff5f-53bb-45a6-2ec0-08d98f2428a8
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2021 15:06:11.4924
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: myak+zlqHBr6n9wGoRDwjTW/dvsAxDuJpIffRF+sVhBv1hzUWyIIID0G2rR2Dibh
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5286
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Oct 14, 2021 at 03:33:21PM +1100, david@gibson.dropbear.id.au wrote:

> > If the HW can attach multiple non-overlapping IOAS's to the same
> > device then the HW is routing to the correct IOAS by using the address
> > bits. This is not much different from the prior discussion we had
> > where we were thinking of the PASID as an 80 bit address
> 
> Ah... that might be a workable approach.  And it even helps me get my
> head around multiple attachment which I was struggling with before.
> 
> So, the rule would be that you can attach multiple IOASes to a device,
> as long as none of them overlap.  The non-overlapping could be because
> each IOAS covers a disjoint address range, or it could be because
> there's some attached information - such as a PASID - to disambiguate.

Right exactly - it is very parallel to PASID

And obviously HW support is required to have multiple page table
pointers per RID - which sounds like PPC does (high/low pointer?)
 
> What remains a question is where the disambiguating information comes
> from in each case: does it come from properties of the IOAS,
> propertues of the device, or from extra parameters supplied at attach
> time.  IIUC, the current draft suggests it always comes at attach time
> for the PASID information.  Obviously the more consistency we can have
> here the better.

From a generic view point I'd say all are fair game. It is up to the
IOMMU driver to take the requested set of IOAS's, the "at attachment"
information (like PASID) and decide what to do, or fail.

> I can also see an additional problem in implementation, once we start
> looking at hot-adding devices to existing address spaces.  

I won't pretend to guess how to implement this :) Just from a modeling
perspective is something that works logically. If the kernel
implementation is too hard then PPC should do one of the other ideas.

Personally I'd probably try for a nice multi-domain attachment model
like PASID and not try to create/destroy domains.

As I said in my last email I think it is up to each IOMMU HW driver to
make these decisions, the iommufd framework just provides a
standardized API toward the attaching driver that the IOMMU HW must
fit into.

Jason