Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 1 Oct 2021 09:25:05 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE
Message-ID: <20211001122505.GL964074@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-12-yi.l.liu@intel.com>
 <20210921174438.GW327412@nvidia.com>
 <BN9PR11MB543362CEBDAD02DA9F06D8ED8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210922140911.GT327412@nvidia.com>
 <YVaoamAaqayk1Hja@yekko>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YVaoamAaqayk1Hja@yekko>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NFSSJC8XvtlFI56LfVjiK6LtddS+IUPLpB+GR/QYZ+Da+B/IiIEd2w5y7fw2?=
 =?us-ascii?Q?ptTQKbHU0G3EzOOKKBQy/4qZvoiR6p8zYVo5oLNgMfRyVWeGiLsOLtv9jn67?=
 =?us-ascii?Q?YWKZqw9X3bzpCDSVcoiREXPptJmhsC61hMn+BdtDoI+TFDxjAanhIcNQJ3DT?=
 =?us-ascii?Q?t0Zna15AZJnKBd6RHk36gtCd7uApDXu5pH8i6ieZx8PZFkleP3Ni0CqMnMky?=
 =?us-ascii?Q?hEy8b93Yo59/GjaJZkNUcRkz15AFnFtSgoSIvjIugMsSO9zqfGk82WwDb2JE?=
 =?us-ascii?Q?/A1xw7T5nbrbp3/fiP//NtkoTzih+85t8rCQcJO0HRlwd2T2HF6BleMv+4DG?=
 =?us-ascii?Q?YQH8lomajexEIqtZQADrqyNxVL0RHSPVvoTmoCQO5AnNQVK2Z+2nFnI42KZW?=
 =?us-ascii?Q?w31ZybQGvdxFbnQNExfaXb6u/9CJ4ssuhiIA8MCU8b7D8nNRuPfpA1klFVXC?=
 =?us-ascii?Q?O/UA25HVsmUA+Gclhyl9Kz1vGMt27r4mXchJ7jwo/ViFSaRdeKcwPD8CTAAq?=
 =?us-ascii?Q?Bsz+g56aS9c6VETIpyMZy618gIbBybRwK8+HOp8NyoQ/YuFjzHjMmyRQOXTB?=
 =?us-ascii?Q?UdPFugerepKyzEj7M9ut2uy6dzUkSdTmU7E6S/OjkKjq0lL9+nPR4gATn+VS?=
 =?us-ascii?Q?KTkswbjO7hqtzjRs7OFSFbMNRA5TtJOUs6Y08niBBw3mjmkTfBbe2VJMndWL?=
 =?us-ascii?Q?twTgidLyY50Ng0LE33wl50MDbB9H0Klu0WRIfMavMkyyPVH//vMAFPt2MjC2?=
 =?us-ascii?Q?WxAjpDUYW3gLL463VvCN82WmiajD/BoLK3IL1f8UpB0y8z5rpXaECwqew3j+?=
 =?us-ascii?Q?EZDV2rm3BbTphjmbgO25lFL8+vxAjZrgLYCoNfMARxgeLbB4HhE4DTdF7HYB?=
 =?us-ascii?Q?FUvSRk78QDepM201a7o0doieBhggApZk7ydtODItZum0Lo/UtwW7PLc7qxd7?=
 =?us-ascii?Q?rhkyKIifdUUQh8NxUaFrI1D6M9o29kgbtWAq/KLiesnNLW3h0QXAF1r38fOx?=
 =?us-ascii?Q?gKFlQnj0BK5/u9oPd5V3lpKuQ3m2qLsgy+ijBDLa6lp5RBPYcgKr2kcYcyjo?=
 =?us-ascii?Q?Jxxv2PKJBdQsadxMWUhBVMqCzpECkgjC+6Ir+gg6lG2kniywizYP3nbRLfh8?=
 =?us-ascii?Q?TMqq2zZ3PVTvjnJLJzWRR6OqrUzVJJcfsOvmAJf2x8UCXvQhSuhozUN+9/Qc?=
 =?us-ascii?Q?HZfa+VPSp4BvJMAa8/h4PvJVyJpOZZcTI0U8IpYHsDrefW3OpvXPUCIWP2Lp?=
 =?us-ascii?Q?UCQBHzojAX5F1wHPmtHOxv8ogme3xuhV0bUl9szy2IDPuRPNwNBHkVOea8Zt?=
 =?us-ascii?Q?Tpd0UEuP/Pi4xYso/ASXif7G+LXEBoi/B4WAWHP/usZBXQ=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 125c7018-dba2-49af-751c-08d984d68123
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Oct 2021 12:25:08.0340
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: s46tZC5ax4P1M0BET1v6sUcsqpb3IBiIuf/YkNzIZhGl7bAuXd+iPmHkPTin9VJD
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5048
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Oct 01, 2021 at 04:19:22PM +1000, david@gibson.dropbear.id.au wrote:
> On Wed, Sep 22, 2021 at 11:09:11AM -0300, Jason Gunthorpe wrote:
> > On Wed, Sep 22, 2021 at 03:40:25AM +0000, Tian, Kevin wrote:
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Wednesday, September 22, 2021 1:45 AM
> > > > 
> > > > On Sun, Sep 19, 2021 at 02:38:39PM +0800, Liu Yi L wrote:
> > > > > This patch adds IOASID allocation/free interface per iommufd. When
> > > > > allocating an IOASID, userspace is expected to specify the type and
> > > > > format information for the target I/O page table.
> > > > >
> > > > > This RFC supports only one type (IOMMU_IOASID_TYPE_KERNEL_TYPE1V2),
> > > > > implying a kernel-managed I/O page table with vfio type1v2 mapping
> > > > > semantics. For this type the user should specify the addr_width of
> > > > > the I/O address space and whether the I/O page table is created in
> > > > > an iommu enfore_snoop format. enforce_snoop must be true at this point,
> > > > > as the false setting requires additional contract with KVM on handling
> > > > > WBINVD emulation, which can be added later.
> > > > >
> > > > > Userspace is expected to call IOMMU_CHECK_EXTENSION (see next patch)
> > > > > for what formats can be specified when allocating an IOASID.
> > > > >
> > > > > Open:
> > > > > - Devices on PPC platform currently use a different iommu driver in vfio.
> > > > >   Per previous discussion they can also use vfio type1v2 as long as there
> > > > >   is a way to claim a specific iova range from a system-wide address space.
> > > > >   This requirement doesn't sound PPC specific, as addr_width for pci
> > > > devices
> > > > >   can be also represented by a range [0, 2^addr_width-1]. This RFC hasn't
> > > > >   adopted this design yet. We hope to have formal alignment in v1
> > > > discussion
> > > > >   and then decide how to incorporate it in v2.
> > > > 
> > > > I think the request was to include a start/end IO address hint when
> > > > creating the ios. When the kernel creates it then it can return the
> > > 
> > > is the hint single-range or could be multiple-ranges?
> > 
> > David explained it here:
> > 
> > https://lore.kernel.org/kvm/YMrKksUeNW%2FPEGPM@yekko/
> 
> Apparently not well enough.  I've attempted again in this thread.
> 
> > qeumu needs to be able to chooose if it gets the 32 bit range or 64
> > bit range.
> 
> No. qemu needs to supply *both* the 32-bit and 64-bit range to its
> guest, and therefore needs to request both from the host.

As I understood your remarks each IOAS can only be one of the formats
as they have a different PTE layout. So here I ment that qmeu needs to
be able to pick *for each IOAS* which of the two formats it is.

> Or rather, it *might* need to supply both.  It will supply just the
> 32-bit range by default, but the guest can request the 64-bit range
> and/or remove and resize the 32-bit range via hypercall interfaces.
> Vaguely recent Linux guests certainly will request the 64-bit range in
> addition to the default 32-bit range.

And this would result in two different IOAS objects

Jason