Date:   Wed, 8 Dec 2021 09:10:38 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc:     iommu@lists.linux-foundation.org,
        LKML <linux-kernel@vger.kernel.org>,
        Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Jacob Pan <jacob.jun.pan@intel.com>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        Raj Ashok <ashok.raj@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Tony Luck <tony.luck@intel.com>, Yi Liu <yi.l.liu@intel.com>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        Barry Song <21cnbao@gmail.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>,
        Dan Williams <dan.j.williams@intel.com>
Subject: Re: [PATCH 0/4] Enable PASID for DMA API users
Message-ID: <20211208131038.GQ6385@nvidia.com>
References: <1638884834-83028-1-git-send-email-jacob.jun.pan@linux.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1638884834-83028-1-git-send-email-jacob.jun.pan@linux.intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1uEWws27wVZr2l2JielrxH+uqfcekB3XniFSdi6kEzDmsr0SlnQFbLw3IEGM?=
 =?us-ascii?Q?8QT3afHU6oUMxpK21bUMIadnL/5yViyGTZoin9N9xG84U/eHhqgvP4QDpKxo?=
 =?us-ascii?Q?7JxXPZx3FeHSI0ZWaWa50ji6pApG8F1Ug6j2Fc/nw9ZBSwmXqupOOrpKGMxh?=
 =?us-ascii?Q?Rk5r6oy4NqTzHd/V0Vw1mCebc0pEbd1TqxoTrjyqWV6yWCGaDMGR7bqX7nZI?=
 =?us-ascii?Q?+2quz83TwJu9VBKjBay9ROE6G7V8nbCdCxCHLRBZ5URwCXt2qgJ129ob66kj?=
 =?us-ascii?Q?EWgjiQbyY+aldzLahGDoFxCHEDLCEsDny+r/akQaL+q3im6zRrQfuv1vrHTG?=
 =?us-ascii?Q?HY+Lc4Jf2uUpj2j6nLJO5bXQjFfOSvg4a6BV4yoAkmB5URTk4krA5lZlHbY7?=
 =?us-ascii?Q?COftDO165uhdoc4XhUE2mYMvEhe9kwzdC08TgtJpKJceKYYb4lYF6CgckKHa?=
 =?us-ascii?Q?C916i66kw03VW91BFCYLnqUQIrzTxEW3On3GBP/b1vWKzIIx9QtIvmC5wFXd?=
 =?us-ascii?Q?lRRxRquSWKH+OLPTQeaAs0CEhi4FP66WFZj02ycdqB7wnuMVHdcDNt66KUgT?=
 =?us-ascii?Q?mf/q4uqlV+GPe0Kg+2QxQKl0N0DuENCYOzpexVGlNlGjorj5m5QnvYu5kJ0V?=
 =?us-ascii?Q?361bclur6Vk1M3JbTDK3Zfn9RcHlbK0rsuD5UDAgZvaga+skypKNhAW7r/uL?=
 =?us-ascii?Q?fMrHPUu2OjWD4t29EaSqUWdd2n5mJ2rY6+5ciL99mUneRemgKT49v+H+LtFc?=
 =?us-ascii?Q?Z/zUx3oFz8aZ6F6K05BBEeYCCp83Jt5GA2nI3XjXDoZ1krP+0IgDQ/+1m2De?=
 =?us-ascii?Q?tn2XI1SxzIUNZBWmLZ5/iTG2ySfBF+mwsDDkiSTPg4NKn0JVWTONtiUpPr3Q?=
 =?us-ascii?Q?8SSj5uSHx7PP53kypfRdL7oshg8jEU3IrrrPORXno6ltk/LsOhvVyweYoyMs?=
 =?us-ascii?Q?zdebJZa8yZPd6E/1HI9L6t4Ve68W4Ln4jt6RrE+xlZ/PYVT9gWT1EWsc4Lio?=
 =?us-ascii?Q?Rad3e+hyKFl+RLGRIMo/vyE7YGj2ceNoaUsqn1AYUxuTq0EReAKGYK50K1Lt?=
 =?us-ascii?Q?4N616T8CjVRvNLXIVSFZfh8Op1jvTh4Lo1mVGpFfVfmNtFiEuED6GVCIc6+d?=
 =?us-ascii?Q?Iw2cbvH3WyeQ1MardvDDmmwfp4CDnQwdxMSDlkMrreqmCRezL049wIXaaLyf?=
 =?us-ascii?Q?qfBZ1SHm9f6zsL4aqISZepaYLc0w+0ZE3yWENI2iASYgZAX+x/y96P/5a1RZ?=
 =?us-ascii?Q?nofXKcihk8oJv/NUvxsBFBoHybUvf9bwEKzoa5d8AXYCUZGYC2dJqo8AYHwf?=
 =?us-ascii?Q?OPAn3Ala7lV5vHUK/rbwj+TNZTDLtPNFvNkIky/aAlMZGRHTUbhar8o53YlX?=
 =?us-ascii?Q?hr7K07ih093aPgjqkjhZ6t1u3N51Nz2mAWGOG/RdjhAv1qVwvCK69+vhlvZT?=
 =?us-ascii?Q?1QNiRo/vl6pkyk4M3X+Pyu+KeOvP+qGiAA/8IUqe2LnagyDQs8JVImbrxxMT?=
 =?us-ascii?Q?Kn45QoRPoOLFOPIWXMozJazZTv/y47WE0AdbGhyePt4RMWP3bDmraf1+j08R?=
 =?us-ascii?Q?NS7HArWOhd8o21vZx8w=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 316d7ef9-c1d8-42b0-0bbc-08d9ba4c2202
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2021 13:10:40.1413
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uKISWEwsqCjlke6loRYBZE7GClDJz6j0MBJuJY2aiqSNtvB8qGVoGVSW1UQTCOdk
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5334
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Dec 07, 2021 at 05:47:10AM -0800, Jacob Pan wrote:
> Modern accelerators such as Intel's Data Streaming Accelerator (DSA) can
> perform DMA requests with PASID, which is a finer granularity than the
> device's requester ID(RID). In fact, work submissions on DSA shared work
> queues require PASID.

Lets use plain langauge please:

DSA HW cannot do DMA from its RID, so always requires a PASID, even
for kernel controlled DMA.

To allow it to use the DMA API we must associate a PASID with the
iommu_domain that the DMA API is already using for the device's RID.

This way DMA tagged with the PASID will be treated exactly the same as
DMA originating from the RID.

> DMA mapping API is the de facto standard for in-kernel DMA. However, it
> operates on a per device/RID basis which is not PASID-aware.
> 
> This patch introduces the following driver facing API that enables DMA API
> PASID usage: ioasid_t iommu_enable_pasid_dma(struct device *dev);

This is the wrong API, IMHO

It should be more like

int iommu_get_dma_api_pasid(struct device *dev, ioasid_t *pasid);
void iommu_destroy_dma_api_pasid(struct device *dev);

> A PASID field is added to struct device for the purposes of storing kernel
> DMA PASID and flushing device IOTLBs. A separate use case in interrupt

And this really should not be touching the struct device at all.

At worst the PASID should be stored in the iommu_group.

> message store (IMS) also hinted adding a PASID field to struct device.
> https://lore.kernel.org/all/87pmx73tfw.ffs@nanos.tec.linutronix.de/
> IMS virtualization and DMA API does not overlap.

This is under debate, I'm skeptical it will happen considering the new
direction for this work.

> Once enabled, device drivers can continue to use DMA APIs as-is. There is
> no difference in terms of mapping in dma_handle between without PASID and
> with PASID.  The DMA mapping performed by IOMMU will be identical for both
> requests with and without PASID (legacy), let it be IOVA or PA in case of
> pass-through.

In other words all this does is connect the PASID to the normal
DMA-API owned iommu_domain.

Jason