From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        "Jiang, Dave" <dave.jiang@intel.com>
CC:     Jacob Pan <jacob.jun.pan@linux.intel.com>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Joerg Roedel <joro@8bytes.org>,
        "Christoph Hellwig" <hch@infradead.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        "Pan, Jacob jun" <jacob.jun.pan@intel.com>,
        "Lu Baolu" <baolu.lu@linux.intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, Barry Song <21cnbao@gmail.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>
Subject: RE: [PATCH 4/4] dmaengine: idxd: Use DMA API for in-kernel DMA with
 PASID
Thread-Topic: [PATCH 4/4] dmaengine: idxd: Use DMA API for in-kernel DMA with
 PASID
Thread-Index: AQHX67qzSUJug20b70uAfG58CL05gawok3UAgAAnooCAACXYAIAAe6NA
Date:   Thu, 9 Dec 2021 01:48:09 +0000
Message-ID: <BN9PR11MB527617F8CC015E4F15EDCC9F8C709@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <1638884834-83028-1-git-send-email-jacob.jun.pan@linux.intel.com>
 <1638884834-83028-5-git-send-email-jacob.jun.pan@linux.intel.com>
 <20211208131358.GR6385@nvidia.com>
 <1ffc1366-2711-3026-fb09-8f60a260f618@intel.com>
 <20211208175116.GC6385@nvidia.com>
In-Reply-To: <20211208175116.GC6385@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?eXV9FWNhNXEMTXNV/IRi8NIuJpCPkimLTzmLQq4QZH+nADNmU2EQj/uBWxCC?=
 =?us-ascii?Q?rb8/YL3LJmbfv1v6LwmCigbGgXl5DXQ3eL1YyKL2EDFG4sBfFv+Vj49HC61N?=
 =?us-ascii?Q?wDFNgz1vsgfmR03myBg2rXLSynAt18k1VepnJrlPfEjKhG8OQll0xqZhw6s1?=
 =?us-ascii?Q?CVOMMWL66ahZeo5Tt3v+gTGiSwSvT5yqDGmzu7lhqVCCdkbq+4r2bvc4wngl?=
 =?us-ascii?Q?MP7+bn/5JoSJKNO9QXUdkEIz8kQnyhKfcK5ZZCWTpNmn4y4zOrpGN64d8ER3?=
 =?us-ascii?Q?1VtMTqL/G49EOKgX/rVVK+Z3ZMf2XQPefxc9dp5n/a4wByb6J8MUpQNA43st?=
 =?us-ascii?Q?0ykqQQrxVzH63UjqJZ1CUGlUiLbkynCMCk3V/Xfkksm1Hw/CwnKMl3z6N3OA?=
 =?us-ascii?Q?md3/L8zgegXo7XwgenwgyvgA71aFHEy3ZkEkbatlmlThGRGPRPFGw7XPgLH2?=
 =?us-ascii?Q?oTphJMSeOBItLLI1Cgzq2CEbS6w3Kcee1JdKPZ4mp+IPIz2EmjV+ZWLYKMqC?=
 =?us-ascii?Q?kupiofr06f8ZGIREuL8oyuXRr9r2A4ey27feHwa/1awJn9d4FAitq6E2T2qn?=
 =?us-ascii?Q?EQ3MbMndWaGz4LJQwI4dpLj4RlTUoUP4kPpSaEVMjk8vbjxECPozAWNt376R?=
 =?us-ascii?Q?urW34adURLjG0mm1kTh14+5yaLvZpFma/B7spgE9T+qToW/+RBBrYboXN0n+?=
 =?us-ascii?Q?mhq9a3C/1CibPk7GwS6bjWkh7z66oPZsURiO0EyqFv0wCBaDKggYQpAWkqmv?=
 =?us-ascii?Q?eBmMBhOAFI/crNfq3/Q7KkYbgP+0oMXHCUxLCMAcd1NN6ir5UWHriR1JATWj?=
 =?us-ascii?Q?cL/9GpmjVFhCc63XlVjL2idqAMuGelQXX5deyNAFf1w5iQbrJZ+Q3lLSsbfF?=
 =?us-ascii?Q?NWo/efgOw58ik/j1951FxKqWoJe6Ep56Hx8COrlFxoE/aD0KxWnq0Cos6i1O?=
 =?us-ascii?Q?1k+ogDx2Unh7fdGy9YhaIC1iCEd6XmL7UuCPfF5HrQ1+D3eTg2SJEomV6gSU?=
 =?us-ascii?Q?XrLqYmH+TasokMbL35ss/kQa5zlLuyb5qrob6YtnUkwXx4E0qbzZ0yPc2lpD?=
 =?us-ascii?Q?GHg56ReulxXRy5HMPMf0SlE4kBfUVP68gBzXoU+Vi/1jG0JhaqThPvRZnWk/?=
 =?us-ascii?Q?G/rgyh4/4mr/1NPBi+JBdMwEfRhRvclSfq+r5eqaWaFAFBvkUpJH7s3t2UyR?=
 =?us-ascii?Q?eDScI+xkbo9q7UXdHAvy7FUGNiU8yBC5HUSQtzWKKoqultPlZfVoCI+YFZeM?=
 =?us-ascii?Q?97QnefY7hf5G5L1azl8CLGNAURXibinm6/H6Yk2KmXWfzWKo3p1hh6qBmbN8?=
 =?us-ascii?Q?wYb8eJXxdkA1RLXqKdkUvEE5OOqcOtCYot9HqfKs774XxGpwxWS3J3LGWpjO?=
 =?us-ascii?Q?NyZT3xkE97TGVJt3gr4OVjZ8UWsgKWroYsu0DCTzuxpkjrpSim7iYMtWIHFB?=
 =?us-ascii?Q?JOv81jxubevDB5fVDxnnDcREnrHLJy6KuR6tKJXjX/7ifKVTnu3r8jwNmKv1?=
 =?us-ascii?Q?g+fIqRng49fXH1qqO1gvfh/yDbNTbSlDfn9j/ES5NfDs235Gvn7QcFSEU8kJ?=
 =?us-ascii?Q?fgky9PjW3zB7IyP22sUay5FtbLw6JFomb5O2zJZMfCUqu7bAtprtM514MX4y?=
 =?us-ascii?Q?YXZ4CKqjN/MsqC0iccq8jK4=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bf187dcd-d786-4b04-7500-08d9bab5f433
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Dec 2021 01:48:09.7002
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: mqlh/sXRexRg9Y9QOjzOcQG7GOzdi5D4ZO1izT/XIb0EEKDOxogytDgs8mr5StZIdhVL0nLkwNkAdvNtKhFo9A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3714
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, December 9, 2021 1:51 AM
>=20
> > > > +	/*
> > > > +	 * Try to enable both in-kernel and user DMA request with PASID.
> > > > +	 * PASID is supported unless both user and kernel PASID are
> > > > +	 * supported. Do not fail probe here in that idxd can still be
> > > > +	 * used w/o PASID or IOMMU.
> > > > +	 */
> > > > +	if (iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_SVA) ||
> > > > +		idxd_enable_system_pasid(idxd)) {
> > > > +		dev_warn(dev, "Failed to enable PASID\n");
> > > > +	} else {
> > > > +		set_bit(IDXD_FLAG_PASID_ENABLED, &idxd->flags);
> > > >   	}
> > > Huh? How can the driver keep going if PASID isn't supported? I though=
t
> > > the whole point of this was because the device cannot do DMA without
> > > PASID at all?
> >
> > There are 2 types of WQ supported with the DSA devices. A dedicated WQ
> type
> > and a shared WQ type. The dedicated WQ type can support DMA with and
> without
> > PASID. The shared wq type must have a PASID to operate. The driver can
> > support dedicated WQ only without PASID usage when there is no PASID
> > support.
>=20
> Can you add to the cover letter why does the kernel require to use the
> shared WQ?
>=20
> Jason

Two reasons:

On native the shared WQ is useful when the kernel wants to offload
some memory operations (e.g. page-zeroing) to DSA. When #CPUs are
more than #WQs, this allows per-cpu lock-less submissions using
ENQCMD(PASID, payload) instruction.

In guest the virtual DSA HW may only contain a WQ in shared mode
(unchangeable by the guest) when the host admin wants to share
the limited WQ resource among many VMs. Then there is no choice
in guest regardless whether it's for user or kernel controlled DMA.

Thanks
Kevin