Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "hch@lst.de" <hch@lst.de>,
        Jean-Philippe Brucker <jean-philippe@linaro.org>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "parav@mellanox.com" <parav@mellanox.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "Tian, Jun J" <jun.j.tian@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>
Subject: RE: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Topic: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Index: AQHXrSGNbNtRgavabUSKJjvt8l12BauwlhaAgAAouwCAACufAIAAgyUAgAAUFACAAAdkgIAAB/8AgAkqbYCAAEeFAIABQYSQ
Date:   Thu, 30 Sep 2021 08:30:42 +0000
Message-ID: <BN9PR11MB5433C9B5A0CD0B58163859EC8CAA9@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-11-yi.l.liu@intel.com>
 <20210922152407.1bfa6ff7.alex.williamson@redhat.com>
 <20210922234954.GB964074@nvidia.com>
 <BN9PR11MB543333AD3C81312115686AAA8CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YUxTvCt1mYDntO8z@myrica> <20210923112716.GE964074@nvidia.com>
 <BN9PR11MB5433BCFCF3B0CB657E9BFE898CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923122220.GL964074@nvidia.com>
 <BN9PR11MB5433D75C09C6FDA01C2B7CF48CA99@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210929123630.GS964074@nvidia.com>
In-Reply-To: <20210929123630.GS964074@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?CztRSlScnZZXTdPM5vqtYaDCn4tGY+MoabF6eNa1Sei1I/kIj14d8aMyySHW?=
 =?us-ascii?Q?6yQzzd1yxoYyjiw2MP6KoMQ4dk6FE9oSHaCqzXW7hSSvBFWSB3DxVQWHdFaS?=
 =?us-ascii?Q?cmHGNv+zwcA6yYGgqWpP+EEVn0GRxF1kDvd6SQiP2Eiujvrmua8J8LB7Hpln?=
 =?us-ascii?Q?XLrtrG27pmwVNaPhzD7dOCUtiWuneNvQW6+5YTAtLib4pl9vfPQa/DLVt21S?=
 =?us-ascii?Q?FWIIYAW9LA6lJxanO1RRvqhOjNkM66G+gyPmRFjOVW35jGfr63HJb/Pd/bXr?=
 =?us-ascii?Q?sR0B/ywdUZbam+Imq3TM5gUK+uuzC8sqjch6CsnkRAOs66PF0sKCR5sHLyiN?=
 =?us-ascii?Q?0CDvGrJbpxjhK3vSumtWDAGArjSePVxjUddik4r/7oQcSnKTWBC1wI4GPtpU?=
 =?us-ascii?Q?WI3aOJrh2zE3hDUod7nIrj4V7+Y/qyQzAAhrycwndD0otggVWyYVlesShdmP?=
 =?us-ascii?Q?1BskHSucre4CJwrbXMHSjVU2U3vH4pI7uRJrpk3PlnkDFnY2LbVrJJJaQmYA?=
 =?us-ascii?Q?t+aD/TaLVMKnu38ZeBTP9NQTq17Y5ZW99F/py6GDbSHV5TFliCS3alxcIAuA?=
 =?us-ascii?Q?DFavze3Bk81ycxPQj6Ti37CsTSWSiqZxWel/USiCizbj1c0kQKTgwA6VoyQc?=
 =?us-ascii?Q?jex3UCm3Im+CDIWHy0s4qY0YRjI/uwLtLicj6bT7UOJzdAZQeMixqqyDaJR5?=
 =?us-ascii?Q?YP0ooRm0H9K9o1aBJVXY61czGRYrTO38mKrBZnlOUczSMxdW09IJrl+q1ERw?=
 =?us-ascii?Q?88lfrUHklsFZXH77f0efCiCqWXl+WLKiqt7OfnyNJ5txe/cyVDH9H8DfCiyy?=
 =?us-ascii?Q?jyJNielcsRRAYUjHWsOs9V36D2769CARHYRObiTpKtCMAA0ZiZ5z9e/VrOYJ?=
 =?us-ascii?Q?gBzARdb+eu/eubKS5MR1FwMegghPM0GdZeAStZsT6pYWqNgSJaasTlImogOF?=
 =?us-ascii?Q?CqYm7X/H7r6eTXWJjhcxLTVZe1Oko6/OtwHw57QJe/IapHHm9uAR4JDQt7Wc?=
 =?us-ascii?Q?C4w9Z8Z5ex3B/Flpqb2UhX1jT6SI47GSEewo+3wY4AUhRI6k6oTevAbdEpzn?=
 =?us-ascii?Q?Z6+EiZTOGmFkuVoVzuTsrQF3BxkM5Klg+gOiybzNMHhxZ0++1Q9/410FlNh4?=
 =?us-ascii?Q?5jPjIuI4AbqsO9b2kQBiC513+Im5w4WWkXDEIRnU4Zw/f4ueCLTLsbuDwQTF?=
 =?us-ascii?Q?larjkWdHFURKVNyKVhqeG/yxbvs64YEQyFOqejKJykpTDaJwzcHXQGIcsSt+?=
 =?us-ascii?Q?7cEQv2rtlSQbaTrEGC7L+7W6u9sF/+rLiNIplLmnUQR2KZV0r7765ob0jAKi?=
 =?us-ascii?Q?9HAXY26eUKvWW0wS5dNltQTL?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4087b062-7bac-47a0-a7a8-08d983ec9787
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 08:30:42.6609
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: /wAFMQ/7cNayD3zMjnpO0wxf86LvwWbsonVMEWyRYveUsO+Rq2s8gZ9g32u+XDNTECu0Qo59VowG3hqMPDfJgQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1101MB2339
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe
> Sent: Wednesday, September 29, 2021 8:37 PM
>=20
> On Wed, Sep 29, 2021 at 08:48:28AM +0000, Tian, Kevin wrote:
>=20
> > ARM:
> >     - set to snoop format if IOMMU_CACHE
> >     - set to nonsnoop format if !IOMMU_CACHE
> > (in both cases TLP snoop bit is ignored?)
>=20
> Where do you see this? I couldn't even find this functionality in the
> ARM HW manual??

Honestly speaking I'm getting confused by the complex attribute
transformation control (default, replace, combine, input, output, etc.)
in SMMU manual. Above was my impression after last check, but now
I cannot find necessary info to build the same picture (except below=20
code). :/

>=20
> What I saw is ARM linking the IOMMU_CACHE to a IO PTE bit that causes
> the cache coherence to be disabled, which is not ignoring no snoop.

My impression was that snoop is one way of implementing cache
coherency and now since the PTE can explicitly specify cache coherency=20
like below:

                else if (prot & IOMMU_CACHE)
                        pte |=3D ARM_LPAE_PTE_MEMATTR_OIWB;
                else
                        pte |=3D ARM_LPAE_PTE_MEMATTR_NC;

This setting in concept overrides the snoop attribute from the device thus
make it sort of ignored?

But I did see the manual says that:
--
Note: To achieve this 'pull-down' behavior, the No_snoop flag might=20
be carried through the SMMU and used to transform the SMMU output=20
downstream.
--

So again, just got confused here...

>=20
> > I didn't identify the exact commit for above meaning change.
> >
> > Robin, could you help share some thoughts here?
>=20
> It is this:
>=20
> static int dma_info_to_prot(enum dma_data_direction dir, bool coherent,
> 		     unsigned long attrs)
> {
> 	int prot =3D coherent ? IOMMU_CACHE : 0;
>=20
> Which sets IOMMU_CACHE based on:
>=20
> static void *iommu_dma_alloc(struct device *dev, size_t size,
> 		dma_addr_t *handle, gfp_t gfp, unsigned long attrs)
> {
> 	bool coherent =3D dev_is_dma_coherent(dev);
> 	int ioprot =3D dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs);
>=20
> Driving IOMMU_CACHE from dev_is_dma_coherent() has *NOTHING* to do
> with no-snoop TLPs and everything to do with the arch cache
> maintenance API

Maybe I'll get a clearer picture on this after understanding the difference=
=20
between cache coherency and snoop on ARM. They are sort of inter-
changeable on Intel (or possibly on x86 since I just found that AMD=20
completely ignores IOMMU_CACHE).

Thanks
Kevin