Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Alex Williamson <alex.williamson@redhat.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Topic: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Index: AQHXrSGNbNtRgavabUSKJjvt8l12BauwlhaAgAAouwCAACufAA==
Date:   Thu, 23 Sep 2021 03:10:47 +0000
Message-ID: <BN9PR11MB543333AD3C81312115686AAA8CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-11-yi.l.liu@intel.com>
 <20210922152407.1bfa6ff7.alex.williamson@redhat.com>
 <20210922234954.GB964074@nvidia.com>
In-Reply-To: <20210922234954.GB964074@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?t4Y2S2J3xYq/jtYWyj8zzjGxTVx6iVukXWsogkjeebROXAyvexf38Xjcwy12?=
 =?us-ascii?Q?xsnoB7QKMpqupWj0o/BjKpg1MsCVWg8ew5N/1JWtAU2l32PRPJk8VbHOjlI2?=
 =?us-ascii?Q?flpw5UiGU1hu0LReesCmQgMaNSUNx2dVqS5biLWj3whuxwFEkcMgWz6MGZ4K?=
 =?us-ascii?Q?MFbEXgK28cvHvXutLrN5S1JfSG5joUcXbiBU+EsX+/nEzaTmJ/0jW0fg8PdG?=
 =?us-ascii?Q?5gdVXUONc2Lf1ekjil21l5DENc0wVEDjUMdlfQOGlyJpLTAQu3C7GwYFSK+7?=
 =?us-ascii?Q?msj16DFCxAne14kx8men5517xHbu8Z0En2kgHVdHasjxD/VVQSc2Dpeo7jc8?=
 =?us-ascii?Q?YmivX3scIUrAKKjVhlpd7kA+a+xs3S7BScbDrHKVtjV0vzv/xhhOLxH9Knrf?=
 =?us-ascii?Q?U9OE+pIy5/cDujFrLt7X4ocHKevHXUB1Vjfg7g2vpxf8FLM5u1wLMN2KwVZi?=
 =?us-ascii?Q?b/b5jwPY1IFzXY/iG9Kfg2UAUk022Sl8Gmm++oKV+G8pKBW993a+dxvis+0m?=
 =?us-ascii?Q?z9/I1VQj689fb9gUnCD6SNYROiY5YF2VN13iVKz1FAwfrcYFuFck9fDDDQMo?=
 =?us-ascii?Q?dG9TYxokTwUIwWyjIpJcH7Kyt7o+DjPdN8il+uRkCRh9zem8qKNW1rYhxXgm?=
 =?us-ascii?Q?fzcQI8iQwpBjAz6dZZCWjlL3RwatADHrjI4LEyBqmbav1kFKoPwv489RJCh8?=
 =?us-ascii?Q?9Q9pVXNc2MRLN9dzxuqZw2iNAUhP3Dzi2zkJVkkWnrDci98jOgmlEd80Uv+l?=
 =?us-ascii?Q?JQyOCsuLov0DkKrkSs74P+5qR1uwPuGxabNZyQyazJI8fLQJ/LpZ7TIJ4hGM?=
 =?us-ascii?Q?dhntBYlj7UtP54b9GGp1Fq+8eyYqMKkxCLVwwNStVR20Psls3tVjtgRFFZ+t?=
 =?us-ascii?Q?yInvINIUWZ2m6hL270OouHG+dV6dn5HKFk3cEP6ZyEgn/Ft6bkLXQFOIIvfV?=
 =?us-ascii?Q?8641o4LFL0iVJqVUjts7CJDhzDHYHUVb4EVXVvYRgL1/4QDkx8xg4epDue7F?=
 =?us-ascii?Q?vNxP2Qs6OrRA+3WRDVPmPqHrCGmG/ShzgLHyKfIs735MaYfo21372/6zkGtr?=
 =?us-ascii?Q?2fKLauRSlpz/IX4/6VGBfwW/B1JHgoYBB7wvJh3SwvOKQ8OKekcMEORljXox?=
 =?us-ascii?Q?ZHo0j4+5SnOnNk77jkFRT+sf8wSdTdFj1x8nrFPwt9dwRZgFivK4I91Td8Uu?=
 =?us-ascii?Q?SvCDf0YCDXhVyGjbesoYOpRLs0pNVrbrcvfGWsCf+evbwingrLhc3ynjbkw0?=
 =?us-ascii?Q?78jjXusbdPaWyaXYhMKUNytH2gf8UfJ8zTk1BluTCFfN1C9EuoqcRNW1D8nD?=
 =?us-ascii?Q?p489wLsL/P4ij8vRSeWa1KNc?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 13d75e3c-e803-423d-dd47-08d97e3fbd72
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2021 03:10:47.3986
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +x/j0gqX2g1RNmv5GqM70AcjSHUtLy/etrVfFsUSSHk++3p2hsXCr/9EFXr3EUU7GqO4nqF4uWhc1RTYtD/Xuw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1617
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, September 23, 2021 7:50 AM
>=20
> On Wed, Sep 22, 2021 at 03:24:07PM -0600, Alex Williamson wrote:
> > On Sun, 19 Sep 2021 14:38:38 +0800
> > Liu Yi L <yi.l.liu@intel.com> wrote:
> >
> > > +struct iommu_device_info {
> > > +	__u32	argsz;
> > > +	__u32	flags;
> > > +#define IOMMU_DEVICE_INFO_ENFORCE_SNOOP	(1 << 0) /* IOMMU
> enforced snoop */
> >
> > Is this too PCI specific, or perhaps too much of the mechanism rather

Isn't snoop vs. !snoop a general concept not pci specific?

> > than the result?  ie. should we just indicate if the IOMMU guarantees
> > coherent DMA?  Thanks,
>=20
> I think the name of "coherent DMA" for this feature inside the kernel
> is very, very confusing. We already have something called coherent dma
> and this usage on Intel has nothing at all to do with that.
>=20
> In fact it looks like this confusing name has already caused
> implementation problems as I see dma-iommu, is connecting
> dev->dma_coherent to IOMMU_CACHE! eg in dma_info_to_prot(). This is
> completely wrong if IOMMU_CACHE is linked to no_snoop.
>=20
> And ARM seems to have fallen out of step with x86 as the ARM IOMMU
> drivers are mapping IOMMU_CACHE to ARM_LPAE_PTE_MEMATTR_OIWB,
> ARM_LPAE_MAIR_ATTR_IDX_CACHE
>=20
> The SMMU spec for ARMv8 is pretty clear:
>=20
>  13.6.1.1 No_snoop
>=20
>  Support for No_snoop is system-dependent and, if implemented, No_snoop
>  transforms a final access attribute of a Normal cacheable type to
>  Normal-iNC-oNC-OSH downstream of (or appearing to be performed
>  downstream of) the SMMU. No_snoop does not transform a final access
>  attribute of any-Device.
>=20
> Meaning setting ARM_LPAE_MAIR_ATTR_IDX_CACHE from IOMMU_CACHE
> does NOT
> block non-snoop, in fact it *enables* it - the reverse of what Intel
> is doing!

Checking the code:

        if (data->iop.fmt =3D=3D ARM_64_LPAE_S2 ||
            data->iop.fmt =3D=3D ARM_32_LPAE_S2) {
                if (prot & IOMMU_MMIO)
                        pte |=3D ARM_LPAE_PTE_MEMATTR_DEV;
                else if (prot & IOMMU_CACHE)
                        pte |=3D ARM_LPAE_PTE_MEMATTR_OIWB;
                else
                        pte |=3D ARM_LPAE_PTE_MEMATTR_NC;

It does set attribute to WB for IOMMU_CACHE and then NC (Non-cacheable)
for !IOMMU_CACHE. The main difference between Intel and ARM is that Intel
by default allows both snoop and non-snoop traffic with one additional bit
to enforce snoop, while ARM requires explicit SMMU configuration for snoop
and non-snoop respectively.

        } else {
                if (prot & IOMMU_MMIO)
                        pte |=3D (ARM_LPAE_MAIR_ATTR_IDX_DEV
                                << ARM_LPAE_PTE_ATTRINDX_SHIFT);
                else if (prot & IOMMU_CACHE)
                        pte |=3D (ARM_LPAE_MAIR_ATTR_IDX_CACHE
                                << ARM_LPAE_PTE_ATTRINDX_SHIFT);
        }

same for this one. MAIR_ELx register is programmed to ARM_LPAE_MAIR_
ATTR_WBRWA for IDX_CACHE bit. I'm not sure why it doesn't use=20
IDX_NC though, when !IOMMU_CACHE.

>=20
> So this is all a mess.
>=20
> Better to start clear and unambiguous names in the uAPI and someone
> can try to clean up the kernel eventually.
>=20
> The required behavior for iommufd is to have the IOMMU ignore the
> no-snoop bit so that Intel HW can disable wbinvd. This bit should be
> clearly documented for its exact purpose and if other arches also have
> instructions that need to be disabled if snoop TLPs are allowed then
> they can re-use this bit. It appears ARM does not have this issue and
> does not need the bit.

Disabling wbinvd is one purpose. imo the more important intention
is that iommu vendor uses different PTE formats between snoop and
!snoop. As long as we want allow userspace to opt in case of isoch=20
performance requirement (unlike current vfio which always choose
snoop format if available), such mechanism is required for all vendors.

When creating an ioas there could be three snoop modes:

1) snoop for all attached devices;
2) non-snoop for all attached devices;
3) device-selected snoop;

Intel supports 1) <enforce-snoop on> and 3) <enforce-snoop off>. snoop
and nonsnoop devices can be attached to a same ioas in 3).

ARM supports 1) <snoop format> and 2) <nonsnoop format>. snoop devices
and nonsnoop devices must be attached to different ioas's in 1) and 2)
respectively.

Then the device info should reports:

/* iommu enforced snoop */
+#define IOMMU_DEVICE_INFO_ENFORCE_SNOOP	(1 << 0)
/* iommu enforced nonsnoop */
+#define IOMMU_DEVICE_INFO_ENFORCE_NONSNOOP	(1 << 1)
/* device selected snoop */
+#define IOMMU_DEVICE_INFO_DEVICE_SNOOP	(1 << 2)

>=20
> What ARM is doing with IOMMU_CACHE is unclear to me, and I'm unclear
> if/how iommufd should expose it as a controllable PTE flag. The ARM
>=20

Based on above analysis I think the ARM usage with IOMMU_CACHE
doesn't change.=20

Thanks
Kevin