Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 23 Sep 2021 09:22:20 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Jean-Philippe Brucker <jean-philippe@linaro.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Message-ID: <20210923122220.GL964074@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-11-yi.l.liu@intel.com>
 <20210922152407.1bfa6ff7.alex.williamson@redhat.com>
 <20210922234954.GB964074@nvidia.com>
 <BN9PR11MB543333AD3C81312115686AAA8CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YUxTvCt1mYDntO8z@myrica>
 <20210923112716.GE964074@nvidia.com>
 <BN9PR11MB5433BCFCF3B0CB657E9BFE898CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5433BCFCF3B0CB657E9BFE898CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YpBHkV/516CuvBSxxB8EcTLDhLzvKpf2gI7aYPeApv6GjLHTZOZZ9iPqBWpV?=
 =?us-ascii?Q?HBQDYIfXcjk04h0lu3CON82f/LZcuyQPENFdyZZNM/8l29ZFnKH9RqkIAZTN?=
 =?us-ascii?Q?0oeqzYJ0qwmYMllp9YbJHs3LjTrBF+dzv14ibgmPh132g3PVsSpqWCXZWDUQ?=
 =?us-ascii?Q?sOPJYtwZ6rfc/6HaqEPtCyIRiyjinfpklQREVNB1fgH/QOqnepYpoTYjogEF?=
 =?us-ascii?Q?TaTLlXgX6CpbPJ/FI2nJWiceIp/QPQl1sq9XnyPX/2hywaGpKlsUtemh/hZ8?=
 =?us-ascii?Q?SiC2kKWOvmWvKE5T9CEYtyHQ+//Ujm6rIE/jHVM49HpV7g0krByoP/knGjKA?=
 =?us-ascii?Q?txo5snf8NWuqsFsKGi9jlzdHGnkA2/SWqu04IQcSgLsOniVaSPd23ESHgHP8?=
 =?us-ascii?Q?9dZ3ytzZk/6TPWLUqYDfRuwi6BoBhx1xLX6MhLEdFc4pX4NYjz9ayZMIvh98?=
 =?us-ascii?Q?/Clm6wzOaOZs/oNRcnvrOuLP3Fr/dq9TN2xz2jVE9xDHDXv2HYTGyBrimchj?=
 =?us-ascii?Q?HYIyMePD4B8oHaqtuBG98oeG52Lg0dxIhYh1+kRpSNL4F6nfHmmsbnCclA2g?=
 =?us-ascii?Q?iB4x1fJWWCnuI1fRfHSz/6zE5JHnWFPvVywIPI6qh9kEc5b+zZhk7zPWEAkv?=
 =?us-ascii?Q?2PdE+zXjMyYxmiBHXosnYIRqtOKoz8f+65PHieZxWIMNDW9p8Eh0TuLQtWsA?=
 =?us-ascii?Q?TV/HPDUV5ehIpWPlk/3cNnxI2t9g0Itt9vB1tM0wo2wb9VJHwMv+Ncz4YEx9?=
 =?us-ascii?Q?RuX3Yl63PCI36WydFfkj8/fzNtNg9WERH9rmMxEEd1griNbTPvXxL49vMzQI?=
 =?us-ascii?Q?YaGFOvF/UUxz5bazkaf8xHbPWDdN/aV1y7fODZ0gon9wihNlYcugin2gCroQ?=
 =?us-ascii?Q?fCvTEi6ztrBqRXhNzsPjfs6q3iq2DEjDKnu2aehRpUrvCP4Nuao//E0sfitq?=
 =?us-ascii?Q?kI68OIg5h/opp19d8Gif4ZlFQH0kpzd35XI58MI3Esls0PP7pfyaGOUKEB6E?=
 =?us-ascii?Q?VfpK2ZHt4Kz2FquDQ0dvH5iPJy5UKgFyQucZNs2iwsZpDcLQMTBoiCC7et8+?=
 =?us-ascii?Q?Ny8ozDowUkELUb7jn8RlBHeZRgcHAXwyguoA9kYN6yfe2HDBTpBmXeHaT7WM?=
 =?us-ascii?Q?9yjJqunfLguFP2H9UMwyUDue0yoHRnLVuITpgPJyPeiYo/uYEe6sKkcu8/0X?=
 =?us-ascii?Q?zz78BuqgI8PMOgCHhW3xsMNi14rE/3vEu1NGbULtY/geQDsHydNd/QPrEGqn?=
 =?us-ascii?Q?QKXK1DNiihVbnvpqIzRIJNNrWUtu7j1RzFcGslnbQVMMhv9MkAyM2PcRylx0?=
 =?us-ascii?Q?O8Jr88jB84XXfZumdxxs38zF?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8d519358-5f63-40c9-bc2d-08d97e8ccafb
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2021 12:22:21.7424
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EV4YMlrPCmTfkOypTurFnYqAoUYBQRPvpw3d/Vj8Wl00/1erctgsX0+SYqt/SgTe
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5175
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 23, 2021 at 12:05:29PM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Thursday, September 23, 2021 7:27 PM
> > 
> > On Thu, Sep 23, 2021 at 11:15:24AM +0100, Jean-Philippe Brucker wrote:
> > 
> > > So we can only tell userspace "No_snoop is not supported" (provided we
> > > even want to allow them to enable No_snoop). Users in control of stage-1
> > > tables can create non-cacheable mappings through MAIR attributes.
> > 
> > My point is that ARM is using IOMMU_CACHE to control the overall
> > cachability of the DMA
> > 
> > ie not specifying IOMMU_CACHE requires using the arch specific DMA
> > cache flushers.
> > 
> > Intel never uses arch specifc DMA cache flushers, and instead is
> > abusing IOMMU_CACHE to mean IOMMU_BLOCK_NO_SNOOP on DMA that
> > is always
> > cachable.
> 
> it uses IOMMU_CACHE to force all DMAs to snoop, including those which
> has non_snoop flag and wouldn't snoop cache if iommu is disabled. Nothing
> is blocked.

I see it differently, on Intel the only way to bypass the cache with
DMA is to specify the no-snoop bit in the TLP. The IOMMU PTE flag we
are talking about tells the IOMMU to ignore the no snoop bit.

Again, Intel arch in the kernel does not support the DMA cache flush
arch API and *DOES NOT* support incoherent DMA at all.

ARM *does* implement the DMA cache flush arch API and is using
IOMMU_CACHE to control if the caller will, or will not call the cache
flushes.

This is fundamentally different from what Intel is using it for.

> but why do you call it abuse? IOMMU_CACHE was first introduced for
> Intel platform:

IMHO ARM changed the meaning when Robin linked IOMMU_CACHE to
dma_is_coherent stuff. At that point it became linked to 'do I need to
call arch cache flushers or not'.

> > These are different things and need different bits. Since the ARM path
> > has a lot more code supporting it, I'd suggest Intel should change
> > their code to use IOMMU_BLOCK_NO_SNOOP and abandon IOMMU_CACHE.
> 
> I didn't fully get this point. The end result is same, i.e. making the DMA
> cache-coherent when IOMMU_CACHE is set. Or if you help define the
> behavior of IOMMU_CACHE, what will you define now?

It is clearly specifying how the kernel API works:

 !IOMMU_CACHE
   must call arch cache flushers
 IOMMU_CACHE -
   do not call arch cache flushers
 IOMMU_CACHE|IOMMU_BLOCK_NO_SNOOP - 
   dot not arch cache flushers, and ignore the no snoop bit.

On Intel it should refuse to create a !IOMMU_CACHE since the HW can't
do that. All IOMMU formats can support IOMMU_CACHE. Only the special
no-snoop IOPTE format can support the final one, and it is only useful
for iommufd/vfio users that are interacting with VMs and wbvind.

Jason