Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 30 Sep 2021 19:04:46 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "hch@lst.de" <hch@lst.de>, "Jiang, Dave" <dave.jiang@intel.com>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "parav@mellanox.com" <parav@mellanox.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "Tian, Jun J" <jun.j.tian@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>
Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Message-ID: <20210930220446.GF964074@nvidia.com>
References: <20210922234954.GB964074@nvidia.com>
 <BN9PR11MB543333AD3C81312115686AAA8CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YUxTvCt1mYDntO8z@myrica>
 <20210923112716.GE964074@nvidia.com>
 <BN9PR11MB5433BCFCF3B0CB657E9BFE898CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923122220.GL964074@nvidia.com>
 <BN9PR11MB5433D75C09C6FDA01C2B7CF48CA99@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210929123630.GS964074@nvidia.com>
 <BN9PR11MB5433C9B5A0CD0B58163859EC8CAA9@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YVWSaU4CHFHnwEA5@myrica>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YVWSaU4CHFHnwEA5@myrica>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tUvmr6mITBcCID8KCzovi1G1FApXao0FkVxMps/3ff/zPSKP7gjqYZ9patIz?=
 =?us-ascii?Q?26qUTttlTQ4HRvhuqhu4pR9q9Zc9gIBaGeZ4BgLIhBNPu6bb/hbVUEkV7C61?=
 =?us-ascii?Q?lvH/R0RoTgLfjz4vmCyMb45jb+6Uxxkm8W2YfTim8/G1uUyS6Ss8Rc2uvTf7?=
 =?us-ascii?Q?ZjvsbG01ZGGBH1y8ytnhEYT1P/jSvubwBwZ9xaUICKU2McbGgaJykQKzIqGQ?=
 =?us-ascii?Q?SIbCCoTDQtrV6d5eJw3I5CSmNEEy6BtR/qCRz+eisb6KiDwJU+ObBeEpAEcC?=
 =?us-ascii?Q?tJQSYbt+OZgNdV1YNo2FypaoyeLoHrlsP3YX7bsDLszqU23isGADmJsmw4U7?=
 =?us-ascii?Q?C/8E+55CUcIX09nZY/WtdZyS+5Ms7fdRXZKi4/+I/ijFrTEL1vLz/hCfBAva?=
 =?us-ascii?Q?Fri9Wda9M5tQn+NfelIGFpeKuniUZQfgS1vM9SLJUPdAGXj5X5459c9LelQl?=
 =?us-ascii?Q?vfIiifrz8YzZWMZ5CEyUrWr5KvMABntgEnSJdYOvQ2KPuHT8uWnvMMGLTJby?=
 =?us-ascii?Q?roIGLGKXjNFUZMT1eJnkDjh7c3RugJi6qpopR4k6RRuRA/bXUymWoUF2gXQJ?=
 =?us-ascii?Q?95ufCH0c7UgPD68Lhw5oHtJmk9wYK17SLjnxY84FOHWSaGqB87c/7yyIAlCI?=
 =?us-ascii?Q?wX0ZUj5zwE3hHVWZSqmif8NEjEKqG3Xzhxxr1moJrOvhCe7A4N1h+je4cbts?=
 =?us-ascii?Q?ckpggSqTwBZ6I9IkO+sEpTvze1++UKg0KNlcczEwE8NOw4BZq5/sKxPSn5ln?=
 =?us-ascii?Q?qEFFTW7Y5oU4IOQjq6jtRkl4xCmyJZiTLqLbWUZ5Qm2Bf/PrQoMvw39SNQkc?=
 =?us-ascii?Q?l8ta8sgURoXL2tULbElI7hV+pWIscwCd1KryJCPxGci1K7XyUbbvAwe2iInm?=
 =?us-ascii?Q?v4i1HNYYgOh63XiGBGXjrhVbFFk5+zd3dtrAOnAgk6haLYI9Qqpc+sZ+gB1T?=
 =?us-ascii?Q?fsoBa6eUUrO+koNiQncMqYyS5U/rdaDguoFN8rz3cpT1YPMb+jsfL/eLLE0S?=
 =?us-ascii?Q?F60qquXKHieKBD1i9Ij7/a8ndkPRuksUr5eQiFe5Mc2Y06gRTYCWMIj7hwWr?=
 =?us-ascii?Q?Ws0tnkECfH0kqgRHru42UcO86ZnC5LzRAtsaPTD8TKavgxB8K7x8S70YXS1G?=
 =?us-ascii?Q?QCqNJwSEJO8S89LVSA5tR1eySZs/goVxy4PTWDmtUHJOhauMbPyJ1/nG8N3x?=
 =?us-ascii?Q?n1ZLK6VrKTFiUS+eMfGNCns5V8gg/P2eCKqqSyxwagNCDQeNlAlGDeVt0AXx?=
 =?us-ascii?Q?1ydiqVJkqaWC2Sm1Lx4lPAbmPRhfDduXJ3pC+jl958LRkwxOMtoMerO0yD5d?=
 =?us-ascii?Q?I1WUSIFiWt6LztuKYEPKKckI?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 61e97d8c-1f70-4266-bccb-08d9845e5162
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2021 22:04:48.0228
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HSkygK45tOONcnIeYWgKNt9ZE4UNQbDkxs+Gc7qSCWwqSzIFF/Xjk2WQyMG8Bohj
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5206
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 30, 2021 at 11:33:13AM +0100, Jean-Philippe Brucker wrote:
> On Thu, Sep 30, 2021 at 08:30:42AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe
> > > Sent: Wednesday, September 29, 2021 8:37 PM
> > > 
> > > On Wed, Sep 29, 2021 at 08:48:28AM +0000, Tian, Kevin wrote:
> > > 
> > > > ARM:
> > > >     - set to snoop format if IOMMU_CACHE
> > > >     - set to nonsnoop format if !IOMMU_CACHE
> > > > (in both cases TLP snoop bit is ignored?)
> > > 
> > > Where do you see this? I couldn't even find this functionality in the
> > > ARM HW manual??
> > 
> > Honestly speaking I'm getting confused by the complex attribute
> > transformation control (default, replace, combine, input, output, etc.)
> > in SMMU manual. Above was my impression after last check, but now
> > I cannot find necessary info to build the same picture (except below 
> > code). :/
> > 
> > > 
> > > What I saw is ARM linking the IOMMU_CACHE to a IO PTE bit that causes
> > > the cache coherence to be disabled, which is not ignoring no snoop.
> > 
> > My impression was that snoop is one way of implementing cache
> > coherency and now since the PTE can explicitly specify cache coherency 
> > like below:
> > 
> >                 else if (prot & IOMMU_CACHE)
> >                         pte |= ARM_LPAE_PTE_MEMATTR_OIWB;
> >                 else
> >                         pte |= ARM_LPAE_PTE_MEMATTR_NC;
> > 
> > This setting in concept overrides the snoop attribute from the device thus
> > make it sort of ignored?
> 
> To make sure we're talking about the same thing: "the snoop attribute from
> the device" is the "No snoop" attribute in the PCI TLP, right?
> 
> The PTE flags define whether the memory access is cache-coherent or not.
> * WB is cacheable (short for write-back cacheable. Doesn't matter here
>   what OI or RWA mean.)
> * NC is non-cacheable.
> 
>          | Normal PCI access | No_snoop PCI access
>   PTE WB | Cacheable         | Non-cacheable
>   PTE NC | Non-cacheable     | Non-cacheable
> 
> Cacheable memory access participate in cache coherency. Non-cacheable
> accesses go directly to memory, do not cause cache allocation.

This table is what I was thinking after reading through the ARM docs.

> On Arm cache coherency is configured through PTE attributes. I don't think
> PCI No_snoop should be used because it's not necessarily supported
> throughout the system and, as far as I understand, software can't discover
> whether it is.

The usage of no-snoop is a behavior of a device. A generic PCI driver
should be able to program the device to generate no-snoop TLPs and
ideally rely on an arch specific API in the OS to trigger the required
cache maintenance.

It doesn't make much sense for a portable driver to rely on a
non-portable IO PTE flag to control coherency, since that is not a
standards based approach.

That said, Linux doesn't have a generic DMA API to support
no-snoop. The few GPUs drivers that use this stuff just hardwired
wbsync on Intel..

What I don't really understand is why ARM, with an IOMMU that supports
PTE WB, has devices where dev_is_dma_coherent() == false ? 

Is it the case that DMA from those devices ignores the IO PTE's
cachable mode?

Jason