Date:   Tue, 21 Mar 2023 08:48:31 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Nicolin Chen <nicolinc@nvidia.com>,
        Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZBmZj2pscX0hx2kQ@nvidia.com>
References: <aa327f9ea61e5a4771c13e53639e33955b9acde3.1678348754.git.nicolinc@nvidia.com>
 <1467e666-1b6c-c285-3f79-f8e8b088718b@arm.com>
 <ZAn7uC9UweiNdGkJ@nvidia.com>
 <ZAqv87fjbdynVaHA@Asurada-Nvidia>
 <ZAtYphmOuEqQ1BiC@nvidia.com>
 <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZBhYq8o4gqFZVb6k@nvidia.com>
 <ZBiF1hZVrp/aJRM6@Asurada-Nvidia>
 <ZBifWkY2Lt/U1Z7R@nvidia.com>
 <BN9PR11MB5276E6E3FCA90582AA61BDDE8C819@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276E6E3FCA90582AA61BDDE8C819@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Y1rdCTSDlcr3ltLanN37FgLt0ShCPAceWwMBZpCXHw9IFJh5R9zGT3eQZkZn?=
 =?us-ascii?Q?HklsVGMNjdFB+03Hkb3l1NKllYwZ4FctVt4bETmtFtwiHQzxfZ9j/1FYzajS?=
 =?us-ascii?Q?RQ5pK9N7lzB4imdGr7780Yzbsolt4th7B7cCx6CZSSLhVIMTLBmdwTyBk3OD?=
 =?us-ascii?Q?cHHxNIdUbGjn+ZbyzPFvZIof9pQYq+O7TnSg+F/d+5p/Eir47L8EQVZwa2Ft?=
 =?us-ascii?Q?fYy3sGsVwMfgqMNdW+yVeNc4lE/OEBGfujtMn6IH25HlVB59WLU2FaNJHAQi?=
 =?us-ascii?Q?tbeqWG9hGDclr3LtE1BXrQINlp+cyWCzuCd7eQNe07sQ4HP+KwZfTPqkZzmL?=
 =?us-ascii?Q?EvTmiZi2QNRhGdktOVJslWgQsHIuNMvokigNSE1ouIFqOMqyI0RgaoLB04ir?=
 =?us-ascii?Q?N/ZWhoZncyDj3hkn07C3RLRLuzL2sr+L9hBRtnjlZacsoRnNB3LnL/tVLHCO?=
 =?us-ascii?Q?4wafR31KtKlnRiI6d4iUpviwxwEfu0UfF4xIvw2fBdFTZ/cuHKzRoZ9nuMzk?=
 =?us-ascii?Q?B+PMLGrpIJOIivFLsNjxCIhg+69J4tRTREHK0vTgiPZPUXviAswhIJ4uyYzQ?=
 =?us-ascii?Q?L/L1CVHww4t40dXaUhKhlXbMQOCpjGzjC6YyQuiyb7Htn1PfOKSAqIedM6x3?=
 =?us-ascii?Q?oEurMsmGgbEjKM7z8RngX4SUIHSPazNgiV917Bx8jzNOw6fQ8do7vQ22USgC?=
 =?us-ascii?Q?3POmfV6hvQHT8QWpwwm2YyMNv6RVwUbfcyXY4TPlkTIyrMpVFtk3CPBh9+ZK?=
 =?us-ascii?Q?aYvEHPIx5BCBMMBRJMbhZqESVMjiXpphI81p8SnyQHw/2QxJXQhv1UEROufo?=
 =?us-ascii?Q?aTMX90PDt3uMPvl/8XUNg7ycGu8dBumnnb0UVYxpzvRvD+1wMnjWHI5wO1Ik?=
 =?us-ascii?Q?9orej+r5GU/kLHbxCOA/JFrCC66qCd9XN8Tf5LQ03xz3Dm/7irO17a+IUfiM?=
 =?us-ascii?Q?4mIDfQS5MoC2BW3UFBrRaKc1tkFTlwLs6+fFig8CRfLH+DB6C8gerjPCEFDC?=
 =?us-ascii?Q?39i1YtXJKq6yEVMb2g+cOaqTCr1RafiBpKRfqn1uLcf7vUNpBj7uNezeA16/?=
 =?us-ascii?Q?GjDaadYnFq2QbzdQSjy+Pyus8pcE4CuTcMo4jdRk6C77zIoM37uas+zRTX+E?=
 =?us-ascii?Q?NCXOJArMpPUy0fTShh+PtikF/F1I3KrPBZ2Oz8EEGRWUn3vLXmRWpJB3ojam?=
 =?us-ascii?Q?vCrKwuEUu0l7bLNhXix31p6fH4kYs4d9/Rku3AMeP5MSXvdZ4X0F8gK5kw4d?=
 =?us-ascii?Q?jUHecIAt/MgZEogu3jEUIfjuY64H0NZdURObhXrLkdvUCV3ygft3XBVVMBTK?=
 =?us-ascii?Q?4ZhYXikg6ZZG/IutgIud7DVGQjdoQI9v7NAEBQAgt+LmjOhDJdtJHQp6fva6?=
 =?us-ascii?Q?74CRhrvbFn0NDiFzph02WiN3V91Tm/Cpd7F3mubIgDqJ5UJ4JGrfrUbCk74X?=
 =?us-ascii?Q?IK/XdNwxHO9kveiUlYJrbyQ7q0V0hp3nSTgnw/DOaw3vkGB/+/BquXjRkhhB?=
 =?us-ascii?Q?Ezf/79BaQzU3tF5GcqMGcF8/bDq15OJyvOXu/vdkJC0MuxaFr0d+Zb1i4ecr?=
 =?us-ascii?Q?+lZIW7RyMB5fswW0p+YSSNSh8yyfEpvCraXA1Aec?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9b172a4f-58cd-45f3-9d7f-08db2a023201
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2023 11:48:32.1887
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /Gf1SRpmQ7DjzuoenSlXotNWUaOK4o2bEtXy64DHA5tSQxynMvkQuzlp9Tf6Ka6W
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB5669
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Mar 21, 2023 at 08:34:00AM +0000, Tian, Kevin wrote:

> > > Rephrasing that to put into a design: the IOCTL would pass a
> > > user pointer to the queue, the size of the queue, then a head
> > > pointer and a tail pointer? Then the kernel reads out all the
> > > commands between the head and the tail and handles all those
> > > invalidation commands only?
> > 
> > Yes, that is one possible design
> 
> If we cannot have the short path in the kernel then I'm not sure the
> value of using native format and queue in the uAPI. Batching can
> be enabled over any format.

SMMUv3 will have a hardware short path where the HW itself runs the
VM's command queue and does this logic.

So I like the symmetry of the SW path being close to that.

> Btw probably a dumb question. The current invalidation IOCTL is
> per hwpt. If picking a native format does it suggest making the IOCTL
> per iommufd given native format is per IOMMU and could carry
> scope bigger than a hwpt.

At least on SMMUv3 it depends on what happens with VMID.

If we can tie the VMID to the iommu_domain then the invalidation has
to flow through the iommu_domain to pick up the VMID.

If the VMID is tied to the entire iommufd_ctx then it can flow
independently.

Jason