Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 28 Mar 2023 09:26:35 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Nicolin Chen <nicolinc@nvidia.com>,
        Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZCLc+5/exFTZJMRi@nvidia.com>
References: <ZAtYphmOuEqQ1BiC@nvidia.com>
 <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZBhYq8o4gqFZVb6k@nvidia.com>
 <ZBiF1hZVrp/aJRM6@Asurada-Nvidia>
 <ZBifWkY2Lt/U1Z7R@nvidia.com>
 <BN9PR11MB5276E6E3FCA90582AA61BDDE8C819@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZBmZj2pscX0hx2kQ@nvidia.com>
 <BN9PR11MB527663094F21D3DCF8A3038C8C849@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZB23WJu2BEH0yC/y@nvidia.com>
 <BN9PR11MB52760914939B62A61E5309858C889@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52760914939B62A61E5309858C889@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bbPZO3abjiSFTGu3aFDTNdwVjlPZJtVWtsJzfJeCwnsZRhhHQq5LMGuW1+gZ?=
 =?us-ascii?Q?d0UfV5VlfoGnToXjww7g1iEF46xNSgy4Y15JsqZhbS5cDJEsqi39iOiTkfPC?=
 =?us-ascii?Q?mmRiQfDrH1fYC5c6UTI1KaDbh/j0L8E4Zg3TGsMO1ZokwwIwMGJm6p/uXhXN?=
 =?us-ascii?Q?Shlx30OQIxjfnfKSMvFzwnexRgzuuNrwa8YqDSgrcbjygXWvS3+Ynh1AwMxE?=
 =?us-ascii?Q?aSZtEQVZ8OCA9Zlz/vZelvEoi2jcId4NKU4s7v6Hr2uLqMBe4drjpD4ispKt?=
 =?us-ascii?Q?Qe/4tBZJLmZtGMBvWOSaR9hXu854IIpY93bIsNTXo4/kl5J3sgRbdu2n9yjj?=
 =?us-ascii?Q?0mc+X71iRIsO0LVOon7YmipX1yLbThIuO79KqfY2sMLaTDfudpHycmoK7S2u?=
 =?us-ascii?Q?WvDrN2Hr4hcFphnCjFOpYB3qNdMkJKwKtvKLeNP5hsKVi8ffGMp7nGaKjLf/?=
 =?us-ascii?Q?LLrCGkIN6cHrPfHTwA7oA9RR2oZXccM0PxgyEh+t26gsZhxmGqQoeUu/HXcA?=
 =?us-ascii?Q?xzP1J6wAtXsZcF491AA4rF9DrH11zgEwcLV2PWNxaIYv0T0yHJwfOs2ZQOTh?=
 =?us-ascii?Q?SYefRsOMFTSY5FtxMI5if/aLBklj1EXnlFP8iDdtVOBRkgHJvEucQrvYgaoK?=
 =?us-ascii?Q?SLiYV6Oe1JWl0zQxpYnBJmMa2Vzw7anmktbMaWbnr0A8T6yFEgS5Q+Ug+8ME?=
 =?us-ascii?Q?TUCxki6jcHl3NLC81dRmauDZfPNd/I/M1ANeFFQf+47N6JL/sBKe2T2uHvyf?=
 =?us-ascii?Q?HhZK6WKZlb7b7fio09LcVWgvDWO6enPZPiFG6QwyQIrQfcz8bMgvw0poLGXx?=
 =?us-ascii?Q?Pv2mTJoz+wFgeFxxjKM7zPjHkvUkz0ubp6l6DiKdMoOFIVLPQsrQnvU3Tu5J?=
 =?us-ascii?Q?rCJpjBGu3yyuDmb1PBnV6/0Y0noxV43VthG231gH1uchbQqu/NLujYQ21H+h?=
 =?us-ascii?Q?3cXC3BTKzkMcGWZLmjQ4qDoxv/pqz+w9pFOV2N2dBnXWtE3873nme98Dgzl4?=
 =?us-ascii?Q?O4JFCTwJp/EkyS5srM26Y486p6yU4AV4WACCM1dqLI+fTtS8OYEmxjumuVoU?=
 =?us-ascii?Q?WJ9NATW8MvzUMZwTWrldi/Tzk4W4WbQfVOdI+gPg0GfPXH4EfhocDxEsdNJz?=
 =?us-ascii?Q?V1z7ohaTNwOan+1kPawfS3AwOA0Kp3XIWCBlZI1K5qnXBmZlN+ADqTHzCHFh?=
 =?us-ascii?Q?rHV0H6pvSnK5+Z8XC558txRHBvnacNL8e3jyVJFqwd6/5T6/3maOyGertDbp?=
 =?us-ascii?Q?w+0Rh3uJ/6ene77i6lVEitid52qeUEx9qbzAhkpA4ZM9RZKpzQfcO9TXbBXG?=
 =?us-ascii?Q?oXzxI0oQxjEGRSFl4SCpQXZpTiHy8Ldb+bBeiS/2Bt/78h+N7nH/U+GoueBK?=
 =?us-ascii?Q?j5WAC/07i8s/5fRyI2jZFzu+yMCu5/giMH+sjkZfWM16yte1NBevo1HjI8LK?=
 =?us-ascii?Q?/QDIOcE2106tNzJENnxS4hyO5q7YxMu/Y+khwZb0roSaFUPxIC5+0LCwW1DF?=
 =?us-ascii?Q?mlQzb9kOndnCOjn1+0O6PIXM3YpQvOQiLmUw4xamE9nOzAJmeYAMpYWfYMqz?=
 =?us-ascii?Q?nnN5QDqR7cfd+M3JiWgxN6Gl6tvGH/TRsXDi8SYQ?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 42b06eb5-7afa-4bfb-4349-08db2f87acbd
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2023 12:26:36.9794
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Z/rArlxII7WuPwxZsIxUjgYInc6DCelgfkG+PHQIFFnOJax5M+tHGBwwDEnJQeud
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5314
X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Mar 28, 2023 at 02:48:31AM +0000, Tian, Kevin wrote:

> But emulated vIOMMUs are typically MMIO-based ring structure
> which requires 1) kvm provides a synchronous ioeventfd for MMIO
> based head/tail emulation; 2) userspace vIOMMU shares its virtual
> register page with the kernel which can then update virtual tail/head
> registers w/o exiting to the userspace; 3) the kernel thread can
> selectively exit to userspace for cmds which it cannot directly handle.

What is needed is for the kvm side to capture the store execute it to
some backing memory, and also trigger the eventfd.

It shouldn't need to be synchronous.

For SMMU the interface is layed out with unique 4k pages per-CMDQ that
contains the 3 relevant 8 byte values.

So we could mmap a page from the kernel that has the 3 values. qemu
would install the page in the kvm memory map and it would 
arrange things so that stores reach the 8 bytes and trigger an
eventfd.

Kernel simply reads the cons index after the eventfd, looks in the
IOAS to get the queue memory and does the operation async.

It is not especially conceptually difficult..

Jason