Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 22 Mar 2023 14:28:38 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZBs6xlqMGYhLbI27@nvidia.com>
References: <ZAtYphmOuEqQ1BiC@nvidia.com>
 <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZBhYq8o4gqFZVb6k@nvidia.com>
 <ZBiF1hZVrp/aJRM6@Asurada-Nvidia>
 <ZBifWkY2Lt/U1Z7R@nvidia.com>
 <BN9PR11MB5276E6E3FCA90582AA61BDDE8C819@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZBmZj2pscX0hx2kQ@nvidia.com>
 <ZBqjUeEjjtCclgLN@Asurada-Nvidia>
 <ZBr3/1NzY5VvJrJQ@nvidia.com>
 <ZBs2xaa7Ow1pnB3C@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZBs2xaa7Ow1pnB3C@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?TpyKToyD2KMhXcOOW39zo565PYgRUwDct6nUcSav4oceDQvF9zMR8c7QXUob?=
 =?us-ascii?Q?fdFPP1riAKbw89MgPdIy7ra7ipXXNvPNNb0QdTbSXQZSIaOs2hbakqPWIzE8?=
 =?us-ascii?Q?BIZZTZZEWrfCzB62X9VKh8/JJ6+Pms9dnWJ2iw/yL9T/jGwlYVy0LqGOjpKM?=
 =?us-ascii?Q?pjstSEPhw81LYDYjnZXg8OsZHEn2lf2VPynKfp0ZiJKu+s/0lxJWLL4uwG/o?=
 =?us-ascii?Q?Tde7hsO7F+AIFtqKjbPj/X5HsApVtbaNL0BQRYvvFhrV6cYdb1bDh+BhdjJK?=
 =?us-ascii?Q?KsV9QnVfNDSfbWM/1eqljmrTtVe4t9r8LBfj6y1ZYXQBfbU3MPrEfbnXJoxe?=
 =?us-ascii?Q?lxtcRBAKZhYH2gFmapHNijCzgLAYvE0p5+P97LUXElF2VH1vTjcJYgEVjdlP?=
 =?us-ascii?Q?41Y1i7T878LqgJ4ilL+nQYgJ56xfZLx81frQdkBQh4Px5ZjZOD3sceSTVGLF?=
 =?us-ascii?Q?P+t0vb3uoCxRuGFlLJdKkc8sg+hDTccWEMjvpNKUlvJQzLRcZfgxcSo7qq6i?=
 =?us-ascii?Q?mwvFbvFAqKg1N1NMvrIaA+SfyqDJylFsJBVWhd3+T9NmfcZHgqXQHN18JXFU?=
 =?us-ascii?Q?7wq/6PGXpRVOl5RKJEjT2FE9y8OgunNmYiub4iPMarasM2gfqUHqVcMd2sEt?=
 =?us-ascii?Q?u8Arhurcdnb+iCqNKAHUbjG2jmvwKLobeJPMoYmfuZLL3JQBg8MJPJS9t8+p?=
 =?us-ascii?Q?cOJQBJdroqbPnhrXl23Hhjl71hxIipRqZIwZu4TD2siuNe//zaWQgWXKYDCW?=
 =?us-ascii?Q?AKY55r/5oxbDP93+2kcw2J2sThbnSwBt8lH5KADBCXQB3B2Tmayk2DzLIDLC?=
 =?us-ascii?Q?tnJZS0xk4EQu4FFKaOzpj80TCobkAnB/c/a2uzK4QENgRgysC0JERgYMDXFp?=
 =?us-ascii?Q?A4lj2I6exh/Y8SAw+7+j3MU8VJHr6GS8W1ZhZE/lG7KTOx1dplXzFwMofUXx?=
 =?us-ascii?Q?drQ+PLg/enzDfMSMkqg/4wvGNTfNOGmnP5OrAXOJdOaZV8/gjgigqNs1ncex?=
 =?us-ascii?Q?zScImYUgZwzKqtbnt4NOB5cBmXOm5PCRejJebtudPZMHn7Wy/7DIDwmQCcAj?=
 =?us-ascii?Q?06tCkkuwEcEoW/qGozk+kJsAlEjplgKnXtlTcsaJq0D8stenStuCBuvd2SYu?=
 =?us-ascii?Q?K5JWr1KtoRdqnEr9MF2aowLXr2KhW8gtZLzIoHLlxNxwVAQFcqngPgDPH14s?=
 =?us-ascii?Q?ji6tMXpls7Ezhqf+vLN1/xoLrhcQ9Ax+qOMGrJ5qil66//EIg15B5UhQdCqO?=
 =?us-ascii?Q?mYQWKktCqQutYZNK1mPlBUzRge9umz1pG1Botoi3FJFYe67VzCWkSpr04qR2?=
 =?us-ascii?Q?H1FDpJW5GRVVnUfJB4SmM53T1c3vLUya82GU4kCt7DdN8EyO0uEbquSFwtVm?=
 =?us-ascii?Q?njaF7NK+4vidAO/nEx1+s2649Sqx32NBuc0XZjjSrj+nUZ6lM5mioQDjZ12h?=
 =?us-ascii?Q?unLGWbyxFV6IVtLRK6erdTwC+xUtb6ZdmzAOoWPJZwcBeYXcI1DDOH6VMdyV?=
 =?us-ascii?Q?Jl34rhe5PNkEhzOvY1tLIfE6+CzIljSFTqI1BQvXJy9xA3gXdRV82QQWdxBz?=
 =?us-ascii?Q?jHU9itOpR3vtBO9MSWDym9t/wEnTb6h5Ei7DyvHg?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e351e17d-6c39-4321-6379-08db2afadfd9
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2023 17:28:39.0532
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lmdNunUB7GgnXe4WVuxR2PaQM6p/n+2A6yFvdkpB5IESwYrIpEesUToqLHhQuJut
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7413
X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Mar 22, 2023 at 10:11:33AM -0700, Nicolin Chen wrote:

> > Yes, there are a few different ways to handle this and still preserve
> > batching. It is part of the reason it would be hard to make the kernel
> > natively parse the commandq
> 
> Yea. I think the way I described above might be the cleanest,
> since the host kernel would only handle all the leftover TLBI
> commands? I am open for other better idea, if there's any.

It seems best to have userspace take a first pass over the cmdq and
then send what it didn't handle to the kernel

> > On the other hand, we could add some more native kernel support for a
> > SW emulated vCMDQ and that might be interesting for performance.
> 
> That's something I have thought about too. But it would feel
> like changing the "hardware" of the VM, right? If the host
> kernel enables nesting, then we'd have this extra queue for
> TLBI commands. From the driver prospective, it would feels
> like detecting an extra feature bit in the HW register, but
> there's no such bit in the SMMU HW spec :)

You'd trigger it the same way vCMDQ triggers. It is basically SW
emulated vCMDQ.

> Yet, would you please elaborate how it impacts performance?
> I can only see the benefit of isolation, from having a SW
> emulated VCMDQ exclusively for TLBI commands v.s. having a
> single CMDQ interlacing different commands, because both of
> them requires trapping and some sort of dispatching.

In theory would could make it work like virtio-iommu where the
doorbell ring for the SW emulated vCMDQ is delivered directly to a
kernel thread and chop a bunch of latency out of it.

The issue is latency to complete invalidation as in a vSVA scenario
the virtual process MM will block on IOMMU invlidation whenever it
does any mm_struct maintenance. Ie you slow a vast set of
operations. The less latency the better.

> Btw, just to confirm my understanding, a use case having two
> or more iommu_domains means an S2 iommu_domain replacement,
> right? I.e. a running S2 iommu_domain gets replaced on the fly
> by a different S2 iommu_domain holding a different VMID, while
> the IOAS still has the previous mappings? When would that
> actually happen in the real world?

It doesn't have to be replace - what is needed is that evey vPCI
device connected to the same SMMU instance be using the same S2 and
thus the same VM_ID.

IOW evey SID must be linked to the same VM_ID or invalidation commands
will not be properly processed.

qemu would have to have multiple SMMU instances according to S2
domains, which is probably true anyhow since we need to know what
physical SMMU instance to deliver the invalidation too anyhow.

Jason