Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 23 Mar 2023 15:27:49 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZByaJYq9S8jHUdWa@nvidia.com>
References: <ZBmZj2pscX0hx2kQ@nvidia.com>
 <ZBqjUeEjjtCclgLN@Asurada-Nvidia>
 <ZBr3/1NzY5VvJrJQ@nvidia.com>
 <ZBs2xaa7Ow1pnB3C@Asurada-Nvidia>
 <ZBs6xlqMGYhLbI27@nvidia.com>
 <ZBtVN4o7GhjHF5Q7@Asurada-Nvidia>
 <ZBtZ7F3NsxngcKIq@nvidia.com>
 <ZBtoj3deE2Y6k9lq@Asurada-Nvidia>
 <ZBxDM9JJSbx7yB8e@nvidia.com>
 <ZByW3CkGpt1W59XO@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZByW3CkGpt1W59XO@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?GaH3C19uqQj5Sbtih9Q/chUMy84ngkhpyO6qk8Q+RAev1tF1PbkzirR9pPGR?=
 =?us-ascii?Q?G+9hHhTKtvr/gz5WgYACZjnYJZ8bobJ7TCTUdOWrqQxyPfaxLJaTh7W+dF67?=
 =?us-ascii?Q?I6OSrf/bdqp8/zPTuqbl99/IxpQl4vJqr6IumNxn2hQild5UxvQZIQhA0IVc?=
 =?us-ascii?Q?Fb6u/omEevM8d0YBuiSUSGpDNsjFKY7r1JGs+CoFbShfmuKGvRCt3ug2PFXT?=
 =?us-ascii?Q?WM1/p+1OMupb/GfVjMnoHMVzJJdbARZb/svtWr7ZPlsIIU5cUKd02VBWm67C?=
 =?us-ascii?Q?9R3hyWsOi4qr+oldySY1+UgFAy5DQMFSBgLehRM8VE133wALrB+LJ7dblOzN?=
 =?us-ascii?Q?uWkigeSnWeZz3zAedzw3gRmFPu0U06eB9hnouibHYY9G4U3AUhYij60qVcZe?=
 =?us-ascii?Q?tioWRQPgiEzmdQ1nsZB1OlHpM4JGcDYcQECTvWBwRVT+qsgmUqRQlzXg5INs?=
 =?us-ascii?Q?HRKRstlGiaTq5lfPq4KNkYadJgTRH51jP4WlC3QsQmd0A7YYsCdLP2jhmsPC?=
 =?us-ascii?Q?lynPgTF/EqvbJuxGOwZ+Mtkd+BJrsgdounavxtBVLbhV+9vrSQAKKwxbCDM9?=
 =?us-ascii?Q?0jQibRfCrsrYugWXwQDab81x5LGZ0m0ko/cMO48Cv8wyYs7imZTYHMQcEiUW?=
 =?us-ascii?Q?IPSqpN5MIjmQp4sh1F/xS/8Lv9/1fPm1Ki4wzQ1UmJWbfvl7vMT10Ogc/lTV?=
 =?us-ascii?Q?5485mWR/DyiuA1iq+MfxqkHVzFYJp+IBpLGT8lVnChKQJYB4xWwvIES/B4AF?=
 =?us-ascii?Q?9QM4H+X2EgdNeWzQYORek/m+7LeaTCatw3dbJ7MwIR3VQotj97rlUXzPBwCt?=
 =?us-ascii?Q?ZjLgDuFDbAZPJcfN+2+/ry/bUyo7q+9YKeylA6TlrVPYZ7vtwbg6Zy0i2xXF?=
 =?us-ascii?Q?uM2147E3WBVzSGZaxx6OH5NqPuoa6ZGeZ2qG6YG3DDO1VAaUl6yE9OxnqXeF?=
 =?us-ascii?Q?/H2HyYvKA8VLCfpz0fgf6TDSpULaCkvnFpEP0WZ8nFGSRdaY7tObd+iikODo?=
 =?us-ascii?Q?Vj4vAJRQFNbW76bOcjU6ZaXr2ls1yRpCHmKFoLgDFNpCHIGqAo3T6YcSYKeS?=
 =?us-ascii?Q?7RoMUSofx45gXJexhU8O8Y/t264mM0WfYS6tUVWe4DlQMDKlG2clb/5PHxVw?=
 =?us-ascii?Q?z2JE4/jdHcD4k7roy45Usrv4TlhIZ9K65bAJrP12fi1FL0UU2sLNXKJF9TKr?=
 =?us-ascii?Q?mUnMzSn04KLzPScHOt8ay2OPx71jxiT1eeayzESxQG3jL6wnSOY6b6wSGPNy?=
 =?us-ascii?Q?rdTQaoH1k+y3WglhGzYp8LInS+LmJFd9kUcS+7cx34/q5cYYU7R0/qT215Tt?=
 =?us-ascii?Q?JWt+umgEOwZiBaJpQlUkGao0YJUfk9Cc2YDZ5SIyMUueJCli4MfsAi7Q4YBO?=
 =?us-ascii?Q?jWOLtMwGEBEHJmumjsVAhOyTLIbJKZYPX8BnB0XlnJx1n+n3XWeNT4GKIZS/?=
 =?us-ascii?Q?7xX72bgRuKqBq2BrAdIeRRVKBD7Yv8ifjX4E0wlCH7wtNpSmmspuB3yckXGL?=
 =?us-ascii?Q?adxXo9aQL9E62q8fEOmPiEKYb0xiq9N1uM7Vv+KR3CCk+E9hSa2D9G6AL7z9?=
 =?us-ascii?Q?YykyPOO1ExjhSRoZxr7oLrglmwkhg9l2+LeF9yqK?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f9c53e46-7703-4f45-1d31-08db2bcc504f
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Mar 2023 18:27:52.5304
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lC7b5y6T3W/JlivyBxQook6MCfovGlgJ3IxQPIyDFGEwdxiKKgiAVkp8TCBEim55
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6594
X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Mar 23, 2023 at 11:13:48AM -0700, Nicolin Chen wrote:
> Oh. So that the s2_cfg data structures can have a shared S2
> IOPT while having different VMIDs. This would be a big rework.
> It changes the two-domain design for nesting. Should we do
> this at a later stage when supporting multi-SMMU instance or
> now? And I am not sure Intel would need this...

If we do nothing right now then the S2 unmanaged iommu_domain will
carry the vm_id and it will be locked to a single SMMU instance.

To support multi-instance HW qemu would have to duplicate the entire
S2 unmanaged domain to get different vm_ids.

This is basically status-quo today because SMMU already doesn't
support sharing the unmanaged iommu_domain between instances.

If we chart a path to using a dedicated S2 domain then qemu side would
have to change to make a normal HWPT to back the S2 and then create a
real S2 as a child.

This implies that the request for S2 has to be in the driver data
today so that the driver knows if it should enable the unamanged
domain for S2 operation and lock it do an instance.

So long as that is OK we are probably OK to be incremental..

> > And a way to request the correctly bound vCMDQ from the guest as well.
> > Sounds really messsy, I'd think multi-smmu is the much cleaner choice
> 
> Yes. I agree, we would need the entire QEMU community to give
> consent to change that though.

I suppose it wasn't consent, it was someone needs to do the difficult
work.

Jason