Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 7 Jun 2023 08:59:57 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Michael Shavit <mshavit@google.com>
Cc:     Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
        Joerg Roedel <joro@8bytes.org>, jean-philippe@linaro.org,
        nicolinc@nvidia.com, baolu.lu@linux.intel.com,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 14/18] iommu/arm-smmu-v3: Support domains with shared
 CDs
Message-ID: <ZIBxPd1/JCAle6yP@nvidia.com>
References: <20230606120854.4170244-1-mshavit@google.com>
 <20230606120854.4170244-15-mshavit@google.com>
 <ZH9oQMmI8CezMCnx@nvidia.com>
 <CAKHBV25eDNXEu+6GWDJqpw1h5N-Jq-GPzf81kNH1dyw3Z=KsZA@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAKHBV25eDNXEu+6GWDJqpw1h5N-Jq-GPzf81kNH1dyw3Z=KsZA@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?GW4Zo9cRhe1qeDidqTTa8A9kkCOJxD4D3k7X29NsP8vCKlCaD5QAYVnKvqy9?=
 =?us-ascii?Q?EAq0IjyJywhDwf5N8bN42bEclY2FIrlCU6azkjrIPwXeJ69tGF4oXk5e2LCA?=
 =?us-ascii?Q?tvG3N2qBasWtneBacLv96LXxWmsRtfSLjsMcOVhyYl9UtXEem/88vIkhAKVk?=
 =?us-ascii?Q?ixFP6l6RgkcW3M8ixaA3jEhr2oUYfeqWh8zt+QOeP3pH9tLDLcNir5KeTBAj?=
 =?us-ascii?Q?2GOSMvD79YpFBWOZr5rosJucEyFmXNslXLNvEF8IR+Vpq1cRgi3L9rSk7hYS?=
 =?us-ascii?Q?e/5qwpmohfIIpSURufSokICWHzDTmcDt578UQJrpgR8urbqnIfn/Pj2+p5ug?=
 =?us-ascii?Q?BmyhmCNU0VOTgHW7QwJX0923lsAb8JYe5KKWm5GjXhOuiOpPcMrC1e24vpLz?=
 =?us-ascii?Q?v8l1lgz6wo2CqaWPgQbU3VGsQqCJloUPcIvW5mPcOafDa3CK06xF08f4Op4q?=
 =?us-ascii?Q?xZkUIcAttebkTn2elqwE87Bzee+DziB9efrqr4Z//zpU5feQDsBaU/WzClqa?=
 =?us-ascii?Q?Vw+nHpDCFustkrMVfVv8SiMuLdlu+6TnLLTSqm2HTVgAUnjzLJjWu5hx923A?=
 =?us-ascii?Q?hgLSV0MrqcnFRzJLZ2DWBT1mifdx1mq+kdWtGgu+KvAOrdySsCDZiOIISSdE?=
 =?us-ascii?Q?OjW9yPHrlWC+O+4bHAwNE3azUDgJ52jO/8HrVCsi4op7GiezrstzFAS3yraB?=
 =?us-ascii?Q?JdYo96wh9FnqIEMPOQu1fgNN4snzzuP9KrYrArD8XEoYsarB/h5YJQma2BLb?=
 =?us-ascii?Q?D/75vtVHVHQjQEDfOHqoRukacB4KLTu1Fyy+IdR9xrj4lsv1M0PHVrB6wfvE?=
 =?us-ascii?Q?72jArBOc9VA9+csv03wfyqw15daSQP8gVeXPgaUDG+zIYVaRX2WmGcaPS98j?=
 =?us-ascii?Q?eu1uEVDLCbXlfV+B1KEC4rNq8PYSuu9OF2RuWnJrzSehL1a7qR6P7Lg1pGgW?=
 =?us-ascii?Q?01r266KkzW4mLEu3/WRg+k118lcgKjIaM6TdOwLBe2YMc63mVz6lwRk48mcQ?=
 =?us-ascii?Q?PCyQAXLod8y47gOv6jENUAAX7S5YpgGjucZ2qm73QF4rQcvfrsbclKWYViDE?=
 =?us-ascii?Q?F5mms7T94GzjllB/W6y8qP3p5hZSdsMhBpqSJblHz7h57VR8k1783V83r+IA?=
 =?us-ascii?Q?AKOcjHZY/mRJQQa3B7G+atYzW8eZk2HYbhUSpTvLnH5vcD1VFaCQEeUsTWmA?=
 =?us-ascii?Q?AztpnC01BuAXF4q5U/0A5Bqx5kDqTiHCdOK6gv0Q6+QUlAng5RmQKOXkbNhq?=
 =?us-ascii?Q?JDyZfUjbT839zelCq/DdFPnXAShTgI5uUDXlT1Y1p3AFk6lll5eo4TU66Gcu?=
 =?us-ascii?Q?yK9oODMLjYpVYooNUb2e/AimRPlEX+gio/SG/l5DzvQjmWvd19DEeXuVtHf0?=
 =?us-ascii?Q?IuhsDH5/emPzQ1Uf0quD0LLckrEMWJf+mGG1GxQisuNrcqfT1RCctJwZX6mq?=
 =?us-ascii?Q?HwCHXognqHAbAQvPFOC2YwRe2Zr4LzgGo3MB0wuOcOl8BBQPgu1gHZFZsDHW?=
 =?us-ascii?Q?CTpY7ar2CCE93RzwvcohOeISuNOHuNc55qLqFtTesCixaVRNMz7o+ISW4F6d?=
 =?us-ascii?Q?IZkYog6WLWRkf3DwuHEUhqEDd2op0OMJP1rLs7Tz?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 297f08ae-620a-4c3e-4dad-08db674eb876
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2023 12:00:00.4412
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qKN+IudnreXY3CdJwBVRfiDIp3venDwxX5d+GAQLvMUDL+pmrxTO1LAckDWuwh+M
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5312
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jun 07, 2023 at 12:06:07AM +0530, Michael Shavit wrote:
> > What we definately shouldn't do is try to have different SVA
> > iommu_domain's pointing at the same ASID. That is again making SVA
> > special, which we are trying to get away from :)
> 
> Fwiw, this change is preserving the status-quo in that regard;
> arm-smmu-v3-sva.c is already doing this. But yes, I agree that
> resolving the limitation is a better long term solution... and
> something I can try to look at further.

I suppose we also don't really have a entirely clear picture what
allocating multiple SVA domains should even do in the iommu driver.

The driver would like to share the ASID, but things are much cleaner
for everything if the driver model has ASID 1:1 with the iommu_domain.

It suggests we are missing some core code in iommu_sva_bind_device()
to try to re-use existing SVA iommu_domains. This would certainly be
better than trying to teach every driver how to share and refcount
its ASID concept...

Today we have this super hacky iommu_get_domain_for_dev_pasid()
thing that allows SVA domain reuse for a single device.

Possibly what we should do is conver the u32 pasid in the mm_struct to
a struct iommu_mm_data * and put alot more stuff in there. eg a linked
list of all SVA domains.

> Splitting this part into a follow-up patch series would definitely be
> easier and helpful if you're all ok with it :) .

I think splitting it into a series to re-organize the way ste/cd stuff
works is a nice contained topic.

Adjusting the way the ASID works with SVA is another good topic

And so on

Jason