Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 6 Jun 2023 14:09:20 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Michael Shavit <mshavit@google.com>
Cc:     Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
        Joerg Roedel <joro@8bytes.org>, jean-philippe@linaro.org,
        nicolinc@nvidia.com, baolu.lu@linux.intel.com,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 14/18] iommu/arm-smmu-v3: Support domains with shared
 CDs
Message-ID: <ZH9oQMmI8CezMCnx@nvidia.com>
References: <20230606120854.4170244-1-mshavit@google.com>
 <20230606120854.4170244-15-mshavit@google.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230606120854.4170244-15-mshavit@google.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?I6mBLjipJDoX/niywMOSrSBDAzuUujUnNuPp4P8BTSb+mrpQX49kzon1jQGK?=
 =?us-ascii?Q?5hdlzvGc99gysgs9nBpPgFPnANI+L7DmbA3PtS4Pa48r48pI2IGMejZ/Y/sG?=
 =?us-ascii?Q?PSqJlp9ckrB8WlwtjXn1jydxpJjAtQx7PlxrcH4FxYz0AJ8B53U90xakDHjJ?=
 =?us-ascii?Q?d1CBSj3y2TACxWIiuRWmEcHK5OPHijAmLo8NDnvNEAyUjrBiu4j+x4DyEsnt?=
 =?us-ascii?Q?dI7nQrb8L1q+izeLWM9PV69yIDPo6WN8TXR90M77JBMjd2TBkp20HnDul64q?=
 =?us-ascii?Q?FMY/WcyybbR4Dr6n9tY8mJbRsa45tBFRAl1mIscGQaVjYEbmRcv8vnk+0Tvy?=
 =?us-ascii?Q?iWpU0pC/TyuKSOG3m3Q7ie/IdsD+dsyq3Jx0xZL8/mzyFCAedqp6U0nF63GR?=
 =?us-ascii?Q?l1ItkEiCZllOI/ZO7oX0wC9pHZx1K+vzPL3dTlk55s7xQCnVcPZ04GSffQ2H?=
 =?us-ascii?Q?g5TKhoed5CZTSBv9AfmmAqhX5mz7Id9HKrVSJk6x6u83ZfFS2F6iYRsEv2QQ?=
 =?us-ascii?Q?4ad+iyYP7cSgL8F3++GTarQ9/ax1McV+60v1Ti1xT0Kj40vQ2VQWJRSW9Is4?=
 =?us-ascii?Q?qiBYDufvSSwDG78tVRKOzeoaFS1T2XU9chxX2+DDz4abxeDzd/P0On/x9GZu?=
 =?us-ascii?Q?ykwEyYj8EKxTpiJr5YWrD3p6iZG6kxyFR6r0NqNQeHLfTOVa2En3LQlgl17j?=
 =?us-ascii?Q?QNaax4ueqTKIwdFZYIpFEYYYUWimIUGv3TPugA0wMOhiA81RVlDJPFFtBN0V?=
 =?us-ascii?Q?A/+EPYqt2uE8banc7T+MSOM/VcOfadKDtLAD40v59RvjB007xVK79pCUswLC?=
 =?us-ascii?Q?ZyBjRmgRBSk+9twIVANx0DavfG9BO0DP8OwsjoN9XeZTK2hq+pdAfRST905V?=
 =?us-ascii?Q?05e0STJSDewNMwSrenCg36TJlYoPfDz5rBIYwvRSBfEiXIQ2+bmS90XdhrJK?=
 =?us-ascii?Q?OjyqNOe/SzStVI8FmkxXMZLPnpkhRHZ0WtwWJdDA5EugBwLtHG9ejDsiFmOK?=
 =?us-ascii?Q?9x6lFD933xu793fMCO0IbrUFFNhyIIjBWmmYvUb0T48cKzcvf+fCFd9gmFEK?=
 =?us-ascii?Q?uCcuwRMsyzj5mzDmB3VRkeOGlB8eoEt9565KpihKznO1TTsQvezgoAUpsO1M?=
 =?us-ascii?Q?Tb4IHEdoxbOD+aMqZYqaDw3EFh229nqTpL/Yb0VbYuejvAdY0xmhZb3Lgm3i?=
 =?us-ascii?Q?R7+P3ROD5mUdGlZyvsoEBUK+1z6VSFZa70nqhoAWVq22XFmrivwMh0rKhjqt?=
 =?us-ascii?Q?Ckp1bPXsG/eYypRsSn7AqnZabW+E4uwMSuQn7r12PeXayxXtcSBMoIjJTIBY?=
 =?us-ascii?Q?JcXoxTcpKwnOXGJl8urd0FuXhimH1EnGG97s0bVMQT9I0crCG7M5YH0Q7vqn?=
 =?us-ascii?Q?NeQS0utGlw8nvXPSL6tqMm+NNlqW8H5yFcUQbWm1sf5gn/kOQUXFJ7IfyUs7?=
 =?us-ascii?Q?uRSoKSbrhZ1d7RlX2zUoSsHRmfibHKV2l5RFnUFAZ2He2hDCn9VVq4R9bX2S?=
 =?us-ascii?Q?6pt0kcDLxiMmvkzN6mQmhj3mKccojBTUFE31BKUGC3Cmfz22IrGo47sXDzi/?=
 =?us-ascii?Q?quAekyXhocRBTja3nA8nfDKHztbyBLHUe/tpCLOU?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 30401537-2278-49f6-36fe-08db66b0c68f
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2023 17:09:23.5913
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mbvw4LdrBqEu5nDEviNZQ+6AprGYQK14sHujRLv/V3O8hwIJ4ucL50JB1UAHboF0
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8922
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Jun 06, 2023 at 08:07:50PM +0800, Michael Shavit wrote:
> SVA may attach a CD to masters that have different upstream SMMU
> devices. The arm_smmu_domain structure can only be attached to a single
> upstream SMMU device however. 

Isn't that pretty much because we don't support replicating
invalidations to each of the different SMMU instances?

I don't really get why there would be anything like a "shared CD".

It looks kind of like the arm_smmu_ctx_desc has become a bit weird
since its main purpose seems to be to refcount the ASID.

But our general design is that the iommu_domain should be 1:1 with the
ASID, so it is really strange that we'd refcount the ASID..

I would expect different SMMU devices to be handled by allowing single
domains to be attached to a list of masters where the masters can all
be from different instances.

When an invalidation comes in we have to replicate it through all the
instances for the IOTLB and all the masters for the ATC.

What we definately shouldn't do is try to have different SVA
iommu_domain's pointing at the same ASID. That is again making SVA
special, which we are trying to get away from :)

You might try to stop your series here and get the first batch of
patches done. This latter bit seems to be a seperate topic?

Jason