Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.160 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C
Date:   Thu, 13 Jul 2023 12:54:47 -0700
From:   Nicolin Chen <nicolinc@nvidia.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Michael Shavit <mshavit@google.com>, Will Deacon <will@kernel.org>,
        "Robin Murphy" <robin.murphy@arm.com>,
        Joerg Roedel <joro@8bytes.org>, <jean-philippe@linaro.org>,
        <baolu.lu@linux.intel.com>, <linux-arm-kernel@lists.infradead.org>,
        <iommu@lists.linux.dev>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v4 02/13] iommu/arm-smmu-v3: Add smmu_s1_cfg to
 smmu_master
Message-ID: <ZLBWh370pPjx2B+z@Asurada-Nvidia>
References: <20230621063825.268890-1-mshavit@google.com>
 <20230621063825.268890-3-mshavit@google.com>
 <ZK9RycgNAVrxe343@Asurada-Nvidia>
 <CAKHBV26wi+xKnNjo-R+QOcVLPH2KJTFP+mF4CW1xE61nOdF5GA@mail.gmail.com>
 <ZLAKQw+DzcpSRSyi@nvidia.com>
 <CAKHBV25YadRVFiag5z5Yc13L093ScWkCjAOCd=VuGm2RUaDyzA@mail.gmail.com>
 <ZLApQjaqoOshT2TJ@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <ZLApQjaqoOshT2TJ@nvidia.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jul 2023 19:55:04.4022
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 2ccb5c79-ce57-4d36-d6fd-08db83db0d37
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT017.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7497
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jul 13, 2023 at 01:41:38PM -0300, Jason Gunthorpe wrote:
> On Fri, Jul 14, 2023 at 12:16:16AM +0800, Michael Shavit wrote:
> > On Thu, Jul 13, 2023 at 10:29 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> > > It would make alot more sense if the STE value used by an unmanaged S1
> > > domain was located in/near the unmanaged domain or called 'unmanaged
> > > S1 STE' or something if it really has to be in the master. Why does
> > > this even need to be stored, can't we compute it?
> > 
> > struct s1_cfg* and struct s2_cfg* are precisely what is used to
> > compute an STE. For example, when s1_cfg is set, arm_smmu_write_strtab
> > will write the s1_cfg's CD table dma_pointer into the STE's
> > STRTAB_STE_0_CFG field. When neither are set, the STE fields are
> > written to enable bypass (or abort depending on the config).
> 
> I guess I never really understood why these were precomputed and
> stored at all. Even more confusing is why we need to keep pointers to
> them anywhere? Compute the STE and CDE directly from the source data
> when you need it?
> 
> eg If I want to install an IDENITY domain into a STE then I compute
> the STE for identity and go ahead and do it.

I think it'd be clear if the master stores STE value directly to
get rid of s1_cfg/s2_cfg in the master struct. We fill in those
domain-related STE fields when the domain attaches to the master
and then call arm_smmu_write_strtab().

For struct arm_smmu_domain, it still has a union of a CD (for an
S1 domain) or an s2_cfg (for an S2 domain). Or we could select
a better naming for s2_cfg too, since s1_cfg is gone.

Does this match with your expectation?

> > > I'd think the basic mental model should be to extract the STE from the
> > > thing you intend to install. Either the default CD table, or from the
> > > iommu_domain. ie some 'get STE from iommu_domain' function?
> > 
> > I don't follow this. When we attach a domain with pasid (whether
> > through SVA or the set_dev_pasid API) , we don't want to install an
> > entirely new CD table. 
> 
> The master object owns an optional CD table. If it is exsists it is
> used by every domain that is attached to that master.
>
> In the code flow there are two entry points to attach a domain, attach
> to a PASID or attach to a RID.
> 
> For attach to PASID the code should always force the master to have a
> CD table and then attach the domain to the CD table.
> 
> For attach to RID the code should do a bunch of checks and decide if
> it should force the master to have a CD table and attach the domain to
> that, or directly attach the domain to the STE.

I think a master own a CD table in both cases, though the table
could have a single entry or multiple entries, depending on its
ssid_bits/cdmax value. And since a master own the CD table, we
could preallocate the CD table in arm_smmu_probe_device(), like
Michael does in this series. And at the same time, the s1ctxptr
field of the STE could be setup too. Then we insert the CD from
a domain into the CD table during the domain attaching.

Yet two cases that would waste the preallocated CD table:
1) If a master with ssid_bits=0 gets attached to an IDENITY S1
   domain, it sets CONFIG=BYPASS in the STE, which wastes the
   single-entry CD table, unlike currently the driver bypassing
   the allocation of a CD table at all.
2) When enabling nested translation, we should replace all the
   S1 fields in the STE with guest/user values. So, the kernel-
   level CD table (either single-entry or multi-entry) in the
   master struct will not be used. Although this seems to be
   unchanged since currently the driver wastes this too in the
   default domain?

With that, I still feel it clear and flexible. And I can simply
add my use case of attaching an IDENITY domain to the default
substream when having a multi-entry CD table.

Thanks
Nicolin