Received-SPF: pass (google.com: domain of linux-kernel+bounces-164829-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Tue, 30 Apr 2024 21:17:58 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: will@kernel.org, robin.murphy@arm.com, joro@8bytes.org,
	thierry.reding@gmail.com, vdumpa@nvidia.com, jonathanh@nvidia.com,
	linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
	linux-arm-kernel@lists.infradead.org, linux-tegra@vger.kernel.org
Subject: Re: [PATCH v6 6/6] iommu/tegra241-cmdqv: Limit CMDs for guest owned
 VINTF
Message-ID: <20240501001758.GZ941030@nvidia.com>
References: <cover.1714451595.git.nicolinc@nvidia.com>
 <4ee1f867e838b90a21de16b12cf2e39ba699eab4.1714451595.git.nicolinc@nvidia.com>
 <20240430170655.GU941030@nvidia.com>
 <ZjE/ZKX7okSkztpR@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZjE/ZKX7okSkztpR@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?WTZIjq2BEEo8pHOHNYO1NM9xhkdlZ61SbmMCcA0Np/mRGl0ihPEaLnwIotGN?=
 =?us-ascii?Q?HiAUBe3E4b8Avj1Z9FV5hlu0iQvT41BvlYeepC3Dh+A7BnqCatBwG3gOe+XN?=
 =?us-ascii?Q?y0iSiSRFWPznUrlBm8etU1ctbZjCK4uu2EXJCJMBpL9qkifivBXiLhKvmogn?=
 =?us-ascii?Q?cMk9t3mftl3Cy/9U6T5xWkQiUESRj+jnR4f4Bj0oDEX2yrrudl5Js4D6eAJX?=
 =?us-ascii?Q?aTy7Q3aK0CBOSLQ/2+M1QpmJ/kHRYTXC7StjGWLEkG7LJHtXi5X0J9c1etTE?=
 =?us-ascii?Q?PsP1raeZPEm3hvRVt8D253VGKmeNTPAwgMoq42vJmquYbqveaMCtfeVNbkcD?=
 =?us-ascii?Q?JimuZfNOAarxelBiod1kSSKyjFBDo3uW7hdAVjFoSxHNPuwkRv/oz6V2GfOx?=
 =?us-ascii?Q?RFSIf6VF1SFkXrtMzmUqgM57IxRug2JJjuVY+ZBgmuzImTCr3/L7gEaPwbkn?=
 =?us-ascii?Q?AD5f1fqt5haaiEUMktgeRvWRQVBDo19UzapSfl0lMHqMhBkkj8EWY7Dw9ICV?=
 =?us-ascii?Q?rHa7o6Ra4KKsAZs6sTgMMnWgRUBsx+czIBHC0mf3cKo+31xNBY4wFPyZ02jA?=
 =?us-ascii?Q?BNpjTYpJcaArFwOxM489puYAV5DAPMtercXSrd7NvRd1miGRiPZ4G8e/Dcuc?=
 =?us-ascii?Q?BawIFcIE+bVqXZIq5eCmFrWvBqPvL/XNanNHm/WoT5ZrstGTs/NRa/TDDe8w?=
 =?us-ascii?Q?llWNdRe+1nXbO/jHUqsUQm42Bk+M7V7Om4+Rz30mcCHqNuABpkrezD/bPnXp?=
 =?us-ascii?Q?puVQQu0iCmSb78zY92uQCMJ0kSLd8NUTrs/Fb9w9eZJnH8l/JO5xlrLcjegU?=
 =?us-ascii?Q?/0wkvgyyPJHAHAJyvvuo/pdFNnFSychyKzruY21NINvHfgLNOukxZhV3h41I?=
 =?us-ascii?Q?tRf/ILmg+QKTJwNlMVPZdJ0Jrjl8F9DYSYBkkDok/Q4MRGTQoX3JXENJoWXU?=
 =?us-ascii?Q?9d39BXDjCCueJaplGTCrdzdEXvS53Aak/NF5Un8vMvprf3PmZwXjS9r4SbPp?=
 =?us-ascii?Q?mPY/nMvhDbkaj+MrvoKuJTb/Q8cgiocOTKGWmuX1z6M/1FH6LCa9mHqDlsDM?=
 =?us-ascii?Q?veR2M3sfKjOeU2Hsz50PX4Y3yE5SOGBx0GRFiomJN+q2aZpYPl0Z0Ca2E9HU?=
 =?us-ascii?Q?fOimCt7LHIBaFP2e64HxT3uO8uEP8XTBkOGqp61hiU2ZlKyAmyyGNAELcowH?=
 =?us-ascii?Q?iE0VSZonXeJ9aTnegFSDH8KbUxFXH32ZcZQ2Xo6vQjXmRHpzq6lzDfKmZbmz?=
 =?us-ascii?Q?sOuffyWawKx2q641pEF1UyYY4Lbh6uBhdCZ/puYShZrQIKJ5TcylCOSThiQv?=
 =?us-ascii?Q?UckcHHMoB9yni5xBEuVl/P8HnQuRpVrtf4dhL0GjjBeT5BVsTATaJisNcAu1?=
 =?us-ascii?Q?3hoh9m3Vt0pxGrgogDrfDZ41ZvYDhuHByZIN0XOe8gEqvltzdU4gI70c3k8P?=
 =?us-ascii?Q?I+OKPJ32rreCBzYeDnaHLxP+uNS4oF9oUWdMrHF6/5EYZc4uhsg4LZ6AatUm?=
 =?us-ascii?Q?N00CVAVpRTpcDjZMyfs+zWiZhTqmqJTyA4d5ey5tN608C85vi6mlpR4zn+OJ?=
 =?us-ascii?Q?V/BgTXlXWM7PGC/Br+3VZn7mL7iKzco7Z8iAgsw8?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 344c541d-0fda-4887-cb18-08dc69742883
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2024 00:17:59.8308
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NfadC1gT754EY1y+RkUhb9PqzqbwBzIZgbfsUj4WBp1cixjylf4kyFeGlQO72XWi
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8016

On Tue, Apr 30, 2024 at 11:58:44AM -0700, Nicolin Chen wrote:

> > Has to push everything, across all the iterations of add/submut, onto
> > the same CMDQ otherwise the SYNC won't be properly flushing?
> 
> ECMDQ seems to have such a limitation, but VCMDQs can get away
> as HW can insert a SYNC to a queue that doesn't end with a SYNC.

That seems like a strange thing to do in HW, but I recall you
mentioned it once before. Still, I'm not sure there is any merit in
relying on it?

> > But each arm_smmu_cmdq_issue_cmdlist() calls its own get q
> > function. Yes, they probably return the same Q since we are probably
> > on the same CPU, but it seems logically wrong (and slower!) to
> > organize it like this.
> > 
> > I would expect the Q to be selected when the struct
> > arm_smmu_cmdq_batch is allocated on the stack, and be the same for the
> > entire batch operation. Not only do we spend less time trying to
> > compute the Q to use we have a built in guarentee that every command
> > will be on the same Q as the fenching SYNC.
> 
> This seems to be helpful to ECMDQ. The current version disables
> the preempts, which feels costly to me.

Oh, yes, definately should work like this then!

> > Something sort of like this as another patch?
> > 
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 268da20baa4e9c..d8c9597878315a 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -357,11 +357,22 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
> >  	return 0;
> >  }
> >  
> > -static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smmu,
> > -					       u64 *cmds, int n)
> > +enum required_cmds {
> > +	CMDS_ALL,
> > +	/*
> > +	 * Commands will be one of:
> > +	 *  CMDQ_OP_ATC_INV, CMDQ_OP_TLBI_EL2_VA, CMDQ_OP_TLBI_NH_VA,
> > +	 *  CMDQ_OP_TLBI_EL2_ASID, CMDQ_OP_TLBI_NH_ASID, CMDQ_OP_TLBI_S2_IPA,
> > +	 *  CMDQ_OP_TLBI_S12_VMALL, CMDQ_OP_SYNC
> > +	 */
> > +	CMDS_INVALIDATION,
> > +};
> 
> Hmm, guest-owned VCMDQs don't support EL2 commands. So, it feels
> to be somehow complicated to decouple them further in the callers
> of arm_smmu_cmdq_batch_add(). And I am not sure if there is a use
> case of guest issuing CMDQ_OP_TLBI_S2_IPA/CMDQ_OP_TLBI_S12_VMALL
> either, HW surprisingly supports these two though.

These are the max commands that could be issued, but they are all
gated based on the feature bits. The ones VCMDQ don't support are not
going to be issued because of the feature bits. You could test and
enforce this when probing the ECMDQ parts.

> Perhaps we could just scan the first command in the batch, giving
> a faith that no one will covertly sneak different commands in it?

Yes with the current design that does seem to work, but it also feels
a bit obtuse.

> Otherwise, there has to be a get_suported_cmdq callback so batch
> or its callers can avoid adding unsupported commands at the first
> place.

If you really feel strongly the invalidation could be split into
S1/S2/S1_VM groupings that align with the feature bits and that could
be passed down from one step above. But I don't think the complexity
is really needed. It is better to deal with it through the feature
mechanism.

If high speed invalidation is supported then the invalidation queue
must support all the invalidation commands used by the SMMU's active
feature set, otherwise do not enable the invalidation queue. It does
make logical sense.

Jason