Received-SPF: pass (google.com: domain of linux-kernel+bounces-177034-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.118.232 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C
Date: Sun, 12 May 2024 15:09:25 -0700
From: Nicolin Chen <nicolinc@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: <will@kernel.org>, <robin.murphy@arm.com>, <joro@8bytes.org>,
	<thierry.reding@gmail.com>, <vdumpa@nvidia.com>, <jonathanh@nvidia.com>,
	<linux-kernel@vger.kernel.org>, <iommu@lists.linux.dev>,
	<linux-arm-kernel@lists.infradead.org>, <linux-tegra@vger.kernel.org>
Subject: Re: [PATCH v7 6/6] iommu/tegra241-cmdqv: Limit CMDs for guest owned
 VINTF
Message-ID: <ZkE+Fa4BS+LTxvgi@nvidia.com>
References: <cover.1715147377.git.nicolinc@nvidia.com>
 <062cf0a1e2b8ec6f068262cc68498b8d72b04bcc.1715147377.git.nicolinc@nvidia.com>
 <ZkDo9US02pD6vysO@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZkDo9US02pD6vysO@nvidia.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2024 22:09:32.1646
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 56e391d4-219c-4a46-1569-08dc72d03391
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource:
	MN1PEPF0000ECD7.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5966

On Sun, May 12, 2024 at 01:06:13PM -0300, Jason Gunthorpe wrote:
> On Tue, May 07, 2024 at 10:56:54PM -0700, Nicolin Chen wrote:
> > When VCMDQs are assigned to a VINTF owned by a guest (HYP_OWN bit unset),
> > only TLB and ATC invalidation commands are supported by the VCMDQ HW. So,
> > add a new helper to scan the input cmd to make sure it is supported when
> > selecting a queue.
> > 
> > Note that the guest VM shouldn't have HYP_OWN bit being set regardless of
> > guest kernel driver writing it or not, i.e. the hypervisor running in the
> > host OS should wire this bit to zero when trapping a write access to this
> > VINTF_CONFIG register from a guest kernel.
> > 
> > Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> > ---
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 20 ++++++-----
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  5 +--
> >  .../iommu/arm/arm-smmu-v3/tegra241-cmdqv.c    | 36 ++++++++++++++++++-
> >  3 files changed, 49 insertions(+), 12 deletions(-)
> > 
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index d1098991d64e..baf20e9976d3 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -332,10 +332,11 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
> >  	return 0;
> >  }
> >  
> > -static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smmu)
> > +static struct arm_smmu_cmdq *
> > +arm_smmu_get_cmdq(struct arm_smmu_device *smmu, u8 opcode)
> >  {
> >  	if (arm_smmu_has_tegra241_cmdqv(smmu))
> > -		return tegra241_cmdqv_get_cmdq(smmu);
> > +		return tegra241_cmdqv_get_cmdq(smmu, opcode);
> 
> It is worth a comment descrbing opcode here, I think.. At least the
> nesting invalidation will send mixed batches.

Right, this makes the "opcode" look bad, though we know that the
"opcode" in the nesting invalidation doesn't matter because VCMDQ
in that case supports all commands with HYP_OWN=1.

> opcode is sort of a handle for a group of related commands. But what
> is the group? Minimally it is opcode + SYNC, right?
>
>  The caller must only send opcode + SYNC commands to this queue.
>  The opcodes XX,YY,ZZ are interchangable and can be sent together.
> 
> ?

The "opcode" is intended to mean the opcode of a repeated commands
in an arm_smmu_cmdq_batch struct. And it is based on an assumption
that the driver doesn't and won't mix different commands into an
arm_smmu_cmdq_batch struct. Though it doesn't probably matter if a
batch mixes NH_ASID and ATC_INV..

A CMD_SYNC, on the other hand, is outside the batch struct. And
here is another assumption that CMD_SYNC is always supported by a
VCMDQ..

I could clarify the "opcode" here with these assumptions. Or maybe
we should think think of a better alternative?

> > -struct arm_smmu_cmdq *tegra241_cmdqv_get_cmdq(struct arm_smmu_device *smmu)
> > +static bool tegra241_vintf_support_cmd(struct tegra241_vintf *vintf, u8 opcode)
> > +{
> > +       /* Hypervisor-owned VINTF can execute any command in its VCMDQs */
> > +	if (READ_ONCE(vintf->hyp_own))
> > +		return true;
> > +
> > +	/* Guest-owned VINTF must Check against the list of supported CMDs */
> > +	switch (opcode) {
> > +	case CMDQ_OP_TLBI_NH_ASID:
> > +	case CMDQ_OP_TLBI_NH_VA:
> > +	case CMDQ_OP_ATC_INV:
> > +		return true;
> > +	default:
> > +		return false;
> > +	}
> 
> When I look at the nesting patch it also includes SYNC, NH_VAA, and
> NH_ALL. Are they supported here? VAA is not supported in the HW at all
> right? What about NH_ALL?

In a nesting case, the host-level VCMDQ runs those comands, i.e.
HYP_OWN=1 meaning no command limitation.

Thanks
Nicolin