Date:   Mon, 20 Mar 2023 13:01:53 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     Robin Murphy <robin.murphy@arm.com>, will@kernel.org,
        eric.auger@redhat.com, kevin.tian@intel.com,
        baolu.lu@linux.intel.com, joro@8bytes.org,
        shameerali.kolothum.thodi@huawei.com, jean-philippe@linaro.org,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZBiDcYwxL7eV1EmQ@nvidia.com>
References: <ZAqpOQ+DDvEfq0Dg@Asurada-Nvidia>
 <92fdb06f-e5b1-8534-fb0e-ad47b5be9e1d@arm.com>
 <ZAx2K08L5TIm6r3y@Asurada-Nvidia>
 <ab762cc6-5adf-2515-d9d2-d21d916eb6ad@arm.com>
 <ZBJcS07G3mt7gjkA@Asurada-Nvidia>
 <c753c2a8-024d-5bef-8987-96582084991e@arm.com>
 <ZBOFcenanpRsGFzF@Asurada-Nvidia>
 <ZBe3kxRXf+VbKy+m@Asurada-Nvidia>
 <ZBhbmqprnux4q00i@nvidia.com>
 <ZBh7hSX5hdW4vxwh@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZBh7hSX5hdW4vxwh@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?046HdMFVxTnOGAS2vV4cd17E3ytDEleMvzbhJmEZ2CXin7QD8WnfynWCW8GQ?=
 =?us-ascii?Q?1A+ZpF/OaS6vrDYCwGyyNaFTi6SRLp5M5HLJB64hj/8Yqe6zExjsoEvu35eb?=
 =?us-ascii?Q?fmiXro2QMBOKQvttJQs1lM4I/EIvIQYJnHkQMCqbcrycvXNpCHBoifwwdZE/?=
 =?us-ascii?Q?fLpyyLkuRYEVj9uKrsEj+FZ1WE7pH8w/rDUygyzeKhENQwuMxT3XbCOtxvj/?=
 =?us-ascii?Q?Acgl3xCQVuObnKog5zefYR+joS+uf8uXYZZi4ZZsurY4Kc74U1pOB4juqfod?=
 =?us-ascii?Q?/IlfASDQXU/4rMZcD/noVcDhZCvCmltBGL4dxqAFkMyyf3w53lOn/W0g7C8S?=
 =?us-ascii?Q?2TCaQsuOnTyqM8KXYu6PPiHGtTwt+hnz2HMZ0EVbGNMFa0QfV5AmPvWvM3QE?=
 =?us-ascii?Q?1sEysn3yWfiUbzIIMTap82cKPp7AfaxJr6eHfECLt6vIRMdrN/Kt9SBeGxbS?=
 =?us-ascii?Q?tluCwg4ndOqskfq77K8Wm42iqQeEqHI/Jg5t2zvrPvlNGKzD0ZGcJo0YjJUp?=
 =?us-ascii?Q?Yv7TPAGkF5f51msJ1IM8yFlVDYqzzne8OP8Hf62TdVBHLCxJCbSzPCD9R9c5?=
 =?us-ascii?Q?hhA8EToLVBImDYbj1Me5VHp69eByRfS99KCzCGcAQNMS+B7AWEsr3ZYlSwJP?=
 =?us-ascii?Q?BvIcZxTU3EicAvlEmdu9bohddYaZZanttWr6kjOnawvUk5J0S7RzOSwNKz32?=
 =?us-ascii?Q?2FaHSLcmJKaR5zbh3UUjSQTr/AojpqLdDZANIlvGJgVXyrpYnakL/QhFdGuf?=
 =?us-ascii?Q?tNCd3vAueeKv/y4W+QeDbpHWCBFuGc+Qn341ZLl5d+fRryZG/VJXkhEqMJy/?=
 =?us-ascii?Q?95tR74tkOr6pI3OMZKJBedxCJjbtqv7B6Y42KL4oEserlSpZWQAKdPtn3s+V?=
 =?us-ascii?Q?+e7D/XIgGNsf2rsmrWBUepHmzfSrZilyD47nuyu3+KYa4k9vRotpo9hRZnfo?=
 =?us-ascii?Q?lbOMqejV5ynyq5xNQ591fYlVsVg4TFapvLWTxHxXogUNChrRjo8ID5rzS760?=
 =?us-ascii?Q?SrlGXlq/SZiMIxxVl1k2V8kYC7LjSZr1z8/DfEdkM6xw8m4bc96F2NVPxYo1?=
 =?us-ascii?Q?PW7eD3HqsFv88DUQgVXD6DxNtYw4SGv2f2kKC2Hl5bNXsS82gE7V7gYqbvwP?=
 =?us-ascii?Q?L+k5XMVLOLpFm3CpPE4hU/MNVO6yHsyBTjgamBbb7v9RA/4ttOi3WfHNyBJ9?=
 =?us-ascii?Q?dIMLADqfiP89PqyCqsz79MKGBwwSxe3LiCObO+xwb3O5crEPB9GKG4pCfp5I?=
 =?us-ascii?Q?luZhaNPeWjvlnnir1SntFD9z1DzrJj4RaP6zHyeF6LokJydoZVzom+FIhB3c?=
 =?us-ascii?Q?y24boJy6iLin8kBYbo2HrXaQE9cXissyZp+ZgoHjKUv3bs1nz9k/5a7RgEr4?=
 =?us-ascii?Q?uFQHLoBADJ0uG37x6WcqqeJDIIKpmedbIMnnlcNBKOLwIafHqhgaoAijr0cW?=
 =?us-ascii?Q?0twRZhW0XzNcvPFiKP9HbzEEK+bcBuWIJgYUpFyIplcnRzcbOmNRLDiXTotE?=
 =?us-ascii?Q?5XTn3RjtSYyF9Ql3CqYp3CK8q1N0gxB+elcZmk27Cal4+gV8P4+s+Yo3bm1m?=
 =?us-ascii?Q?qQL/fpSSh3DbsDl2xcFETHYW0eL1GOjnIH6HQ0VQ?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 91e77002-1a94-403a-7ff2-08db295c6d1f
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2023 16:01:54.9442
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8fIOlxZdxslK655nGI2RGwNRV9eaGwB5RZKUwGoKPVuIgnHkvMbNwZdvUG7VkY1E
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7214
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 20, 2023 at 08:28:05AM -0700, Nicolin Chen wrote:
> On Mon, Mar 20, 2023 at 10:11:54AM -0300, Jason Gunthorpe wrote:
> > On Sun, Mar 19, 2023 at 06:32:03PM -0700, Nicolin Chen wrote:
> > 
> > > +struct iommu_hwpt_invalidate_arm_smmuv3 {
> > > +       struct iommu_iova_range range;
> > 
> > what is this?
> 
> Not used. A copy-n-paste mistake :(
> 
> > 
> > > +       __u64 cmd[2];
> > > +};
> > 
> > You still have to do something with the SID. We can't just allow any
> > un-validated SID value - the driver has to check the incoming SID
> > against allowed SIDs for this iommufd_ctx
> 
> Hmm, that's something "missing" even in the current design.
> 
> Yet, most of the TLBI commands don't hold an SID field. So,
> the hypervisor only trapping a queue write-pointer movement
> cannot get the exact vSID for a TLBI command. What our QEMU
> code currently does is simply broadcasting all the devices
> on the list of attaching devices to the vSMMU, which means
> that such an enforcement in the kernel would basically just
> allow any vSID (device) that's attached to the domain?

SID is only used for managing the ATC as far as I know. It is because
the ASID doesn't convey enough information to determine what PCI RID
to generate an ATC invalidation for.

We shouldn't be broadcasting for efficiency, at least it should not be
baked into the API.

You need to know what devices the vSID is targetting ang issues
invalidations only for those devices.

Jason