Date:   Mon, 20 Mar 2023 10:03:04 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     Robin Murphy <robin.murphy@arm.com>, will@kernel.org,
        eric.auger@redhat.com, kevin.tian@intel.com,
        baolu.lu@linux.intel.com, joro@8bytes.org,
        shameerali.kolothum.thodi@huawei.com, jean-philippe@linaro.org,
        linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZBhZiCOQHAxEaamy@nvidia.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <aa327f9ea61e5a4771c13e53639e33955b9acde3.1678348754.git.nicolinc@nvidia.com>
 <1467e666-1b6c-c285-3f79-f8e8b088718b@arm.com>
 <ZAn7uC9UweiNdGkJ@nvidia.com>
 <ZAqv87fjbdynVaHA@Asurada-Nvidia>
 <ZAtYphmOuEqQ1BiC@nvidia.com>
 <ZAxsgjuuG3ziqs9E@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZAxsgjuuG3ziqs9E@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?V8iqjB0LZzYVdvCVi8CeB+HYmiT5LM6a8kLp+f00ab3wjiU9HLNQ2hi56Ggn?=
 =?us-ascii?Q?U/+/QLLaNvf7LLrAlYFu984SjCQXHnhwXQUpDLgXhdlvKbDM/ywzHZErv6cd?=
 =?us-ascii?Q?/E3o4TOJ2WMwC5DVjUjByQxVt9gFliaANg3ONsEU7K82KtrSNDrkuERFhxm8?=
 =?us-ascii?Q?dJrqCYjxG/tU6NTIJgZuxC/6xvh6UO9OGqVGmcAqQ4+GK4EuNKAPZrR5AHiQ?=
 =?us-ascii?Q?h/2fJEJFzkMkkLO+0BP1W1huWADAhsRZ575M/mfCWAvKitqxtarjbS29LtMa?=
 =?us-ascii?Q?BPSOMv1cy/dQ9NgDVfJFDNSDdEFEfN5coTWk7WoUrukQa2B2zAhoAknoip0p?=
 =?us-ascii?Q?obGLGiBSuOe8NSkypi79lIvOz0pM04sU/5PyGt8QrEUKPDdWPOduut62+5WJ?=
 =?us-ascii?Q?gnZ3GGbOXVJhrrFh9LHLGTHF5UNDK7sXrdIriRrz34WWEygVCp9Ggc9IKSa8?=
 =?us-ascii?Q?62W52MZIUN9fZ7Tf3bHBKLLahFvpy1jP/+XzIY40P0W1uoDgPV7O7d4yFDKA?=
 =?us-ascii?Q?SK3uVeFgUObDG+rWlmyTsXzr5BLk/JTSrVzfLXebyF2F2c0RC6IAwO2zf7od?=
 =?us-ascii?Q?PTbcJXZ9hwWRsW3H3Y89uuOXcbHFXRs3bCpaZFTqha+gNIHbAn/2ibxNry6M?=
 =?us-ascii?Q?w98f23YmnC8g/aFgN9WHGb/Rj/gRAVo8I/6w/E5bVQ2w4UIAlwBEnWKLL+1a?=
 =?us-ascii?Q?3VobgskYY49zfh6j9sm5nER3bpE3+IYxx4x9NFT07ueU394r7AbZnzhOaQDS?=
 =?us-ascii?Q?kLmJXnYR/pGMYKaffqLcfepzM0hg9D+VqQ/cAwpcRxf71I2InuK9loZnCkX/?=
 =?us-ascii?Q?fZlL4nJcFORs9b/zaXeyXt+0eN9yx8dBSd1W9CDKfDBRu0Uns8wwYkLKAXlr?=
 =?us-ascii?Q?859jnO0TpB2GVqxnUnlqJC1+3XNDPwLlaZ34faqEsjnH8U1cR3Wz/LRTNLo3?=
 =?us-ascii?Q?nwL35K8pzLubvCGk7Bhuuf1TKk0sSgOQQs6FFPpsFa2Nzj5QzWhoOknyNU8v?=
 =?us-ascii?Q?LAQF3u+V0pWvq40s+I/dp3Dw+a2dRMeGIb3NazVm17G714N3enQIo8CLdMAp?=
 =?us-ascii?Q?8h+DCwFgPpXkpEQWBopxnNH2xWOtgTOcHpQRVb+wgDUW7QnN4HJrz/W7RoVG?=
 =?us-ascii?Q?U0h/0CRlLbb0QLKl4lURQ10ue8SRDbrd1zr86NCLqt1zFWy7psIhlIwkS6mL?=
 =?us-ascii?Q?3KtcGOSlL2HiisfKANWFCCJ4RWpU2Edu8d047E0XgH95WVDTzhPpCeUR5Qxt?=
 =?us-ascii?Q?A96uTG+4gcyeH8qfKAcHXaa09E8baUPnl5DiSRktXqDXghENUezebEuTLWgd?=
 =?us-ascii?Q?JDgqxpYrS3v65cr1ub4/DfWw+iqr9uGXsNnfXo/18C1/CTQY5zcEMhlGFS/N?=
 =?us-ascii?Q?pLwnp8Hv0/HK0vFJO4WN6jqcZTW0EScbZ/T1K9mJh8cC6BzyXFhMvYb6brS9?=
 =?us-ascii?Q?3BDSwtCoYLjUox0upV8XBkcby+JA10wFroC8pUCz4walIX2jsSBWbxo2a1yY?=
 =?us-ascii?Q?0rVC5ngC9uO+uThqF2wpfGzkC+7b9ofWbImy8VxZs4gbBYIuaCnTcqrmw8ow?=
 =?us-ascii?Q?Wszbxco0csNhK+gs0g3zAh2gpqlSSMbtmh/evnpa?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7139f3d5-e767-405f-8765-08db29437251
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2023 13:03:06.1800
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4QtcgotkxfDrsb0bZOGchrRNTFQwpys2mLkgZYy/1oTO1ZV8+twS/vt05/EcCMul
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8466
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sat, Mar 11, 2023 at 03:56:50AM -0800, Nicolin Chen wrote:

> I recall that one difficulty is to pass the vSID from the guest
> down to the host kernel driver and to link with the pSID. What I
> did previously for VCMDQ was to set the SID_MATCH register with
> iommu_group_id(group) and set the SID_REPLACE register with the
> pSID. Then hyper will use the iommu_group_id to search for the
> pair of the registers, and to set vSID. Perhaps we should think
> of something smarter.

We need an ioctl for this, I think. To load a map of vSID to dev_id
into the driver. Kernel will convert dev_id to pSID. Driver will
program the map into HW.

SW path will program the map into an xarray

> > I suspect the answer to Robin's question on how to handle errors is
> > the most important deciding factor. If we have to capture and relay
> > actual HW errors back to userspace that really suggests we should do
> > something different than a synchronous ioctl.
> 
> A synchronous ioctl is to return some values other than defining
> cache_invalidate_user as void, like we are doing now? An fault
> injection pathway to report CERROR asynchronously is what we've
> been doing though -- even with Eric's previous VFIO solution.

Where is this? How does it look?

Jason