Date:   Mon, 20 Mar 2023 09:59:23 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Nicolin Chen <nicolinc@nvidia.com>,
        Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Message-ID: <ZBhYq8o4gqFZVb6k@nvidia.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <aa327f9ea61e5a4771c13e53639e33955b9acde3.1678348754.git.nicolinc@nvidia.com>
 <1467e666-1b6c-c285-3f79-f8e8b088718b@arm.com>
 <ZAn7uC9UweiNdGkJ@nvidia.com>
 <ZAqv87fjbdynVaHA@Asurada-Nvidia>
 <ZAtYphmOuEqQ1BiC@nvidia.com>
 <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?PFSYoYnShXE14aJgNdWzocLqxQc8dv90uUJ8YlRfmPwAtIVwQsYmAZitvzBt?=
 =?us-ascii?Q?bGg5e/T856m3vzXaKKMuR0sYu4g3cHV9k60oOMOJn1vkLO24JC92/pakt85Y?=
 =?us-ascii?Q?TpcZleI9UD3eOGNg/95CBXrxSE5wzM31LxiVYIyOchTXX41eZiRmSrAyGUSu?=
 =?us-ascii?Q?go9so2UOlQXoCF9mV6Ql0QOeVYfUuWjqUNTIzzO3K9OAQ2cZYFHKvxAWpFRn?=
 =?us-ascii?Q?YGpIJDgrE1+avyW48KaNabKlcRWID9Or1yeKIc3YiK47G86b+thdFZvEtoyH?=
 =?us-ascii?Q?jsIVwyFP2cwIgqWFqZc3dtd8RAH55BETccIw4mWlpU+a8uASvFC5Kneo29v9?=
 =?us-ascii?Q?d4LFmMDWqH2mbcB0/UJmaxwu/OQomLj0477owXCs5cjlb0pKx1/wMk5+bfhM?=
 =?us-ascii?Q?9a4iaZn4mHN7CClCx7EBfzYZhd1+BOgS6lF5IA4ebJtsGdC/tNgAROz9wTnY?=
 =?us-ascii?Q?Y1SC5+Qk9OzdfVk2KE44bG3ItykSNNwf+D98qx9N4kaVDJY9ck3F48LKz0sR?=
 =?us-ascii?Q?wAdbLmluq0rqsDZo1Z/Hcmt1xdnO5WQQuCdx7MzIV/LUpkuvS6W0w1ER0nhX?=
 =?us-ascii?Q?U9jzLd4msju7UjsZj2BfP0TLrukcYIs+Wg6bjyZEY68DWWjO95pujQYouqwL?=
 =?us-ascii?Q?tqKgA9PxnI/aJ9RnnMvkBTHJFPsN0bQIWVcUABMvRfSVMWruvR1oUzt5m0XP?=
 =?us-ascii?Q?NJfmkHv0eBCydJlGbxbKIdIp2boFmI6mSl6LwHCtSe+fxUnQRqBw9timCvg2?=
 =?us-ascii?Q?macAJcjTMTkr28IVv6FZ+haBC9O4pQKzw/B/+82mqVE+O1ThT4jh1esjb2yX?=
 =?us-ascii?Q?EPJcRKNoSP0NfC7G9BpOLVfxyT6bWNn3IP2vPs7dGeTdNcOV6ccnXTp1OSM1?=
 =?us-ascii?Q?m2xkax0lt6LPKUWMqFpKChX3U2j9pNgtNyPw7dQq/9b8izgcD4uj25bSgbhn?=
 =?us-ascii?Q?fbkOPT2DVQlVrAqQA/zaSvuCqjlwEEYYVO4WkmNMT7XDov1LV7AoShdZbB5b?=
 =?us-ascii?Q?JCAheah/QVlRAu74B7/7Qa5dk3jeT5dqI1rb2Vz5B00vseh/od9EvBAy55tb?=
 =?us-ascii?Q?PgPqZ6GdYx/F/xUdgQAR+hIAVAW1A9J+eHy0noGo6vauyaVYy4NnHoKJiSws?=
 =?us-ascii?Q?HTBRKcVBYrUBOJuNi0+BAjZ5c9trWIYFAKU4TiZTWxxRgUWFI9zgfnxmREO9?=
 =?us-ascii?Q?6v8ToedrFoVUdSBG4rEuKU9z8LMPrUUylIHXJx60c8XNuB7p3eCrDk2fX3hf?=
 =?us-ascii?Q?c/PAK2wO3qwX/J1J1oybCgDF/JwUQhNfb++bq34STV/mp9Ed+Jixykfi9l9t?=
 =?us-ascii?Q?YymoL1svKupydaPRcYO5Je85e+ntz8gcGkG5nDIMU1ltKMnsR0XOp5mLfuw+?=
 =?us-ascii?Q?zkjV+00a9z2UUuDVg8pEvvoV7zvkfwT+1aJkVxg1s54N/KKDzJp8kBzvvHch?=
 =?us-ascii?Q?Yx37qtQnHUmuIZgsWp0CwFWHaUFZLLznKAXnRWHJjzN28t2S4kSUORPOXfbv?=
 =?us-ascii?Q?UgSSNacEs3RtmRUs4NFvyEuI7KUD4SnJ2uC27cE4CeDQDQpv9sc6wN2uvEU3?=
 =?us-ascii?Q?QzYKxRyvpSYHIgrtvs8IHVtsDlMFphTZD0s6Ema5?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7c4865ef-32b5-4e9d-d267-08db2942ee68
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2023 12:59:24.9046
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pZ+kpNqRqytg2W5MGHmzCwjbrrjP9ElNknmS/rvV+uPzIdAib/00cjoyNobusqAw
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7030
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Mar 17, 2023 at 09:41:34AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Saturday, March 11, 2023 12:20 AM
> > 
> > What I'm broadly thinking is if we have to make the infrastructure for
> > VCMDQ HW accelerated invalidation then it is not a big step to also
> > have the kernel SW path use the same infrastructure just with a CPU
> > wake up instead of a MMIO poke.
> > 
> > Ie we have a SW version of VCMDQ to speed up SMMUv3 cases without HW
> > support.
> > 
> 
> I thought about this in VT-d context. Looks there are some difficulties.
> 
> The most prominent one is that head/tail of the VT-d invalidation queue
> are in MMIO registers. Handling it in kernel iommu driver suggests
> reading virtual tail register and updating virtual head register. Kind of 
> moving some vIOMMU awareness into the kernel which, iirc, is not
> a welcomed model.

qemu would trap the MMIO and generate an IOCTL with the written head
pointer. It isn't as efficient as having the kernel do the trap, but
does give batching.

Jason