Received-SPF: pass (google.com: domain of linux-kernel+bounces-90758-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Mon, 4 Mar 2024 10:53:05 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: ankita@nvidia.com
Cc: alex.williamson@redhat.com, yishaih@nvidia.com,
	shameerali.kolothum.thodi@huawei.com, kevin.tian@intel.com,
	aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
	targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
	apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
	rrameshbabu@nvidia.com, zhiw@nvidia.com, anuaggarwal@nvidia.com,
	mochs@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 1/1] vfio/nvgrace-gpu: Convey kvm to map device memory
 region as noncached
Message-ID: <20240304145305.GX9179@nvidia.com>
References: <20240229193934.2417-1-ankita@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240229193934.2417-1-ankita@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?/CYwH6TinYaf+ojwf5pmhci68wi/I12HBbz/Ocfryt3oU8ufcx0ooynAmxeo?=
 =?us-ascii?Q?HzAO76M8rkjCVaZqpHsqKeguiFyJGR+1kCfpp/GQhuQv9tJe4S4/OYc2q4zU?=
 =?us-ascii?Q?6nCUStQkYLu1a5VIpERd/oyBSLhMMmTpgVHImM2R/+y8IDDyzsdlqxFqFBM4?=
 =?us-ascii?Q?5rmyfcMw0WNdGkd99uVUnxwEV0+mOENYnDz5Q2sCf2aMNF9G4YJiIY5ONi0G?=
 =?us-ascii?Q?g+05F9oQrYQl337jYT0ZT07oEXVYmCNe5LLWSmSYYB+/S0fwY4hmgB+aLvVJ?=
 =?us-ascii?Q?ROfVZSQV/jn4tcFuExGjKlqg/7C8T69ygK8M7wpKeFf2XztMIyquNSZWAhHS?=
 =?us-ascii?Q?H10NiLrss9sG/jEH/CCKkELSlUAKfhg04ZeWMjvORuptteToK9eCAtRwLfM3?=
 =?us-ascii?Q?Vh9H3IkNElez4lGbLoxO5xc7TweTcmcHhPAnV0QmjdNmllZIFzQt+q/acFaf?=
 =?us-ascii?Q?i8MprwCpktbT3aIPe+XsRRn+42+sKwBMk//riIL3nwtr/0bX96qfwk0nmbga?=
 =?us-ascii?Q?7iQKorMSToLhsnavk4FnuzcHEjL7LgsFmewT8LYd6JpD1GWHPuDeV12VN9/I?=
 =?us-ascii?Q?ogxOHiQyiTmtlDZHJz0iOZdyJMdslDS6kxa0V5t6f6EtMdhrI6m2H9wWTo7F?=
 =?us-ascii?Q?EJBWkWyTSrRRrhY27nWre0japyFiMVyReqQJUfggQBjPwy1ug8BxLJ91Rggg?=
 =?us-ascii?Q?tvs045ltWLWCd3n7wpC4qF4NsLyumcsmGVdgnjTgB61zfREI6yapjFt92CGL?=
 =?us-ascii?Q?BkjrnbaeBVDbMD7V6LBskawJOgF6oZIvuX7UDkuWg02lviX0k9jdY315DUJs?=
 =?us-ascii?Q?7blKZphw/b+MCtGnd4CjxaI3e8jR3epVZhxfs6xggh6VaTYnTWwNPOweDOki?=
 =?us-ascii?Q?eilp0Hu/8lExwTtEX9C7jRx/PvNlc0EnRlVD1Zipvm4Lw9Br4DT9EpmHOyKM?=
 =?us-ascii?Q?msGfxepnMf4rQQ0ZD1OW3sOCfT/MTGXx55a+uCIXm/OodBvGtJ4B9WP2bHty?=
 =?us-ascii?Q?Bej3j6NsEEfKKYxdsAqDFTDFcUH5q3p9SdYuHcLS/4GVQTdihKS+B1MbM2NA?=
 =?us-ascii?Q?IsjeWtRgUX8cyuzDJjw3ey4XCufUd71hdNkR5XeewkVx0ybMT4/Q52XO8ypQ?=
 =?us-ascii?Q?wspRc0uPYqL8hG5ToopIdgI4/O/0P1fscK+bVrxXffofzFfj6k0HoT4km4KO?=
 =?us-ascii?Q?/GKz//aVu7z56nSNCGyRigNDZLPMPknBlD0GZBcuZUo+OvBb6VpDfzyfMzD6?=
 =?us-ascii?Q?AHfpI6JG/kY9oLCx1cHiBxIHgq8oiP2t72Ap/9I0MXH9C9vp7GwOz0A3aG81?=
 =?us-ascii?Q?FyQ3GsTPZr+83tbz55DuPgZeOQQzArzdmsASN3sNCUrNqR2IXj94WB0G1ous?=
 =?us-ascii?Q?FSVwZeMTfeigYXOIVRiMj3Jt0sP1u1eDRGH4n/8jrdI3QhWJ1H2Aag9/0zOY?=
 =?us-ascii?Q?qtigG+5HrpcXTR9VpVyGYKMzlqjrTOZH4K7j09guXvXC6+vdFi70Jkf/zWOj?=
 =?us-ascii?Q?yQX7dZeOcktdgBbMtaTvLNDuTEe7zlxupxxHaGGHGhEZzKuv5q1icID3WXLc?=
 =?us-ascii?Q?jU4xEiMB17BhbuSIo0gLTILjO6sdUNW7HBjY/blM?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 949a9a7f-3d35-4565-7ed9-08dc3c5acd81
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2024 14:53:07.4918
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3cncKe7MjPlda19IfA18c4v8SF+l3HsLjVR5oGn+/KR4TnG5G5UUhjCpUCJUA80o
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8750

On Thu, Feb 29, 2024 at 07:39:34PM +0000, ankita@nvidia.com wrote:
> From: Ankit Agrawal <ankita@nvidia.com>
> 
> The NVIDIA Grace Hopper GPUs have device memory that is supposed to be
> used as a regular RAM. It is accessible through CPU-GPU chip-to-chip
> cache coherent interconnect and is present in the system physical
> address space. The device memory is split into two regions - termed
> as usemem and resmem - in the system physical address space,
> with each region mapped and exposed to the VM as a separate fake
> device BAR [1].
> 
> Owing to a hardware defect for Multi-Instance GPU (MIG) feature [2],
> there is a requirement - as a workaround - for the resmem BAR to
> display uncached memory characteristics. Based on [3], on system with
> FWB enabled such as Grace Hopper, the requisite properties
> (uncached, unaligned access) can be achieved through a VM mapping (S1)
> of NORMAL_NC and host mapping (S2) of MT_S2_FWB_NORMAL_NC.
> 
> KVM currently maps the MMIO region in S2 as MT_S2_FWB_DEVICE_nGnRE by
> default. The fake device BARs thus displays DEVICE_nGnRE behavior in the
> VM.
> 
> The following table summarizes the behavior for the various S1 and S2
> mapping combinations for systems with FWB enabled [3].
> S1           |  S2           | Result
> NORMAL_NC    |  NORMAL_NC    | NORMAL_NC
> NORMAL_NC    |  DEVICE_nGnRE | DEVICE_nGnRE
> 
> Recently a change was added that modifies this default behavior and
> make KVM map MMIO as MT_S2_FWB_NORMAL_NC when a VMA flag
> VM_ALLOW_ANY_UNCACHED is set [4]. Setting S2 as MT_S2_FWB_NORMAL_NC
> provides the desired behavior (uncached, unaligned access) for resmem.
> 
> To use VM_ALLOW_ANY_UNCACHED flag, the platform must guarantee that
> no action taken on the MMIO mapping can trigger an uncontained
> failure. The Grace Hopper satisfies this requirement. So set
> the VM_ALLOW_ANY_UNCACHED flag in the VMA.
> 
> Applied over next-20240227.
> base-commit: 22ba90670a51
> 
> Link: https://lore.kernel.org/all/20240220115055.23546-4-ankita@nvidia.com/ [1]
> Link: https://www.nvidia.com/en-in/technologies/multi-instance-gpu/ [2]
> Link: https://developer.arm.com/documentation/ddi0487/latest/ section D8.5.5 [3]
> Link: https://lore.kernel.org/all/20240224150546.368-1-ankita@nvidia.com/ [4]
> 
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Cc: Kevin Tian <kevin.tian@intel.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Vikram Sethi <vsethi@nvidia.com>
> Cc: Zhi Wang <zhiw@nvidia.com>
> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
> ---
>  drivers/vfio/pci/nvgrace-gpu/main.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Jason