Received-SPF: pass (google.com: domain of linux-kernel+bounces-87085-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: Ankit Agrawal <ankita@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>, Alex Williamson
	<alex.williamson@redhat.com>
CC: Yishai Hadas <yishaih@nvidia.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "kevin.tian@intel.com"
	<kevin.tian@intel.com>, Aniket Agashe <aniketa@nvidia.com>, Neo Jia
	<cjia@nvidia.com>, Kirti Wankhede <kwankhede@nvidia.com>, "Tarun Gupta
 (SW-GPU)" <targupta@nvidia.com>, Vikram Sethi <vsethi@nvidia.com>, Andy
 Currid <acurrid@nvidia.com>, Alistair Popple <apopple@nvidia.com>, John
 Hubbard <jhubbard@nvidia.com>, Dan Williams <danw@nvidia.com>, Rahul
 Rameshbabu <rrameshbabu@nvidia.com>, Zhi Wang <zhiw@nvidia.com>, "Anuj
 Aggarwal (SW-GPU)" <anuaggarwal@nvidia.com>, Matt Ochs <mochs@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1 1/1] vfio/nvgrace-gpu: Convey kvm that the device is wc
 safe
Thread-Topic: [PATCH v1 1/1] vfio/nvgrace-gpu: Convey kvm that the device is
 wc safe
Thread-Index: AQHaan8STeXbyerayU2Kw+zSatmIBbEhesSAgAACnQCAAAUMTA==
Date: Thu, 29 Feb 2024 16:29:04 +0000
Message-ID:
 <SA1PR12MB71990CC60F96FEBFA9CABC58B05F2@SA1PR12MB7199.namprd12.prod.outlook.com>
References: <20240228194801.2299-1-ankita@nvidia.com>
 <20240229085639.484b920c.alex.williamson@redhat.com>
 <20240229160600.GJ9179@nvidia.com>
In-Reply-To: <20240229160600.GJ9179@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
msip_labels:
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=nvidia.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: SA1PR12MB7199:EE_|MN2PR12MB4406:EE_
x-ms-office365-filtering-correlation-id: bdec695d-8e8c-4b71-e4e1-08dc39438b72
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
 hm3HR27jKXDZynHbXwrVd0p9UocD8FxnLzgpUK+JkrDb/Cp3CeXzA0FV9b8Wn/j24bAxziO06uYa9ohPCMQ41rr5jqyFGIzak1u7mIF6pYNpZ6CYjxfcQjqxx8EApQ2gyC7t87uXKfEB4BOXkAMCwxUdoKF60ZIXvgPpvfTqoIl70yXR2mCsjyRTlIAOiDQj16iD2ciczhz0Im7Y9R6iWmIZyLYb3gGX8c+HZZUse/u/BF5+4/XOnLE1YZz27olk2HoSBiQz/srAKOxu2//dTUOrLY+mQnk880n1v9Mil91Evmw9OpwQyOIAypi6iv0UuH2Ak6enUeid1epLG4ucVZpPjr9P3R01bGCBo+4PfUadVEgoiIx3eyU4klYwvqEBVaLaQnN/Eu0ZkD7CxW3CvaZhIbR4/mluOa0Vs8H9KYajvD9Myj1nYf30OBaH0fklvr/U7izHptiPytNBPKjcWuvuF5s5yGMC1fLi9VnT0gQIDVQnNjz+YyCj4q20NEFDgDPp7mXQhLScotyOCvUG+fS7fMpvz76Q+N5/zjBsBDHVD6ame2enP+40JUYGTgSO2qcHxTZiMwzC28AFloEeL/5BjOXiONUn2F/lBIj266gYAYBv2ktJkGlOmYnzrKVFM3Xr5xTT4WWqD094Z6Ud3GsSXBW/bKFpnaC/eXELaLM0UxLs795ie19PVBpPzomcaPde39iCyFK8iwakviGGX50Mzw8jxumRPs8JQWnVySI=
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA1PR12MB7199.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(38070700009);DIR:OUT;SFP:1101;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?iso-8859-1?Q?GwYtnkjb8nZLDjAr/wlrcKynY5Z94gxipXw0K87V+DrTKVRfvNJSflrU3k?=
 =?iso-8859-1?Q?Cywv1hN3Xkqa1WRtrYQZB8BJH9YOPpVV2UbQ5PnildKr1dbrilHNDYbvq0?=
 =?iso-8859-1?Q?/hej4qt/ezQwu/L3RyOHCmp2D5t/wu+1EoUiSlBddPKcOX/Gnzzt67XDfr?=
 =?iso-8859-1?Q?Ijl45Y7XQ/8fjlrQlTYp08BrMCuNQfb2ruXsqRj1c7VYvC3uytGuK7BMhC?=
 =?iso-8859-1?Q?GK9T4l1E265SK96Z6tCaGYSkChF0WbHZzdyIZq9FJt9V9gWbpucAFkh3zV?=
 =?iso-8859-1?Q?Y7cE5ydr2jSi1FIFHGo+pxjFxhDJu2//IyZoeuLELF/I6qVFzUjWmJNohY?=
 =?iso-8859-1?Q?FC8likAAEansbtkXP3acLggsFsJusHsZAKYMlJNhD62beXMhu6YYyFd1th?=
 =?iso-8859-1?Q?sa87QDoeALyN3Ahdu+pivNFk4c94hXYHTawYA4pYo2Yq4BGzVgXwC3ghAV?=
 =?iso-8859-1?Q?KXtVgk6cTY+C0UPAEgppl1oFbNP7YNmv374jeEVJ/rWP2kffMwf5nWlrhj?=
 =?iso-8859-1?Q?Sfk1/hhaeyqFwRwWtNEcbojfeV9JcntaUuOngSoDjtdJdKlMwN3ITJ2oKK?=
 =?iso-8859-1?Q?pPaHOZcsmhNY5qS5Tb6FS2kSfyCjAZUl759/BLn16oJvAqAJkjGsBk24Hw?=
 =?iso-8859-1?Q?5koY/Sqxej+Y3uO10GNBZdEQrj35Qqw5On09t3WW+Aco229mL6zeweOMoN?=
 =?iso-8859-1?Q?h5zhRreGLes7wnulWb25qCLG5598ReAiSu/sz6i3N+uDZwC5Zq3PSX281a?=
 =?iso-8859-1?Q?+D8F9glZHJRju6LslSupTqFF9CsCp91A4aKVgK3N0ZFnmjxep5MQZoXRLG?=
 =?iso-8859-1?Q?Dt5dGq0Ec2nFTr+fh98cEMgmQzxLjHa8aFnePSrtZ2+4izgJP/TV+jSqWs?=
 =?iso-8859-1?Q?m3bneNM45TxXaondAMif2UJeOSVzl7q7c5oZArofiqtNy02pnMlWPrzk7U?=
 =?iso-8859-1?Q?5RgFg4WzCUuCMSXwEmdlQFOtIfSMJuwMx5FK3ZdIW5QK1qIXodhbB+I0v/?=
 =?iso-8859-1?Q?VJFCGgw8MVrINOwowGeLIrdtXYQ8AJGv4Y6LycYmXD2dE7x500CXf+Wo6v?=
 =?iso-8859-1?Q?kW3mZfDyOZhJ3y86tlAG53O/KRBpjgcFW/XD2E8sjd18Xl55Dtkk7ZqaFA?=
 =?iso-8859-1?Q?FtctJbMIRmRCTYQMJPvyt7GtKvR8cy1CJyy9Nch7n+En2UuGX+6HXJuSD0?=
 =?iso-8859-1?Q?7xTngtYiH7f3Bp04cdPJMaCM9LRe+6dWZ2gnh9l/+82eKZJBa42ZczHBmP?=
 =?iso-8859-1?Q?BGjIv9dHMwv//iakqWObY+wxTQOZGi3MUam7ZlKlxf3xQ/NtvXBDxe2j8K?=
 =?iso-8859-1?Q?sDF+kSwNgqMaVT8DL5ot9JwBRqgurWZ6oeujOWQXdrrmo8iArSlRB9LrG/?=
 =?iso-8859-1?Q?YmZ/15ucglCAV7JIrdsCxq3MLGWWPoKM0zyCawyGFTDnsc7iroKhefmX7F?=
 =?iso-8859-1?Q?tB0q7RCfzAjoRecAdxRIv4uwv4rP4UzwsmPjKRMTPU18J3FOLGE5RBx3Yt?=
 =?iso-8859-1?Q?TSjma46+U64ZHCGn87DGhTVfBhsF1hAkRNbdEmpS0rtKLql+cMygJzvV1e?=
 =?iso-8859-1?Q?iV9YnZupv2fVAL3aS5Pg9V2s39JC1zvbfPZ6BRAw4DLL+XULxRU8QwfG/y?=
 =?iso-8859-1?Q?WUK5kb/MNqGDc=3D?=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR12MB7199.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bdec695d-8e8c-4b71-e4e1-08dc39438b72
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Feb 2024 16:29:04.4890
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: GpJbdJeY+J1wYM3yj5wEVbsscxQdvopSbMvw1Czfk4HWqBTIkttHwy+zTYYmxTyz6zjcUdofJxBWrD7VGgiD1g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4406

>> >=A0=0A=
>> > +=A0=A0 /*=0A=
>> > +=A0=A0=A0 * The VM_ALLOW_ANY_UNCACHED VMA flag is implemented for ARM=
64,=0A=
>> > +=A0=A0=A0 * allowing KVM stage 2 device mapping attributes to use Nor=
mal-NC=0A=
>> > +=A0=A0=A0 * rather than DEVICE_nGnRE, which allows guest mappings=0A=
>> > +=A0=A0=A0 * supporting write-combining attributes (WC). This also=0A=
>> > +=A0=A0=A0 * unlocks memory-like operations such as unaligned accesses=
=0A=
>> > +=A0=A0=A0 * This setting suits the fake BARs as they are expected to=
=0A=
>> > +=A0=A0=A0 * demonstrate such properties within the guest.=0A=
>> > +=A0=A0=A0 *=0A=
>> > +=A0=A0=A0 * ARM does not architecturally guarantee this is safe, and =
indeed=0A=
>> > +=A0=A0=A0 * some MMIO regions like the GICv2 VCPU interface can trigg=
er=0A=
>> > +=A0=A0=A0 * uncontained faults if Normal-NC is used. The nvgrace-gpu=
=0A=
>> > +=A0=A0=A0 * however is safe in that the platform guarantees that no=
=0A=
>> > +=A0=A0=A0 * action taken on the MMIO mapping can trigger an uncontain=
ed=0A=
>> > +=A0=A0=A0 * failure. Hence VM_ALLOW_ANY_UNCACHED is set in the VMA fl=
ags.=0A=
>> > +=A0=A0=A0 */=0A=
>> > +=A0=A0 vm_flags_set(vma, VM_ALLOW_ANY_UNCACHED);=0A=
>> > +=0A=
>> >=A0=A0=A0=A0=A0 return 0;=0A=
>> >=A0 }=0A=
>> >=A0=0A=
>>=0A=
>> The commit log sort of covers it, but this comment doesn't seem to=0A=
>> cover why we're setting an uncached attribute to the usemem region=0A=
>> which we're specifically mapping as coherent... did we end up giving=0A=
>> this flag a really poor name if it's being used here to allow unaligned=
=0A=
>> access?=A0 Thanks,=0A=
>=0A=
> Yeah, I sugged to fold that hunk into this:=0A=
>=0A=
>=A0=A0=A0=A0=A0=A0=A0 if (index =3D=3D RESMEM_REGION_INDEX)=0A=
>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 vma->vm_page_prot =3D pgprot=
_writecombine(vma->vm_page_prot);=0A=
>=0A=
> So it makes more sense. VM_ALLOW_ANY_UNCACHED shouldn't be used on the=0A=
> cachable mapping. The comment should be more specific to this driver=0A=
> and not so generic:=0A=
>=0A=
> /*=0A=
> * nvgrace has no issue with uncontained failures on NORMAL_NC=0A=
>=A0* access. Tell KVM to open up guest usage of NORMAL_NC for this mapping=
=0A=
>=A0*/=0A=
=0A=
Sure, I will restrict this to resmem. Also will update the comment accordin=
gly.=0A=
=0A=
>=0A=
> Jason=