Received-SPF: pass (google.com: domain of linux-kernel+bounces-58497-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 8 Feb 2024 13:54:37 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: ankita@nvidia.com, maz@kernel.org, oliver.upton@linux.dev,
	james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com,
	reinette.chatre@intel.com, surenb@google.com, stefanha@redhat.com,
	brauner@kernel.org, catalin.marinas@arm.com, will@kernel.org,
	mark.rutland@arm.com, kevin.tian@intel.com, yi.l.liu@intel.com,
	ardb@kernel.org, akpm@linux-foundation.org, andreyknvl@gmail.com,
	wangjinchao@xfusion.com, gshan@redhat.com, ricarkol@google.com,
	linux-mm@kvack.org, lpieralisi@kernel.org, rananta@google.com,
	ryan.roberts@arm.com, aniketa@nvidia.com, cjia@nvidia.com,
	kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com,
	acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com,
	danw@nvidia.com, kvmarm@lists.linux.dev, mochs@nvidia.com,
	zhiw@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v6 4/4] vfio: convey kvm that the vfio-pci device is wc
 safe
Message-ID: <20240208175437.GN10476@nvidia.com>
References: <20240207204652.22954-1-ankita@nvidia.com>
 <20240207204652.22954-5-ankita@nvidia.com>
 <20240208103022.452a1ba3.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240208103022.452a1ba3.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?r77S6RF0yHkBa4u6f2ffuw3rxX/X8gTB9T/kBpP4oZMl4E8T6scZvzX0r1r5?=
 =?us-ascii?Q?IwWN411/pgy9bNg6CBVxRlxY90JsQihr45+ROtMIDCFjhQJ8taenAsIs6ssK?=
 =?us-ascii?Q?4gIOwvDaQj9Jn+C7v/g/nw7IfLcinpMa/wT4AYe7DYsu7sLR8RuWeUAsbpQD?=
 =?us-ascii?Q?P7mSkFCDRxMvlfpQW8YjcxgfP8nx2QIqk5ZdYgIoRBXKTFTzXzcpRX7r9Jwv?=
 =?us-ascii?Q?YOS8H+qTI7f7I7vinvO8oXV199icdq3Q5IvPRHlIDJQYH66lZ0gmst02nQRk?=
 =?us-ascii?Q?2Axt6fYgATFNR3CfeIvOcfznJypDBgFxUcsXUtheDApGH8VsHZBxhpwSEwkM?=
 =?us-ascii?Q?UsRW6iFurmB9ax3TKFwXWtI+tZDbrkJimURAdnJW87t0nTtM52rIP3jK2qgk?=
 =?us-ascii?Q?GDtZUfvb/+7/danRztAjIMGZh4VXIjYcznmzJ/swSG2Pyk9Ps1nKEXeP2RT2?=
 =?us-ascii?Q?3p08PU5aSQ6K/92iHojLDcYkF0BNDOTf4EdO/HAeH/fDLxgMwdnB1enInW+S?=
 =?us-ascii?Q?MZkJW4c0taU4gpz291qbwaRMfxgYcTDU9jrs86VrILIJxJbe5F0jzKc+X4hv?=
 =?us-ascii?Q?XO6dw5+iYAAY9HCVMKVLFLXCZtEpEWqZW3KEYzLHhhjEXwOqrL8/HBPYdYG5?=
 =?us-ascii?Q?xHQ0EmEs4s5L+UhzEgUpAveeRUPTwxTe9YW7mNc57E+AUtikPL9aQ3f6+8qD?=
 =?us-ascii?Q?42a7no+ZawNPucQ7CHGdYHLch10wNaOAR9LUg0gnPrW79qNmbs8AC+yR8Ltm?=
 =?us-ascii?Q?OqQfb7DifrwHUxyXeWZG0sTQxtkb11O8c2BdiS5HJL0R6wvqOk+VLGUEhffj?=
 =?us-ascii?Q?+olYyNf/xgQE53+yuLhrUkTYj4FEGhVtS0DqhsW8Bp/PjnrQBIA0H5uON+Fi?=
 =?us-ascii?Q?pE/Gde3SiFvBacxHrjhsMY2kiGX048+0AwEN3rxTV4J6a0zgtDUgmiEGWFSh?=
 =?us-ascii?Q?C0BOl9KoNMG9R51tg1v62Bya1tyNm0gzwVwnE1J5pswl2Iydtqxc6wgfsylx?=
 =?us-ascii?Q?YIAkK4OeWapBvesKZdYfopYW50/Fx+jQd5osQmOxKjU6tykVGV1cp9bah5zo?=
 =?us-ascii?Q?AmA+Opge3rmiE1UK9SaGfngJzig/Rc6h+QcHZTfC8zYszKj+PQMsjfceI/Po?=
 =?us-ascii?Q?6dxe0oJ7l7Z0xpOctOLqhZFIx+TZ6ebsPrOBqGAeQo91OFXz4GT6sebHyoLS?=
 =?us-ascii?Q?RrLYnXmE0R9CymhYp3VII3VwbjoYS5gi8+66nSz18crSo9IYXXih2BNRAKbE?=
 =?us-ascii?Q?wH/SuTqGsKsRev9PcfD6DRFG10ZAuelzi4/JqgUHIfXjOw9nv14y7/KVOcT1?=
 =?us-ascii?Q?AvI7IKlmXnM/XPAwoGN4/YE/eXTzLOfLYw2Ul7BaSfa4P6lkGN6clThkbOPw?=
 =?us-ascii?Q?tBoqoajEtd7BUlVMBJUqeg82mnwbEpC7vtLNNPxHBcJIj4/NisBwcdlcTmUT?=
 =?us-ascii?Q?c+YgBmhPHopuZ78RgjzGdFVGS6fxFXs4Qh2PQ3MkjL/QDJdvo1c6xRWA8h1/?=
 =?us-ascii?Q?DjdgkeS+Vc6H1LZAuFHF5NFvcoQTg5dNSq5wfPM3QzC+AfMY2A3tb5l84WxU?=
 =?us-ascii?Q?YFATxeoDW4EcEre20U5+R4tHikE9llKo9VPy+ToD?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 14dfc7ae-81a6-475c-fa85-08dc28cf0509
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2024 17:54:38.8848
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JDzxDXXfzjLbLFSHSyKpd9JhBWIRyDdWPF/tQyfg4X5BmZv18CAtdYvUlHlKjcSg
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4382

On Thu, Feb 08, 2024 at 10:30:22AM -0700, Alex Williamson wrote:
> On Thu, 8 Feb 2024 02:16:52 +0530
> <ankita@nvidia.com> wrote:
> 
> > From: Ankit Agrawal <ankita@nvidia.com>
> > 
> > The code to map the MMIO in S2 as NormalNC is enabled when conveyed
> > that the device is WC safe using a new flag VM_VFIO_ALLOW_WC.
> > 
> > Make vfio-pci set the VM_VFIO_ALLOW_WC flag.
> > 
> > This could be extended to other devices in the future once that
> > is deemed safe.
> > 
> > Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
> > Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
> > Acked-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> >  drivers/vfio/pci/vfio_pci_core.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> > index 1cbc990d42e0..c3f95ec7fc3a 100644
> > --- a/drivers/vfio/pci/vfio_pci_core.c
> > +++ b/drivers/vfio/pci/vfio_pci_core.c
> > @@ -1863,7 +1863,8 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
> >  	 * See remap_pfn_range(), called from vfio_pci_fault() but we can't
> >  	 * change vm_flags within the fault handler.  Set them now.
> >  	 */
> > -	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
> > +	vm_flags_set(vma, VM_VFIO_ALLOW_WC | VM_IO | VM_PFNMAP |
> > +			VM_DONTEXPAND | VM_DONTDUMP);
> >  	vma->vm_ops = &vfio_pci_mmap_ops;
> >  
> >  	return 0;
> 
> The comment above this is justifying the flags as equivalent to those
> set by the remap_pfn_range() path.  That's no longer the case and the
> additional flag needs to be described there.
> 
> I'm honestly surprised that a vm_flags bit named so specifically for a
> single driver has gotten this far.  

IIRC there was a small bike shed and this is what we came up
with. Realistically it should not be used by anything but VFIO and KVM
together. Generic names do sometimes invite abuse :)

> It seems like the vfio use case for
> this and associated FUD for other use cases could all be encompassed
> in

I think Ankit is talking about vfio-platform drivers by "other
devices".

This is largely why it exists at all, there is a fear that the non-PCI
VFIO devices will not be implemented the same as the PCI devices. If
any platform devices have workloads that require WC and have HW that
is safe then they will set the flag somehow in the vfio platform
drivers.

> the comment where the bit is defined and we could use a name like
> VM_ALLOW_ANY_UNCACHED or VM_IO_ANY.  Thanks,

I'd pick VM_ALLOW_ANY_UNCACHED of those two

Thanks,
Jason