Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Date:   Wed, 6 Dec 2023 15:03:56 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Marc Zyngier <maz@kernel.org>, ankita@nvidia.com,
        Shameerali Kolothum Thodi 
        <shameerali.kolothum.thodi@huawei.com>, oliver.upton@linux.dev,
        suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org,
        ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org,
        lpieralisi@kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 1/1] KVM: arm64: allow the VM to select DEVICE_* and
 NORMAL_NC for IO memory
Message-ID: <20231206190356.GD2692119@nvidia.com>
References: <86bkb4bn2v.wl-maz@kernel.org>
 <ZW9ezSGSDIvv5MsQ@arm.com>
 <86a5qobkt8.wl-maz@kernel.org>
 <ZW9uqu7yOtyZfmvC@arm.com>
 <868r67blwo.wl-maz@kernel.org>
 <ZXBlmt88dKmZLCU9@arm.com>
 <20231206151603.GR2692119@nvidia.com>
 <ZXCh9N2xp0efHcpE@arm.com>
 <20231206172035.GU2692119@nvidia.com>
 <ZXDEZO6sS1dE_to9@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZXDEZO6sS1dE_to9@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?10QolA+TLjWeK+LcibziUc+KVOZj26TphedV+k2oKSFcSv6BrviLLXP1jSUB?=
 =?us-ascii?Q?XsBvvGC3IZ7z1z70yhPteuJDS+5NMCW11hTBNeDHQl8icgaegb+jPUO6njBV?=
 =?us-ascii?Q?NzUPnE1VuoqjQYm2Qul9LQP9FCvLPFBgMgwavPqt9v3McnfJ04z9XqQUxxw9?=
 =?us-ascii?Q?XneBZdzJfQb+x6jsfUZEABQPqjw6awPm7gJJGBZeSez5zBEXoj9H51FrgIkT?=
 =?us-ascii?Q?3Oxoc7pczOUfO1KiLtLzBnWW9LvV0eSMuhEH6jaxmuuegi2OTvonYogfAH0l?=
 =?us-ascii?Q?P+DczJ9Od106jPudbjSvmiQPqnbFn9ZkYYQSeBcfyjuiNcAvfSCVhgBO2uqo?=
 =?us-ascii?Q?ojUAwMo6kfiMqnOdhDnE9Ou2GagDNfBGLsvlQjHv2AHbdFtEmzQ3ozVw09rL?=
 =?us-ascii?Q?ttil+q1VTsyGW7P+sPk5XSEqo0bYqJLihMlOFiZQ1yOKU8edMDvGv86tkTkM?=
 =?us-ascii?Q?1WuLtKfLnNM4c4av4NqRsoK6tE+MGSByETrosBQU1Bc0wkS0vzE6Mn3okIpn?=
 =?us-ascii?Q?yBLMxIX0eFgYXUbaKTi0yDz9TRnDmGv7iFlfN/PTAJ3SuDxUzjYQmF3wVfgC?=
 =?us-ascii?Q?5+M2AlJxHtEJey0Zf/VeaYiN18KTYpfaYY4haS34RsFrugWepx4iz+2XvHjk?=
 =?us-ascii?Q?qxGIkfDfl4gkkUPcz2AbgOLdSykFThzm6Fnei9lJANIpwhvKs2XRPxQsNaIM?=
 =?us-ascii?Q?yLyhpaxVfxwd1EKxXl7C5FuNwwLvIsEJ9HUbuu9fZ8M6JJBAGE22mC/5myrO?=
 =?us-ascii?Q?/ZoVTmTFkvUygsZEn/ozfYDNAEUWPlvQRkPp6jtYBhaCvRWyhnbmZ4n2QNAe?=
 =?us-ascii?Q?aJAZgyiLVdfmilV1UXVyAwPKdCR4bF4UjNnhUJ/cBmw9BQ225OOtmVSv6dIf?=
 =?us-ascii?Q?JOoBhmYaHQ2mu20A7z783d0pYk7nUYvABbAB+GvxfMGs6SyTXfe49QZxbCUB?=
 =?us-ascii?Q?cfk0LQDsQDLkCg2CMXjbsyTwtutghGmII2bwRf8UfwM/BeBm/uuQUN0kOUwC?=
 =?us-ascii?Q?+/wsjWZLtBk+YV/PF90mBO6MZi4yhjWKQ8tyvPEmlT1o9NC8x1i/kajVQsZh?=
 =?us-ascii?Q?JZDixLDfC+AglDtykVf0gXMZolDc4YuIq9LSMhW4o7sfhuC1E9Xom+xtj0Ci?=
 =?us-ascii?Q?/XMd+CL2093mI2I9ePnQH2N9NHTqULb80v8JNKmQbREeMxoce0+vc8K6OVxR?=
 =?us-ascii?Q?9iP1y0DAREnQuYl+9/OpjS8pCA6L/k9oUTEuxsOXClNKkpGKWA24UDxLQ+3s?=
 =?us-ascii?Q?pw0866q2COdxo8j/K7CjtZLLqpPxa6TH1+BM2uRYvFv4szUKKeUrKSIAWRAI?=
 =?us-ascii?Q?z3PfdxNIZx8GmJBadhnl9jIruNM9xzzfq2hKvGnXTp3gz8zeGbUyvDB81Vtr?=
 =?us-ascii?Q?foPnBbtBSxs3Sc2KmspLHcbku4/xJXK8oTUhvYDn9NhAlcZwE83P0REG4Ob5?=
 =?us-ascii?Q?jV1LC0pCObaEtzIyXtUmdSFV22n4jAPlVd0AfzUK7UFjk8Jp6PqV+HPF1IhV?=
 =?us-ascii?Q?LY+rn5v6qbsbjxKIXfDU8Seyb3ohxlkeNwW8MVuvwutj66M44FDCLbBHUaXg?=
 =?us-ascii?Q?LuQ5qLFQ09pMlm/DrDw0jpKbBpfJuX6/vJC360zV?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1d08d957-f149-4023-8a02-08dbf68e194b
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 19:03:57.4940
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pFknsj/jYuGbD5Z1yEixkeLsw7qJbJwSVWhcZVinPfzE0TN0CdC72R7EVFvDLEx/
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9335
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 06 Dec 2023 11:04:37 -0800 (PST)

On Wed, Dec 06, 2023 at 06:58:44PM +0000, Catalin Marinas wrote:

> -------------8<----------------------------
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index 1929103ee59a..b89d2dfcd534 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -1863,7 +1863,7 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
>  	 * See remap_pfn_range(), called from vfio_pci_fault() but we can't
>  	 * change vm_flags within the fault handler.  Set them now.
>  	 */
> -	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
> +	vm_flags_set(vma, VM_VFIO | VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
>  	vma->vm_ops = &vfio_pci_mmap_ops;
> 
>  	return 0;
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 418d26608ece..6df46fd7836a 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -391,6 +391,13 @@ extern unsigned int kobjsize(const void *objp);
>  # define VM_UFFD_MINOR		VM_NONE
>  #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
> 
> +#ifdef CONFIG_64BIT
> +#define VM_VFIO_BIT		39
> +#define VM_VFIO			BIT(VM_VFIO_BIT)
> +#else
> +#define VM_VFIO			VM_NONE
> +#endif
> +
>  /* Bits set in the VMA until the stack is in its final location */
>  #define VM_STACK_INCOMPLETE_SETUP (VM_RAND_READ | VM_SEQ_READ | VM_STACK_EARLY)
> -------------8<----------------------------
> 
> In KVM, Akita's patch would take this into account, not just rely on
> "device==true".

Yes, Ankit let's try this please. I would not call it VM_VFIO though

VM_VFIO_ALLOW_WC ?

Introduce it in a separate patch and summarize this thread, with a
suggested-by from Catalin :)

Cc Sean Christopherson <seanjc@google.com> too as x86 kvm might
support the idea

> I think that's a key argument. The VMM cannot, on its own, configure the
> BAR and figure a way to communicate this to the guest. We could invent
> some para-virtualisation/trapping mechanism but that's unnecessarily
> complicated. In the DPDK case, DPDK both configures and interacts with
> the device. In the VMM/VM case, we need the VM to do this, we can't
> split the configuration in VMM and interaction with the device in the
> VM.

Yes

Thanks,
Jason