Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Date:   Wed, 11 Oct 2023 15:38:39 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     ankita@nvidia.com, maz@kernel.org, oliver.upton@linux.dev,
        will@kernel.org, aniketa@nvidia.com, cjia@nvidia.com,
        kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com,
        acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com,
        danw@nvidia.com, linux-arm-kernel@lists.infradead.org,
        kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v1 1/2] KVM: arm64: determine memory type from VMA
Message-ID: <20231011183839.GC3952@nvidia.com>
References: <20230907181459.18145-1-ankita@nvidia.com>
 <20230907181459.18145-2-ankita@nvidia.com>
 <ZR7hKYU1Wj+VTqpO@arm.com>
 <20231005165458.GM682044@nvidia.com>
 <ZSVe0nb02S4kz5D2@arm.com>
 <20231010150502.GM3952@nvidia.com>
 <ZSWHkvhutlnvVLUZ@arm.com>
 <20231010182328.GS3952@nvidia.com>
 <ZSbfUNLwDkaYL4ts@arm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZSbfUNLwDkaYL4ts@arm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cbPrIU2J7R6fQZl5DzTdWCf7oz66GWhCLdb1XHLkGeHSNMVttZxj09K/x8QN?=
 =?us-ascii?Q?DYdMI+kVLD7j5DrpeTzKxo+EJ7pdoZQYg4S7s3ret2EACaulhhazUGxqdURO?=
 =?us-ascii?Q?uPEYmqZT67sE+iIpA8xS+Z4yfkJWRzu4B83hbDfcX2G54IiYIp/HlyNk2jgG?=
 =?us-ascii?Q?A46HACjurjL+TRIYUm/SJ0gmcCt8q/+ce5T0GmzsAXCIbxY91WVExTH7FIx1?=
 =?us-ascii?Q?2NDp/G5+oR0fU01lpZXxOTCS1DaxuzlkYDPVpLyCFYAe+b8qtxcCG87Sudaa?=
 =?us-ascii?Q?wZIICklPeE29aXaJpLX3aDsPVkHgFmRzhsupAiuYJBZCGBUCArfqgJGlY/nn?=
 =?us-ascii?Q?RyOHPRXQq3LHv2H4NPYGG8ChWJia38iXnUonEZu3SpXJyXzcL65we5ncAQPn?=
 =?us-ascii?Q?oPp7exu3J0C6+yM6agm/3IKdi5B3uVCi+xbZyTszx4dZiuV9KcrIJeDD88fe?=
 =?us-ascii?Q?1vAPY9n4r5dhcDOAHRM5SyrZ3ibCzoQvADfvAivybY9ux5eIjxEHvi2k+kF2?=
 =?us-ascii?Q?w6ARF6171wkEkq6LVZ6zVcNxknm2/dgXds+yZUrfUV78mZegorwZ/buSTX5m?=
 =?us-ascii?Q?uu243NPsPYvJz6xinSQp2ftbouK1hi4EnUyPlTaO90xFLlBu2n4ozTj9YYtX?=
 =?us-ascii?Q?9U2lPvT8psTcBaLZXgaelghyxgCXw9y4yBdbQDZza9rJfzikKiTdJjeC3bCP?=
 =?us-ascii?Q?Al8uSaK1M3vjanUkw2Xi2raBW/Ph8OFHNgHegi9VhH9ZLP9uTW74Go6P+brG?=
 =?us-ascii?Q?i61LdU0BSmcAF2GvLPu2DQIrZLtdoOKyvYMbDOOW+5LdT7B9SBNUF5fMirgD?=
 =?us-ascii?Q?cjqO8yEghtscJjwVoJeoQJbjAlwwsVC+fUyODRsuGjvWHAlmh1m+XWD6byJs?=
 =?us-ascii?Q?Zr9AUFXHvhoWrDGxGw80HdW/Vx0qNp28mh8e0SxJY/z6VfHjH8peh9dMgxNR?=
 =?us-ascii?Q?qtDHijJ1Vw7rL9gSo0RPbk5SOBF3kNUUYwmt79c7HXoKE2C+FNk+9KN/A3nq?=
 =?us-ascii?Q?cg1UQf+ymeSuXvfTKeuETB5a8ohO+Apf8pRugzy0cYV6Fz1UYKt98lSFO+NI?=
 =?us-ascii?Q?dQFudMbYgE2xMiqAc3ZFNmqvR2ljvoa/+Dim5xpb9cHEMiF5I/tfkgQqD7yR?=
 =?us-ascii?Q?ZOr9i8H421Mq2WDK/N8qj/QfvOc992mAHeEPkb9zUvB9JlMQxEaBtSAQNPxs?=
 =?us-ascii?Q?my8qsZ6YZQvINvUxCqbxuBy44v2wA5whqSUlAniF98lc5R+So6syD46Yr4SM?=
 =?us-ascii?Q?wNfuI/e6cHHaTXLJ2OH7kpUefIDs17Xj/PoCvTh1b58Zz0eHPSOWfKTz0omt?=
 =?us-ascii?Q?XAvbqWXpaj3IzM+VwurJ21+VeDm8FYx0ZnyJ4o+s9ajWnXYu9644fTgcEWOc?=
 =?us-ascii?Q?cvjWtIkcXPvB92JIOWKx7oECsKCtiLlWNyUhsHq7rXCSmaa/wtDgLazJbsrA?=
 =?us-ascii?Q?M41qae5FaBHJknXbEKXIyxCPbtUkd8/vyF/KpriZ/7nIy438FmJjwj6CWGWY?=
 =?us-ascii?Q?6WM7Hgnw7+2333ghzxjFsNsk6QyCSnQjnqVu/vtSBbOoGRsmySFAOjEM1BB+?=
 =?us-ascii?Q?gn0cnG8DU5+yD8EzjMsQD1r0g3tn591LFSUFooV6?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 74120893-dda6-4f08-80f9-08dbca894a46
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB5859.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2023 18:38:41.0306
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ruduFyJoK+l0rrAvitqYp5sTWq/Jhzw5AkHs9revYIybankxHws6Kj57tBAVNq0l
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8893
X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 11 Oct 2023 11:38:59 -0700 (PDT)
X-Spam-Level: **

On Wed, Oct 11, 2023 at 06:45:52PM +0100, Catalin Marinas wrote:

> > From the mm perspective we can't have two kinds of cachable struct
> > pages running around that are functionally different.
> 
> From a Linux+MTE perspective, what goes into ZONE_MOVABLE should work
> fine, all pages interchangeable (if it doesn't, the hardware is
> broken).

Yes, and we imagine adding GPU and CXL memory to ZONE_MOVABLE (on a
NUMA node)

> These are added via the add_memory_resource() hotplug path. If a
> platform is known not to support this, it better not advertise MTE as a
> feature (the CPUs usually have some tie-off signal when the rest of the
> SoC cannot handle MTE). We could claim it's a hardware erratum if it
> does.

Seems logical
 
> But for ZONE_DEVICE ranges, these are not guaranteed to support all the
> characteristics of the main RAM. I think that's what memremap_pages()
> gives us. I'm not too familiar with this part of the kernel but IIUC
> that falls under the HMM category, so not interchangeable with the
> normal RAM (hotplugged or not).

DAX pages use ZONE_DEVICE and they are cachable, and not "HMM".

They are not fully interchangable, but they get into the page cache,
they can back .data segements, they could be subject atomics/etc. So
they should be fully functional like DDR.

> I don't see the mm code doing this but I haven't looked deep enough.
> At least not in the way of doing an mmap(MAP_ANONYMOUS) and the kernel
> allocating ZONE_DEVICE pages and passing them to the user.

Not ZONE_DEVICE. One popular coherent GPU approach is to use
ZONE_MOVABLE pages.

> > > > > I've seen something similar in the past with
> > > > > LSE atomics (or was it exclusives?) not being propagated. These don't
> > > > > make the memory safe for a guest to use as general purpose RAM.
> > > > 
> > > > At least from a mm perspective, I think it is important that cachable
> > > > struct pages are all the same and all interchangable. If the arch
> > > > cannot provide this it should not allow the pgmap/memremap to succeed
> > > > at all. Otherwise drivers using these new APIs are never going to work
> > > > fully right..
> > > 
> > > Yes, for struct page backed memory, the current assumption is that all
> > > are the same, support all CPU features. It's the PFN-based memory where
> > > we don't have such guarantees.
> > 
> > I see it got a bit confused, I am talking about memremap_pages() (ie
> > include/linux/memremap.h), not memremap (include/linux/io.h) for
> > getting non-struct page memory. It is confusing :|
> > 
> > memremap_pages() is one of the entry points of the struct page hotplug
> > machinery. Things like CXL drivers assume they can hot plug in new
> > memory through these APIs and get new cachable struct pages that are
> > functionally identical to boot time cachable struct pages.
> 
> We have two mechanisms, one in memremap.c and another in
> memory_hotplug.c.

Yes

>  So far my assumption is that only the memory added by
> the latter ends up in ZONE_MOVABLE and can be used by the kernel as any
> of the ZONE_NORMAL RAM, transparently to the user. 

Probably for now, yes. But CXL/GPU memory goes there too.

> For ZONE_DEVICE allocations, one would have to explicitly mmap() it
> via a device fd.

Not quite for DAX, it gets in through the page cache, but it is still
mmap on a FD and not anonymous memory.

> If a VMM wants to mmap() such GPU memory and give it to the guest as
> general purpose RAM, it should make sure it has all the characteristics
> as advertised by the CPU or disable certain features (if it can).

This is the VFIO flow we are talking about here, I think. PFNMAP
memory that goes into a VM that is cachable.

> Currently we don't have a way to tell what such memory supports (neither
> ACPI tables nor any hardware probing). The same assumption w.r.t. MTE is
> that it doesn't.

Indeed, but my GPU driver hot plugged it as ZONE_MOVABLE and my VFIO
driver turned in into PFNMAP.. So these things seem incompatible.

> > > We have an additional flag, VM_MTE_ALLOWED, only set for mappings backed
> > > by struct page. We could probe that in KVM and either fall back to
> > > non-cacheable or allow cacheable if MTE is disable at the vCPU level.
> > 
> > I'm not sure what this does, it is only set by shmem_map? That is
> > much stricter than "mappings backed by struct page"
> 
> This flag is similar to the VM_MAYWRITE etc. On an mmap(), the vma gets
> the VM_MTE_ALLOWED flag if the mapping is MAP_ANONYMOUS (see
> arch_calc_vm_flag_bits()) or the (driver) mmap function knows that the
> memory supports MTE and sets the flag explicitly. Currently that's only
> done in shmem_mmap() as we know where this memory is coming from. When
> the user wants an mmap(PROT_MTE), the arch code checks whether
> VM_MTE_ALLOWED is set on the vma before allowing tag accesses.
> 
> Memory mapped from ZONE_DEVICE won't have such flag set, so
> mmap(PROT_MTE) will fail. But for KVM guests, there's no such mmap()
> call into the hypervisor. A guest can simply enable MTE at stage 1
> without the hypervisor being able to tell.

Yes, so this is all safe for DAX usages, not safe for GPU CXL NUMA
memory hotplug. :|

> > Still, I'm not sure how to proceed here - we veered into this MTE
> > stuff I don't know we have experiance with yet.
> 
> We veered mostly because on arm64 such GPU memory is not guaranteed to
> have all the characteristics of the generic RAM. I think only MTE is the
> dangerous one and it needs extra care but I wouldn't be surprised if we
> notice atomics failing.

So, at least for the system this change is being tested on, the
"pre-CXL" GPU memory is 100% interchangable with DDR memory. It is
surfaced to the OS as ZONE_MOVABLE and it should work in VMs like this
too.

> It looks like memremap_pages() also takes a memory type and I suspect
> it's only safe to map MEMORY_DEVICE_COHERENT into a guest (as generic
> RAM). Is there any sanity check on the host kernel side to allow VMM
> cacheable mappings only for such memory and not the other
> MEMORY_DEVICE_* types?

I guess it is this current KVM code we are discussing, it probably
happens by the pfn_is_map_memory() check?
 
> Going back to KVM, we can relax to cacheable mapping at Stage 2 if the
> vma is cacheable and either VM_MTE_ALLOWED is set or KVM_CAP_ARM_MTE is
> disabled.

This seems logical to me, thanks

> From the earlier discussions, we can probably ignore VM_IO
> since we won't have a cacheable mapping with this flag. Not sure about
> VM_PFNMAP.

PFNMAP is the interesting one for VFIO, at least. Can we use the same
reasoning that it will be !VM_MTE_ALLOWED and we can close the MTE
discussion.

Currently no VFIO driver is doing cachable that has memory that is
different from DDR memory. So this is sort of theoretical discussion
about future cachable HW that does use VFIO that does have a
non-uniformity.

Maybe that HW should set VM_IO on its VFIO PFN map and obviously not
use ZONE_MOVABLE?

Where does that leave us for this patch? We check the VM_MTE_ALLOWED
and check for ZONE_MOVABLE struct pages as one of the conditions for
NORMAL?

Jason