Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 12 Apr 2023 09:53:07 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Marc Zyngier <maz@kernel.org>
Cc:     ankita@nvidia.com, alex.williamson@redhat.com,
        naoya.horiguchi@nec.com, oliver.upton@linux.dev,
        aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com,
        targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com,
        apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org
Subject: Re: [PATCH v3 0/6] Expose GPU memory as coherently CPU accessible
Message-ID: <ZDapsz2QOdjhcBHJ@nvidia.com>
References: <20230405180134.16932-1-ankita@nvidia.com>
 <86sfd5l1yf.wl-maz@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <86sfd5l1yf.wl-maz@kernel.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tVFBLgiG+lvmFapjqAz+uB9XiEw4pzrcJg6+V9V/rcAJ7glTzkwpotMMMXUi?=
 =?us-ascii?Q?Lp4BpM3r2Qqo+/yTc8jjndUY1THgJLf9gBRd854UZOZ187W6iNLSCzcICeS9?=
 =?us-ascii?Q?7VUV5Ii9UwqYHB1Dkkny9J4UlINNCdanLX5Y8KnxnSg09bGYI0LUWCrFBiyr?=
 =?us-ascii?Q?uofNAdN1eWldYCj9JFog6T/8aU8M3O4Bn7L6Ux5MMoH2GGFM6DI/ru6QLh07?=
 =?us-ascii?Q?BvR4oAv7BgCjvF1DYH7U8bgNjcCoqAsCYm513fyh8RmjiOk2mv+VkE1H36JR?=
 =?us-ascii?Q?hBldtc0/8FcanG0baKSZxOVQ55ueHvs75my2W4uskyEPxq2+s6Sem495RSol?=
 =?us-ascii?Q?ozRRZ+vgnnipu3HMBOkQOAxpl9x24Y0rQkNBVeedmOsiIAj4eORUBrKySA4n?=
 =?us-ascii?Q?a5VZODCbz6eKLu4gCwd+YxDsxjsEFF4MoZFOqLt888qAQoUQYlLbkyWi3o1A?=
 =?us-ascii?Q?JDhEKr1hHAgbUkn4iRtNIwkjqQ1+dbVxfGN24mo0OY1CdI1v9YpiY985yjTR?=
 =?us-ascii?Q?soj57QU7B4i3YIpglTjbFxXez/CylDYlmbpC0CFx8WaxonEJ4oHb6DddZYOY?=
 =?us-ascii?Q?ugrFy5zCuo9ZXscbW7FlkvqFiX2ff+RuHmP045/QpbhYlUZAM9bUHP3E+7Iy?=
 =?us-ascii?Q?RgUCvXFiKieaKLOsqLqXNoGoWizPcGydWQOk//WFVRpxflA1bNvUki9CupJ0?=
 =?us-ascii?Q?5M0SGXBDB4AgdyUQjkxiIDSGdHxDadKh0YrgIIHfqfNbTUMLcIxa2Bu2p9yG?=
 =?us-ascii?Q?Z/Zhx5qD0oQZ4H11m3RgZffrJz2y8Q/FV1Y62SwLmv4NRGuuZ1CcqXfQUgy7?=
 =?us-ascii?Q?An5OKSnO0U2Kz1WsTO6J0JPS6GfrD7XssCYvx035uY9rAtWq9TzrJnVF2mef?=
 =?us-ascii?Q?sQ1e5XDFRFEhldM/O7DrW5XXuxh22/UNIOAjLz4zjhTeHQUog0uMr6DQiDiD?=
 =?us-ascii?Q?b93Ad/AfLDpC4jiKKRisf1Hp7Nd8SoEKojaRr7aBy8uvBBdGAXaNfShaxhqX?=
 =?us-ascii?Q?nHW0MsNq45zszH3g9bMzzeHF5KE1Grrar+xMgiJYFCAUzDU4QHJZ+3NOPm28?=
 =?us-ascii?Q?tVZSL8D5B9Ma54MFI3gV/XZdf844amgccm5e7zUoShqxcUXJY/en9DYdE8JH?=
 =?us-ascii?Q?vXMMPtxEhSKZka6gyBQG9m1z+3XNSQmuMd6wWhRgb+K5Ios7eCl56xA9O/RH?=
 =?us-ascii?Q?3MKPEsXOcix8i2HmuxoVK5qVnmHbLYRk29MXw5sN9gdW3bUQMBsD71n0/r1b?=
 =?us-ascii?Q?h9TdVNBU5WC+yapnm94ezyU3LEjWFn1ZxTGB3R8iUeUXKEm5K+VHFHSK4A1Y?=
 =?us-ascii?Q?5Dm/IdeJKoH6ka2AU3dLSsEPtczVoqtLQNIliW7p7OrNZDlB94NGwAtbUFOq?=
 =?us-ascii?Q?CR9QUwvKtn2J5Q6dtN1tz9hYaTTSlzu11XIueinN5xbmjUPvFngEffcliIUj?=
 =?us-ascii?Q?OT3dGBTipX4ctDBgDRGPeWt3QaQduWgdGDDC3Iv2noVQo7kdxRDzzyYDkZV4?=
 =?us-ascii?Q?n+L8NvUXqMvrLKvGy39Zoi9QvgPejgxV5brobIgMMf3KeQKg7Rh6M0Nk/sta?=
 =?us-ascii?Q?BfWPyDYUO3ZkKy24dw/mkNrdJdJQgKK/+Zurq7CS?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2f1b5f76-a9fa-424e-3c8d-08db3b54de2c
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2023 12:53:10.0266
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: alhKhjJu/CHy0D3IzFiaKYhAyd9JYjtRfFzWN6irXkKMlmpPcBVTaOoS3c1DG92a
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6358
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Apr 12, 2023 at 01:28:08PM +0100, Marc Zyngier wrote:
> On Wed, 05 Apr 2023 19:01:28 +0100,
> <ankita@nvidia.com> wrote:
> > 
> > From: Ankit Agrawal <ankita@nvidia.com>
> > 
> > NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device
> > for the on-chip GPU that is the logical OS representation of the
> > internal propritary cache coherent interconnect.
> > 
> > This representation has a number of limitations compared to a real PCI
> > device, in particular, it does not model the coherent GPU memory
> > aperture as a PCI config space BAR, and PCI doesn't know anything
> > about cacheable memory types.
> > 
> > Provide a VFIO PCI variant driver that adapts the unique PCI
> > representation into a more standard PCI representation facing
> > userspace. The GPU memory aperture is obtained from ACPI, according to
> > the FW specification, and exported to userspace as the VFIO_REGION
> > that covers the first PCI BAR. qemu will naturally generate a PCI
> > device in the VM where the cacheable aperture is reported in BAR1.
> > 
> > Since this memory region is actually cache coherent with the CPU, the
> > VFIO variant driver will mmap it into VMA using a cacheable mapping.
> > 
> > As this is the first time an ARM environment has placed cacheable
> > non-struct page backed memory (eg from remap_pfn_range) into a KVM
> > page table, fix a bug in ARM KVM where it does not copy the cacheable
> > memory attributes from non-struct page backed PTEs to ensure the guest
> > also gets a cacheable mapping.
> 
> This is not a bug, but a conscious design decision. As you pointed out
> above, nothing needed this until now, and a device mapping is the only
> safe thing to do as we know exactly *nothing* about the memory that
> gets mapped.

IMHO, from the mm perspective, the bug is using pfn_is_map_memory() to
determine the cachability or device memory status of a PFN in a
VMA. That is not what that API is for.

The cachability should be determined by the pgprot bits in the VMA.

VM_IO is the flag that says the VMA maps memory with side-effects.

I understand in ARM KVM it is not allowed for the VM and host to have
different cachability, so mis-detecting host cachable memory and
making it forced non-cachable in the VM is not a safe thing to do?

Jason