+ Lorenzo
On Thu, Sep 07, 2023 at 11:14:59AM -0700, [email protected] wrote:
> From: Ankit Agrawal <[email protected]>
>
> Linux allows device drivers to map IO memory on a per-page basis using
> "write combining" or WC. This is often done using
> pgprot_writecombing(). The driver knows which pages can support WC
> access and the proper programming model to generate this IO. Generally
> the use case is to boost performance by using write combining to
> generate larger PCIe MemWr TLPs.
>
> Allow VMs to select DEVICE_* or NORMAL_NC on a page by page basis for
> all IO memory. This puts the VM in charge of the memory attributes,
> and removes the KVM override to DEVICE_nGnRE.
>
> Ultimately this makes pgprot_writecombing() work correctly in VMs and
> allows drivers like mlx5 to fully operate their HW.
>
> After some discussions with ARM and CPU architects we reached the
> conclusion there was no need for KVM to prevent the VM from selecting
> between DEVICE_* and NORMAL_NC for IO memory in VMs. There was a fear
> that NORMAL_NC could result in uncontained failures, but upon deeper
> analysis it turns out there are already possible cases for uncontained
> failures with DEVICE types too. Ultimately the platform must be
> implemented in a way that ensures that all DEVICE_* and NORMAL_NC
> accesses have no uncontained failures.
>
> Fortunately real platforms do tend to implement this.
>
> This patch makes the VM's memory attributes behave as follows:
>
> ?S1?????????? |?? S2????????? |? Result
> ?NORMAL-WB????|? NORMAL-NC????|? NORMAL-NC
> ?NORMAL-WT????|? NORMAL-NC????|? NORMAL-NC
> ?NORMAL-NC????|? NORMAL-NC????|? NORMAL-NC
> ?DEVICE<attr>?|? NORMAL-NC????|? DEVICE<attr>
>
> See section D8.5.5 of DDI0487_I_a_a-profile_architecture_reference_manual.pdf
> for details.
>
> Signed-off-by: Ankit Agrawal <[email protected]>
From the discussions with the hardware people in Arm and Nvidia, we
indeed concluded that relaxing S2 to Normal-NC is not any worse than
Device (and actually Device memory is more prone to generating
uncontained errors, something to do with the Device memory ordering
semantics).
For this change:
Reviewed-by: Catalin Marinas <[email protected]>
--
Catalin