Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
 165.204.84.17 as permitted sender) receiver=protection.outlook.com;
 client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C
From:   Bharata B Rao <bharata@amd.com>
To:     <linux-kernel@vger.kernel.org>
CC:     <tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
        <dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>,
        <nikunj@amd.com>, <Abraham.Shaju@amd.com>
Subject: [RFC FIX PATCH] x86/e820: Stop kernel boot when RAM resource reservation fails
Date:   Mon, 18 Jul 2022 14:28:15 +0530
Message-ID: <20220718085815.1943-1-bharata@amd.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2022 08:58:29.4922
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: c30ad9fe-3724-4660-8b1e-08da689baf5b
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT016.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2824
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Currently it is possible to start a guest with memory that
is beyond the addressable range of CPU. This can typically
be done by using QEMU without explicilty specifying the max
physical addressable bits (via phys-bits or host-phys-bits
options). In such cases QEMU will start the guest with more
than 1TB memory but would implicitly limit the phys-bits to 40.

In this scenario, iomem_resource.end gets set to 1TB and
hence subsequent resource reservations of RAM regions beyond
1TB would fail. Since this failure is ignored, there can be
a situation where kernel is using the entire RAM (beyond 1T),
but the RAM range is not part of iomem resource tree.

This can lead to both performance as well as correctness
issues. For example, gettimeofday() calls will take more
time as the vvar_page gets mapped as uncacheable memory
type (_PAGE_CACHE_MODE_UC_MINUS). The vvar fault handler
will default to uncacheable type when it fails to find the
vvar_page pfn as part of any RAM range in iomem_resource.
Here is a comparision of the time taken (in us) by an
application doing lots (10240) of gettimeofday() calls, to
complete in case of 999G and 1T guest RAM:

Iteration	999G	1T
----------------------------
1		291	1178
2		316	3286
3		582	2982
4		284	1808
5		252	4503

This is how /proc/iomem looks like for the above two cases:

999G guest RAM
---------------
00001000-0009fbff : System RAM
00100000-bffdbfff : System RAM
100000000-f9ffffffff : System RAM
  1549c00000-154fe09107 : Kernel code
  1550000000-1552f3cfff : Kernel rodata
  1553000000-15544aea3f : Kernel data
  1554d67000-15553fffff : Kernel bss

1T guest RAM
------------
00001000-0009fbff : System RAM
00100000-bffdbfff : System RAM
6752200000-6758409107 : Kernel code
6758600000-675b53cfff : Kernel rodata
675b600000-675caaea3f : Kernel data
675d367000-675d9fffff : Kernel bss
(Last System RAM entry is missing)

It is also seen that any memory region reservation requests
(say by using request_free_mem_region()), whose sizes fall
below 1TB, will be satisfied, leading to ranges overlapping
with actual RAM range (though the RAM range is missing in the
resource tree).

Fix this problem by stopping the kernel boot when resource
reservation fails for system RAM.

Reported-by: Shaju Abraham <Abraham.Shaju@amd.com>
Signed-off-by: Bharata B Rao <bharata@amd.com>
---
1. It appears that we should fail for other types of
resources too and not just for RAM, but wasn't sure
and hence checking for RAM explicitly in this version.
2. There is an attempt to fix this on the QEMU side too
https://lore.kernel.org/qemu-devel/20220718081734.135598-1-nikunj@amd.com/

 arch/x86/kernel/e820.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
index f267205f2d5a..1cfe640afe71 100644
--- a/arch/x86/kernel/e820.c
+++ b/arch/x86/kernel/e820.c
@@ -1185,7 +1185,10 @@ void __init e820__reserve_resources(void)
 		 */
 		if (do_mark_busy(entry->type, res)) {
 			res->flags |= IORESOURCE_BUSY;
-			insert_resource(&iomem_resource, res);
+			if (insert_resource(&iomem_resource, res) &&
+			    entry->type == E820_TYPE_RAM)
+				panic("%s: Failed to reserve resource %s with range (%llx-%llx)\n",
+				      __func__, res->name, res->start, res->end);
 		}
 		res++;
 	}
-- 
2.25.1