Received-SPF: pass (google.com: domain of linux-crypto-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
 165.204.84.17 as permitted sender) receiver=protection.outlook.com;
 client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C
From:   Ashish Kalra <Ashish.Kalra@amd.com>
To:     <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
        <kvm@vger.kernel.org>, <linux-coco@lists.linux.dev>,
        <linux-mm@kvack.org>, <linux-crypto@vger.kernel.org>
CC:     <tglx@linutronix.de>, <mingo@redhat.com>, <jroedel@suse.de>,
        <thomas.lendacky@amd.com>, <hpa@zytor.com>, <ardb@kernel.org>,
        <pbonzini@redhat.com>, <seanjc@google.com>, <vkuznets@redhat.com>,
        <jmattson@google.com>, <luto@kernel.org>,
        <dave.hansen@linux.intel.com>, <slp@redhat.com>,
        <pgonda@google.com>, <peterz@infradead.org>,
        <srinivas.pandruvada@linux.intel.com>, <rientjes@google.com>,
        <dovmurik@linux.ibm.com>, <tobin@ibm.com>, <bp@alien8.de>,
        <michael.roth@amd.com>, <vbabka@suse.cz>, <kirill@shutemov.name>,
        <ak@linux.intel.com>, <tony.luck@intel.com>, <marcorr@google.com>,
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        <alpergun@google.com>, <dgilbert@redhat.com>, <jarkko@kernel.org>
Subject: [PATCH Part2 v6 48/49] *debug: warn and retry failed rmpupdates
Date:   Mon, 20 Jun 2022 23:15:13 +0000
Message-ID: <e78c2c9bd5c5b7dc262f41a85e75056f4af20b88.1655761627.git.ashish.kalra@amd.com>
In-Reply-To: <cover.1655761627.git.ashish.kalra@amd.com>
References: <cover.1655761627.git.ashish.kalra@amd.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2022 23:15:24.0841
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d64c1ae-c0ae-4d38-72f9-08da5312c12c
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT048.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1852
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-crypto.vger.kernel.org>
X-Mailing-List: linux-crypto@vger.kernel.org

From: Michael Roth <michael.roth@amd.com>

In some cases on B0 hardware exhibits something like the following
behavior (where M < 512):

  Guest A                        | Guest B
 |-------------------------------|----------------------------------|
 |                               | rc = rmpupdate pfn=N*512,4K,priv
 | rmpupdate pfn=N*512+M,4K,priv |
 | rc = FAIL_OVERLAP            | rc = SUCCESS

The FAIL_OVERLAP might possible be the result of hardware temporarily
treating Guest B's rmpupdate for pfn=N*512 as a 2M update, causing the
subsequent update from Guest A for pfn=N*512+M to report FAIL_OVERLAP
at that particular instance. Retrying the update for N*512+M immediately
afterward seems to resolve the FAIL_OVERLAP issue reliably however.

A similar failure has also been observed when transitioning pages back
to shared during VM destroy. In this case repeating the rmpupdate does
not always seem to resolve the failure immediately.

Both situations are much more likely to occur if THP is disabled, or
if it is enabled/disabled while guests are actively being
started/stopped.

Include some debug/error information to get a better idea of the
behavior on different hardware, and add the rmpupdate retry as a
workaround for Milan B0 testing.

Signed-off-by: Michael Roth <michael.roth@amd.com>
---
 arch/x86/kernel/sev.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c
index 6640a639fffc..5ae8c9f853c8 100644
--- a/arch/x86/kernel/sev.c
+++ b/arch/x86/kernel/sev.c
@@ -2530,6 +2530,7 @@ static int rmpupdate(u64 pfn, struct rmpupdate *val)
 {
 	unsigned long paddr = pfn << PAGE_SHIFT;
 	int ret, level, npages;
+	int retries = 0;
 
 	if (!pfn_valid(pfn))
 		return -EINVAL;
@@ -2552,12 +2553,26 @@ static int rmpupdate(u64 pfn, struct rmpupdate *val)
 		}
 	}
 
+retry:
 	/* Binutils version 2.36 supports the RMPUPDATE mnemonic. */
 	asm volatile(".byte 0xF2, 0x0F, 0x01, 0xFE"
 		     : "=a"(ret)
 		     : "a"(paddr), "c"((unsigned long)val)
 		     : "memory", "cc");
 
+	if (ret) {
+		if (!retries) {
+			pr_err("rmpupdate failed, ret: %d, pfn: %llx, npages: %d, level: %d, retrying (max: %d)...\n",
+			       ret, pfn, npages, level, 2 * num_present_cpus());
+			dump_stack();
+		}
+		retries++;
+		if (retries < 2 * num_present_cpus())
+			goto retry;
+	} else if (retries > 0) {
+		pr_err("rmpupdate for pfn %llx succeeded after %d retries\n", pfn, retries);
+	}
+
 	/*
 	 * Restore the direct map after the page is removed from the RMP table.
 	 */
-- 
2.25.1