Received-SPF: pass (google.com: domain of linux-kernel+bounces-139433-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <e24e0de1-d984-474a-8b79-ea0934ac668e@intel.com>
Date: Thu, 11 Apr 2024 10:26:50 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 1/5] x86/kexec: do unconditional WBINVD for bare-metal
 in stop_this_cpu()
References: <cover.1712493366.git.kai.huang@intel.com>
 <33b985a8f4346f4bcf0944eaf37193a906b11af3.1712493366.git.kai.huang@intel.com>
 <e94d208e-964f-cfc0-60a4-fe70db52bec1@amd.com>
 <cbf171e1-a518-429b-fd07-3526a0b252bf@amd.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
To: Tom Lendacky <thomas.lendacky@amd.com>, <linux-kernel@vger.kernel.org>
CC: <x86@kernel.org>, <dave.hansen@intel.com>, <bp@alien8.de>,
	<kirill.shutemov@linux.intel.com>, <tglx@linutronix.de>, <mingo@redhat.com>,
	<hpa@zytor.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<rick.p.edgecombe@intel.com>, <ashish.kalra@amd.com>, <chao.gao@intel.com>,
	<bhe@redhat.com>, <nik.borisov@suse.com>, <pbonzini@redhat.com>,
	<seanjc@google.com>
In-Reply-To: <cbf171e1-a518-429b-fd07-3526a0b252bf@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?b3NJTEoydUtTb1IxQWxXVUZRWGpOZ1lxRlhFR21talFhdklqbVBScTlTd1dY?=
 =?utf-8?B?YzV6WWNPcUk2MlFaZ3VENGZpOVEzZmNTU2xkMEhGL0tYRkhlUkl1ZnlRTy9m?=
 =?utf-8?B?cWZwUWxzMkhrQUY5MHZQck43U0grN04wVE1aQ2hhb2hCWmZiajdSVnJtanNi?=
 =?utf-8?B?Y0lXVFkzTFpldC84NkJ5N3B0L0NWaEI4emV0ZXprM3RteG5Ja3dLTGJOcUVV?=
 =?utf-8?B?a3crSTNyRWU2MWl3Rm1YeFJkRUFUQ3AvYmJETDBHZzQ3ZytBelZtejh4eVpi?=
 =?utf-8?B?cHhyWnduTFV6eFNVS0lES29DMXI3MldBYmN2RnVzOXJIUld4Q3dwcnpobE1v?=
 =?utf-8?B?MjJHMUpCVkdyTWU3TUdWeVZlNStFOG5HbEpKUEYvNk5iall6djRodkRQMi8x?=
 =?utf-8?B?VU9hKzRGV1ZzbGh1V3JKQnNnN0ZWTjUzQWFRRFFMQ29zbC9YUS83VTF4dmNB?=
 =?utf-8?B?UWd2V1pPNmU0S3B0THRJdDRreGxaUTlid2Z0OVY2M0R4QUZ3UUY0NGhtNnFS?=
 =?utf-8?B?TDJNWWdhbDdLYnRqMERXeTZXVXd4Y3Q4dGtzeDNWdld3SFMrNzFHTWl5U0Rz?=
 =?utf-8?B?Tkl6dFFmdVhMenorR2ovVU8zSGtzYURsMGRHSUlyM0NpYlNZb1ZXZlRKN2hL?=
 =?utf-8?B?VmJFTlFwOWwwcVE3cWYrcUtGTzV2c29wVFFkNGFEQndsU1JyWjIxTmZYVzJp?=
 =?utf-8?B?SGZoT1NvRmVjcjEzZ3BDazNzc1Y0SlVYNnhlY1A2R0V3V2hLbXl0bFpJaXQ2?=
 =?utf-8?B?aDVZMWQ2bzJ1TUc2cThjbzNKb0UwMmdKcG1HN2l2RERTakFMQk0xeXBLcVp4?=
 =?utf-8?B?NmhvcHM4S1V3Qkc0RndnbE4rK0RPazhqbnpUTUxGT20wUWw0NFRGTGVrOVND?=
 =?utf-8?B?WVJ2cVJUUGZRVDU2Sk5OcEI2YjJQMG95eXoyQ0VLOEltbXBYNWFGbHJhcDB2?=
 =?utf-8?B?UmxDRzJEbkNjUHkrUzR4NEtxSFhJWkJsWXdRSTFtNXhrbHc0U1FkVEl6Tmlz?=
 =?utf-8?B?MkVKSVZmUHFrQWRsQldiU0J0VEtNSExQNFBwSEpyTDlxU0MzdVR3Tm5ueG95?=
 =?utf-8?B?dWMwdnZzZW1CNEoydkdPdHdveDNXcXlXdk5nRUp3ZWc4clZxWmxIRlRRZUdj?=
 =?utf-8?B?MEdabHgxV0dXT2hNNmtVV3Y4MmQrWW1qeGFramNLY2xMMFVGajhMR1dVWFlL?=
 =?utf-8?B?NklLdzFtczlYZEF5QS85N3NFZ1JoMURmT2QxRUJtT0dwTVREMytBZHFEMXYv?=
 =?utf-8?B?dW00U1BPMy9Fa0FBZUZNWjl1OFNvS2FlRGljSHVnOTBKL05seWQvT0hZYTBQ?=
 =?utf-8?B?ZXJaSDlrV0tDdW1LODJteEgydjlUWWJGNHFsaG9lNFUvSC9JOVU5dVlyenov?=
 =?utf-8?B?NE13ZnpCTGhZdWxMaTU4aEV3aTFpWEE3TjlPVnl1SWhONXpwNXRxYXZvMUMw?=
 =?utf-8?B?YUhXUDFMWjU2UUU3Ykl2bG1FM0hQMUtXbWtTajdiT0kydVZ2bFBkOEVoemJD?=
 =?utf-8?B?WTE1QW16YVBZdE9RV1NXS2gyeGdZa0tmak5uVzdxamxjbUpLL1lhRzB6bjk0?=
 =?utf-8?B?K3VCczJwbkhsYUFkbW5sUWpRa2NENnErdmFucFBBbWZoTWZmckVCYVR6K2JV?=
 =?utf-8?B?enBmV1pFSXlVTkRSSXdxR0VLRVRKbG8wTXpzTzZybWFjdlZHTHd1MTcyYVBk?=
 =?utf-8?B?Y0JYUlBNeFZqT1FSZTJYWCt2bVBvWCsyKysybmNWR3V0OTJzdU9lNG9mRWdm?=
 =?utf-8?B?czZZUDBicEhaNHZqRVI4eHZqNUlDSGpRUXM2ajd6bnJUeWNaK0pWamFBckE3?=
 =?utf-8?B?MnN1TkZReUpKZnVpZVR6WVFobitqY2N5a2Zhb1lIaDg0cjgyd3lsOTl5dkJS?=
 =?utf-8?B?U0NwcWxDY3BCRDlhTnloVWhPNGFnYmJyc1dMNjkyWTRaRmJ1d29ucG1nakE1?=
 =?utf-8?B?MXA5RnNkdUQrb0tYRHRKSGxsMGhFdFFLd2RPR3V1SFRydWozeGZFZUVzc1ZK?=
 =?utf-8?B?SC9VTTZTWEhhVWdiaVhSRHlSK3NjWGRDZk5RUktBWHFObkNCOWI2WjNuaFU1?=
 =?utf-8?B?cGpEb01sS2FOV1c4MnRsRkdNVTBhcDE3Z1cyRmpmdlp4Z21rM1pJVUJRblly?=
 =?utf-8?Q?MFb6JZPWVa+o3ByDuBWgYgCjS?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ce2a7808-d511-4693-dbd8-08dc59ad5805
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 22:27:02.2575
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 32lI7a3ByhMrdCKTDgINT0w5yz6ahcJ3lBwpu8nC0S8XgKAjY2rwLe+4u9nruTW3nHLSL2HF7ZKuipGDXOf/TQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB7013
X-OriginatorOrg: intel.com


On 11/04/2024 4:14 am, Tom Lendacky wrote:
> On 4/10/24 11:08, Tom Lendacky wrote:
>> On 4/7/24 07:44, Kai Huang wrote:
>>
>>> diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
>>> index b8441147eb5e..5ba8a9c1e47a 100644
>>> --- a/arch/x86/kernel/process.c
>>> +++ b/arch/x86/kernel/process.c
>>> @@ -813,18 +813,16 @@ void __noreturn stop_this_cpu(void *dummy)
>>>       mcheck_cpu_clear(c);
>>>       /*
>>> -     * Use wbinvd on processors that support SME. This provides support
>>> -     * for performing a successful kexec when going from SME inactive
>>> -     * to SME active (or vice-versa). The cache must be cleared so that
>>> -     * if there are entries with the same physical address, both 
>>> with and
>>> -     * without the encryption bit, they don't race each other when 
>>> flushed
>>> -     * and potentially end up with the wrong entry being committed to
>>> -     * memory.
>>> +     * The kernel could leave caches in incoherent state on SME/TDX
>>> +     * capable platforms.  Flush cache to avoid silent memory
>>> +     * corruption for these platforms.
>>>        *
>>> -     * Test the CPUID bit directly because the machine might've cleared
>>> -     * X86_FEATURE_SME due to cmdline options.
>>> +     * stop_this_cpu() is not a fast path, just do unconditional
>>> +     * WBINVD for simplicity.  But only do WBINVD for bare-metal
>>> +     * as TDX guests and SEV-ES/SEV-SNP guests will get unexpected
>>> +     * (and unnecessary) #VE and may unable to handle.
>>
>> In addition to Kirill's comment on #VE...
>>
>> This last part of the comment reads a bit odd since you say 
>> unconditional and then say only do WBINVD for bare-metal. Maybe 
>> something like this makes it a bit clearer?:
>>
>> For TDX and SEV-ES/SEV-SNP guests, a WBINVD may cause an exception 
>> (#VE or #VC). However, all exception handling has been torn down at 
>> this point, so this would cause the guest to crash. Since memory 
>> within these types of guests is coherent only issue the WBINVD on 
>> bare-metal.
> 
> Hmmm... actually it was the other WBINVD in patch #2 that caused the 
> crash, so what I wrote above isn't accurate. You might want to re-word 
> as appropriate.

Yeah that's why I used "may unable to handle" in the comment, as I 
thought there's no need to be that specific?

I tend not to mention "memory within these types of guests is coherent". 
  I mean the current upstream kernel _ONLY_ does WBINVD for SME, that 
means for all non-SME environment there's no concern to do WBINVD here.

Here we only extend to do WBINVD on more environments, so as long as 
there's no harm to do WBINVD for them it's OK.

How about below?

	/*
	 * The kernel could leave caches in incoherent state on SME/TDX
	 * capable platforms.  Flush cache to avoid silent memory
	 * corruption for these platforms.
	 *
	 * For TDX and SEV-ES/SEV-SNP guests, a WBINVD causes an
	 * exception (#VE or #VC), and the kernel may not be able
	 * to handle such exception (e.g., TDX guest panics if it
	 * sees #VE).  Since stop_this_cpu() isn't a fast path, just
	 * issue the WBINVD on bare-metal instead of sprinkling
	 * around vendor-specific checks.
	 */
> 
> Thanks,
> Tom
> 
>>
>> And you can expand the comment block out to at least 80 characters to 
>> make it more compact.

OK I can do.  I guess I have to change my vim setting to do so, though :-)