Received-SPF: pass (google.com: domain of linux-kernel+bounces-199888-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <e54d68e7-3d57-4128-8c07-dd6c66196a7e@intel.com>
Date: Tue, 4 Jun 2024 12:57:50 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 1/5] x86/kexec: do unconditional WBINVD for bare-metal
 in stop_this_cpu()
To: Tom Lendacky <thomas.lendacky@amd.com>, <linux-kernel@vger.kernel.org>
CC: <x86@kernel.org>, <dave.hansen@intel.com>, <bp@alien8.de>,
	<kirill.shutemov@linux.intel.com>, <tglx@linutronix.de>, <mingo@redhat.com>,
	<hpa@zytor.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<rick.p.edgecombe@intel.com>, <ashish.kalra@amd.com>, <chao.gao@intel.com>,
	<bhe@redhat.com>, <nik.borisov@suse.com>, <pbonzini@redhat.com>,
	<seanjc@google.com>
References: <cover.1713439632.git.kai.huang@intel.com>
 <327899e13c4dbbfbe8ca95cab83ba229d17fb74b.1713439632.git.kai.huang@intel.com>
 <1ca2c140-8ab5-46a6-a26e-6840d7ba051c@intel.com>
 <90ee763e-e059-12a4-d7c3-9a710a1c5db4@amd.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <90ee763e-e059-12a4-d7c3-9a710a1c5db4@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a2FtcmVnMVk3ZmFMTXBNdWxQL05PQ1JKdjRRT29FSTY1Vkh5N2NHSzBkbHd3?=
 =?utf-8?B?QXFEa1FjYW1hU1VzQVJmV3hoa29LaUhTK0h1OUVtRStZWkphWjZrajB3ZVc1?=
 =?utf-8?B?ZVdLUXo2VjVkRTZ0eUxwYVNJbzE1NjlCUkZ3QnNSOExEUHpZU3FnTVZtRFA3?=
 =?utf-8?B?c0VHMW9qRUNST1hmcVRFUFhJRVlISU9RUm92Nkc1L0FucEZ3N1hLQWNveVFz?=
 =?utf-8?B?SlZ6ZkJjL1E3YnE1SU9ZQWlKK0tjNUNQam1NWWxyUkhraE1uc3k3T0tTZDc3?=
 =?utf-8?B?STBYdGpSZVZTKzk4bEVndWlUN0dpMmNxbDN2ejFWSk9MOHpMVFY1RzRDVEJL?=
 =?utf-8?B?Zkc2ZnQ0aWg0SFI4Y21zT3JDckVFQmxPZDJ4eDRGS2tlK3UxSEZodktMQmxn?=
 =?utf-8?B?bUFzQ3AzczFvaXNTQmd2VStmRm53OTU0RGtmMFNlQUkxcXcvUHpHOHZVQVR4?=
 =?utf-8?B?UjR1QVlEWGY2clhtSFhYdlRWVzNWWnZFMy9nSjZHMVN1bStITzBwK3huNHRQ?=
 =?utf-8?B?bXJVNnZUaGFKbDdjNHQ0OUlmM0ZCRWNYZkFzNFhUWkdHQ1R5eEliUExaRFNX?=
 =?utf-8?B?bjQvUGpNb1k2SmJuRHk3Q2JVMVVRVi9STHI1bDRueDVIS2YwMzNTSy9tb1pU?=
 =?utf-8?B?ekt4eUNnZWppUnJFQUoxd0ZNcmNPQ1ZCMjhLNnliS3BqWkhpWHNkd0EzSldS?=
 =?utf-8?B?OG5MMFZ1cjlJb1RKOUtKQUZLSnhjY2o1RFpYZ0dhN2FwdktFSU9penRKMlc2?=
 =?utf-8?B?M1VYc2YwR1ExczVGS2pwajc4WndlYUFuQnZRZGY0NXhiaWUxZTljMTFFTzZa?=
 =?utf-8?B?RkVBQmpoZ3pFS0FnRFVtSDZ5djR6ZktrZlZrZlBzUlh3N3R5cG14dXRXTGNy?=
 =?utf-8?B?VFEwb3FSSjhmcHBBbjdwbHBjczhCSHg3MkY3TERHMVpZYnRKajVIblBYY0k1?=
 =?utf-8?B?VU4zQUpyaGJzK0h1SzVrRjUxWld0NGxuOHFZNHRKTzRRY0hpYkxEMUlzQmZF?=
 =?utf-8?B?WDRWdXdZOGI2clF6b0RyRjJIaENSSEJYeGdPS2RwWEJKMDdBSkVOR1JBeHdr?=
 =?utf-8?B?c0wxUzNadytVNi9UeDg0aUR4WCtzaStiWC9od1BXb2lhR1ZYVDFwazJOMDEz?=
 =?utf-8?B?a3ZSWmZzVE5GdG82TkViS1oyUFFTWjR1L0lUejNtbzlRV2RHdWJpcjdZTzBT?=
 =?utf-8?B?SFdrbmhJbHErRHEyenkvVjA0Z2ZVNEdrUjExVG8rZkYwSlZJTld1Unowcm9X?=
 =?utf-8?B?Rm8wdHE2alFLejV4aUkwZHZSbGNnY3JTV1pxeXF5Q0RUWFZSRmlORWdQYmV2?=
 =?utf-8?B?cCtyNEZlZnhKb0hTUTQwR0I1TTZocWRobXcwMGs5aGZtUHRCVWF6TTI1VEFz?=
 =?utf-8?B?ZE44U3hzbjdvaEtGWXp5SXBybmhwT1F1dlFPMGdtb0JNQU5rMi9NZDNhazZp?=
 =?utf-8?B?OTRZQjRyd2gzVUovemxxRjBZTit4ZExNSjUxYTc0a3pEY1h3SmJHaHNQamsv?=
 =?utf-8?B?OHRrSTJxenF2aHRjQ0tRL0FsUGNpR0E2MENuaFVrRFUyYVJsZFdKQTNwZUNT?=
 =?utf-8?B?OVlnUlp2cndIa1lHcDhjWVNldFdaSUhCVk90aXN0TE96Tm9saDJOYXRxWFlo?=
 =?utf-8?B?dFZNQk9HZHY3VGpuTEZQMUVaYlFnTncrbHNJZzBKalRZa29RL2ZKQjZTMEpN?=
 =?utf-8?B?dzBLTk1VZldCMGdGSStvY1BWYnV4UDliWUdsQXlNY3c1aUhaanY5SE1WWFhn?=
 =?utf-8?B?SUdVL09zeW5USkNNY0RwWlp4Y1phZERMK3hHWHQxTDNtTzJ4em5uSlVJam1w?=
 =?utf-8?B?QVNXaXpoRXFyUW94N0Y2RUlPV1FuRjlkK0hhQkNEZWUwYmJlYUhXQzRzVnlO?=
 =?utf-8?B?RnhzaGdhWjFkeUFFbHNpQW13c2JJeGtJU0NtdGNkTUpTdUNkVWV2SFNsck1y?=
 =?utf-8?B?SHRRRURjbHN1Ym5JdEpjVG9VYjhlOGF1eTNFcmIzQ3gwMUkreE5FOUFlS1Ay?=
 =?utf-8?B?RkJyTkRGRldqWnhEeXpyVFowWDBmaUhBVXZKSlo1R2tzdjVlb0VsL21JOTNV?=
 =?utf-8?B?VWd2UjIwdHp6RC9IRnc5NmpLY2kzUVNMeElrVUNkc2NWcHA0enZBdWpwanJU?=
 =?utf-8?Q?29vCfCr9jrAkbi07+l1sMZAp8?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 595856ad-1702-4e79-743e-08dc84316164
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2024 00:58:00.3515
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wmOAWQGx3OSFKvSCsiEfbmU6zVbEePKdJMNcfaCS1Wqk00WqPZIfDEt9diEnf2CZXYmwgTkhi7h7cKdO9dID+A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7688
X-OriginatorOrg: intel.com


On 1/06/2024 8:45 am, Tom Lendacky wrote:
> On 5/22/24 21:49, Huang, Kai wrote:
>> On 18/04/2024 11:48 pm, Kai Huang wrote:
>>> TL;DR:
>>>
>>> Change to do unconditional WBINVD in stop_this_cpu() for bare metal
>>> to cover kexec support for both AMD SME and Intel TDX, despite there
>>> _was_ some issue preventing from doing so but now has it got fixed.
>>>
>>> Long version:
>>>
>>> Both AMD SME and Intel TDX can leave caches in an incoherent state due
>>> to memory encryption, which can lead to silent memory corruption during
>>> kexec.  To address this issue, it is necessary to flush the caches
>>> before jumping to the second kernel.
>>>
>>> Currently, the kernel only performs WBINVD in stop_this_cpu() when SME
>>> is supported by hardware.  To support TDX, instead of adding one more
>>> vendor-specific check, it is proposed to perform unconditional WBINVD.
>>> Kexec() is a slow path, and the additional WBINVD is acceptable for the
>>> sake of simplicity and maintainability.
>>>
>>
>> Hi Tom,
>>
>> May I ask how does SME work with kdump in crash_kexec().  Looking at 
>> the code, AFAICT the crash_kexec() path doesn't use stop_this_cpu() to 
>> stop all other cpus.  Instead, kdump_nmi_shootdown_cpus() is called to 
>> send NMI to remote cpus and crash_nmi_callback() is invoked to stop them.
>>
>> But the crash_nmi_callback() doesn't invoke WBINVD for SME AFAICT.  It 
>> does call the kdump_nmi_callback() callback where a WBINVD is 
>> performed for the SNP host:
>>
>> void kdump_sev_callback(void)
>> {
>>          /*
>>           * Do wbinvd() on remote CPUs when SNP is enabled in order to
>>           * safely do SNP_SHUTDOWN on the local CPU.
>>           */
>>          if (cc_platform_has(CC_ATTR_HOST_SEV_SNP))
>>                  wbinvd();
>> }
>>
>> So if I read correctly, what's the reason the WBINVD is skipped for 
>> SME in case of crash_kexec()?
> 
> The system is rebooted after a crash and doesn't continue directly on 
> into a new kernel.
> 

How about the kdump kernel itself?  Would the stale cachelines 
potentially corrupt it?

And how about /proc/vmcore, which reflects the system RAM used by the 
first, crashed, kernel?  Is it OK to have stale cachelines for it?