Received-SPF: pass (google.com: domain of linux-kernel+bounces-197463-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <90ee763e-e059-12a4-d7c3-9a710a1c5db4@amd.com>
Date: Fri, 31 May 2024 15:45:09 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [PATCH v4 1/5] x86/kexec: do unconditional WBINVD for bare-metal
 in stop_this_cpu()
Content-Language: en-US
To: "Huang, Kai" <kai.huang@intel.com>, linux-kernel@vger.kernel.org
Cc: x86@kernel.org, dave.hansen@intel.com, bp@alien8.de,
 kirill.shutemov@linux.intel.com, tglx@linutronix.de, mingo@redhat.com,
 hpa@zytor.com, luto@kernel.org, peterz@infradead.org,
 rick.p.edgecombe@intel.com, ashish.kalra@amd.com, chao.gao@intel.com,
 bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com
References: <cover.1713439632.git.kai.huang@intel.com>
 <327899e13c4dbbfbe8ca95cab83ba229d17fb74b.1713439632.git.kai.huang@intel.com>
 <1ca2c140-8ab5-46a6-a26e-6840d7ba051c@intel.com>
From: Tom Lendacky <thomas.lendacky@amd.com>
In-Reply-To: <1ca2c140-8ab5-46a6-a26e-6840d7ba051c@intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?VG9QZmpLcFpsRi9TRkJpQkpNY1RqZEcvNmIzSG9vbXoyby9YNFY2MFYyTlB2?=
 =?utf-8?B?K3ZBTDg5TzBCN1J6a1JocUoweStrQ2o3eXVTK2NGUzJFdnpPNDA2N291Tkw0?=
 =?utf-8?B?R05OTHMrZkppcSt0T1FDVWFQNU45SmR6TEZpb1NPN1ZoNFlTcmZEancrRDRD?=
 =?utf-8?B?V2lMMHpjb3Jhd1pyY3pFVUVkUzJjRmtNNVlZZlc4RkVyWlc1YnRFYzIxYjVx?=
 =?utf-8?B?anMxb1Btc01lYk9uZkw5UXRHVWVsL1FrZlppeHlwcFN0dXArSGM1WEV3dm9R?=
 =?utf-8?B?eWhqOUtLTEVMSERTcG55MU54Mm9nWU11dUNlR3lLaDJ0d0pGR2VWV2tXQ2JM?=
 =?utf-8?B?bXdodGRmc2NCWTRueGYzUXlGQTRVMkxsSEVZbjZIRU16QmZxMFcvWFlFdXJC?=
 =?utf-8?B?WlJZWlZXb0pQTXJjclE0STBlQjRKbFpjdllnc2pWWUQ1Vkd3aXl4d0FlUjJz?=
 =?utf-8?B?R2V2UzNIYk5OS1NFN1ZEeWJOckZpRE5FT2hXeEErdFFKaTlBdUZqRTlPMThV?=
 =?utf-8?B?YlNwa1RkV0Rib0QwRFVad1paZ294YmZYNlE0ZStEd2I1aldCdVNVYzdwZ2Fr?=
 =?utf-8?B?TUF6OVA2bUlVblB2MmxSMk1OY1E3K0xWSENUQWdienhINGV6KzY2ZVgvQXJL?=
 =?utf-8?B?SDJuYW9GbWxRdjJ4M0dkUnczMXMybCs1b0QrVVpjdmRFTGcrL3c3VEdvMStM?=
 =?utf-8?B?LytUWEhlOFh1cHJYdEd0U1ZnVXlacEhIanZNcFFINklQVGRzclhLeFBNaHFS?=
 =?utf-8?B?bTdENjZ5L3ppcFlKcWdLeG4vVlpPMzk0enpsQk9SWHhvTkpFZ291Ly9CRHpJ?=
 =?utf-8?B?UDBWTDkwbVFVenIyNkd5VEM5VTBGU0NEMEdMbDJKdUc5OXJXSUpzci84OHUy?=
 =?utf-8?B?cm1zYkF6QVZaQmtuOWVTUGhwaWVTd3NRT3RORWNOOE5vQmkvaXVBWEg1anRG?=
 =?utf-8?B?eVQ3cTE4ay9kN0xFZEx1WE5nRTdta0tXZkx2RHNDY01YOGdyVFRYdTE0M3N2?=
 =?utf-8?B?dE92ZFhaczQvWHk0TGt3bGJXeHBQYk1UR0MvTUFPQVZ1aytyUEFNSmNzKytU?=
 =?utf-8?B?VldKcWgwRkVKdDYyL2NqY1VhSGUwa0JhditTbnVQZzlTN2lIKzJ1SkpuZzJa?=
 =?utf-8?B?eUVrYlppTkZmTnBmRkJpNjZiaC9xZFNENW9TOG1lNUo5a1pmclVMSmkvNGVZ?=
 =?utf-8?B?YTViaCt1bmR6TzZGR3d2cFVLaFlCWDlVZ0w3LzBLYkpuM0hwME1mdkRrMFZ5?=
 =?utf-8?B?MVpsNm9vUWs1dWVrdTVQMGlUSU90dGNLOVQvOUtlaWMyaWluSkNHalZMaGpw?=
 =?utf-8?B?M3BrTFhRMXJVcUR1MGVNcjluSklxVm9tR01jWlN0bTR0c1hTWkE2aThEUzFa?=
 =?utf-8?B?eGJSaVlmN2t3V1dSOVBkZXlVMlRWRWxxL1dlRzhMTm0yaFJRZ2o2bHh4ZE9v?=
 =?utf-8?B?d1BreExTQnBtYXpNUnRadVMxQUFpanRHdmk5S0pIcHhDN3BUN002R2xXanBV?=
 =?utf-8?B?eHRJcXZLMFFmU084UVQwU0pjVmdsc2ZPV3FqRi9USTIzQ0tMSmJTY3M1YVRY?=
 =?utf-8?B?bWpDWEs0V0ppL0RHV3pFRXZYZmVwc2lpbE5lOW5pQWxiNGlROUc2RzAwUHNu?=
 =?utf-8?B?cUxIL09FV1VmV3hRdURCcTJwTU1tbDJ5aTRJclRlanRlQmZXQzJmUFZlTWNj?=
 =?utf-8?B?NXpBSCtjVERDNUM3VFg3Z1pWVnFscjAvLzJEek4zUk5GMG90a1pTVWFuQWxL?=
 =?utf-8?B?QUlTaGxPUVIwWStvZVhwU3FDTXRsbHVFcFNQQjhtNTlNNjJzSXFsSXZyOU9o?=
 =?utf-8?B?QU5COTFKeVpYZDg4RDBQK0UydjNGRkVtYzVNZGxLbmhwNzU4VXZkZlJRQmxQ?=
 =?utf-8?B?NHZvK3FuUCtuY0MxRHFucnAveFZHRXZPdmtDbGlxWGJHY3BFQzhteEV1bVZi?=
 =?utf-8?B?aUZGMG0ybjZxUHV1UHN3bC9YOTFWQWsxQWk0bmxiMnYrbFJ6eGV2S1AwSkd4?=
 =?utf-8?B?aExsQWxad0FIMVhiSHNJM1hhTlBsNkpaVHRzZHBkK084U3RWWjZ6Qm5BZ2VB?=
 =?utf-8?B?Vjkxcjg2aDJwWDNwM2JLTDFrZDJENloyejRNK0VxSzg0V3Y0WmdMckFNMC9v?=
 =?utf-8?Q?yGSX1QeUVlEuD3JPoq0NDy8f2?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 961e87fd-5622-442a-3042-08dc81b29077
X-MS-Exchange-CrossTenant-AuthSource: BL1PR12MB5732.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 20:45:10.9049
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: bwXUKh/YA5jXgY9/SuCDsY4Xu9/ygiwFPo6DN2zP42IUQHVotztF6kRxbx0PqJUSYGH4/uiqmAL2g08dGiwzbA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7761

On 5/22/24 21:49, Huang, Kai wrote:
> On 18/04/2024 11:48 pm, Kai Huang wrote:
>> TL;DR:
>>
>> Change to do unconditional WBINVD in stop_this_cpu() for bare metal
>> to cover kexec support for both AMD SME and Intel TDX, despite there
>> _was_ some issue preventing from doing so but now has it got fixed.
>>
>> Long version:
>>
>> Both AMD SME and Intel TDX can leave caches in an incoherent state due
>> to memory encryption, which can lead to silent memory corruption during
>> kexec.  To address this issue, it is necessary to flush the caches
>> before jumping to the second kernel.
>>
>> Currently, the kernel only performs WBINVD in stop_this_cpu() when SME
>> is supported by hardware.  To support TDX, instead of adding one more
>> vendor-specific check, it is proposed to perform unconditional WBINVD.
>> Kexec() is a slow path, and the additional WBINVD is acceptable for the
>> sake of simplicity and maintainability.
>>
> 
> Hi Tom,
> 
> May I ask how does SME work with kdump in crash_kexec().  Looking at the 
> code, AFAICT the crash_kexec() path doesn't use stop_this_cpu() to stop 
> all other cpus.  Instead, kdump_nmi_shootdown_cpus() is called to send NMI 
> to remote cpus and crash_nmi_callback() is invoked to stop them.
> 
> But the crash_nmi_callback() doesn't invoke WBINVD for SME AFAICT.  It 
> does call the kdump_nmi_callback() callback where a WBINVD is performed 
> for the SNP host:
> 
> void kdump_sev_callback(void)
> {
>          /*
>           * Do wbinvd() on remote CPUs when SNP is enabled in order to
>           * safely do SNP_SHUTDOWN on the local CPU.
>           */
>          if (cc_platform_has(CC_ATTR_HOST_SEV_SNP))
>                  wbinvd();
> }
> 
> So if I read correctly, what's the reason the WBINVD is skipped for SME in 
> case of crash_kexec()?

The system is rebooted after a crash and doesn't continue directly on into 
a new kernel.

Thanks,
Tom