Received-SPF: pass (google.com: domain of linux-kernel+bounces-186925-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <1ca2c140-8ab5-46a6-a26e-6840d7ba051c@intel.com>
Date: Thu, 23 May 2024 14:49:41 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 1/5] x86/kexec: do unconditional WBINVD for bare-metal
 in stop_this_cpu()
From: "Huang, Kai" <kai.huang@intel.com>
To: <linux-kernel@vger.kernel.org>
CC: <x86@kernel.org>, <dave.hansen@intel.com>, <bp@alien8.de>,
	<kirill.shutemov@linux.intel.com>, <tglx@linutronix.de>, <mingo@redhat.com>,
	<hpa@zytor.com>, <luto@kernel.org>, <peterz@infradead.org>,
	<rick.p.edgecombe@intel.com>, <thomas.lendacky@amd.com>,
	<ashish.kalra@amd.com>, <chao.gao@intel.com>, <bhe@redhat.com>,
	<nik.borisov@suse.com>, <pbonzini@redhat.com>, <seanjc@google.com>
References: <cover.1713439632.git.kai.huang@intel.com>
 <327899e13c4dbbfbe8ca95cab83ba229d17fb74b.1713439632.git.kai.huang@intel.com>
Content-Language: en-US
In-Reply-To: <327899e13c4dbbfbe8ca95cab83ba229d17fb74b.1713439632.git.kai.huang@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cm13NTlleDN3QTJGRnFsbmN0NzlHT21reFhPVGhYTElrdktvNUVxMWtWdVlq?=
 =?utf-8?B?VnA3alUwaXJlVW1xRHd5eXZ6bnI0UWkwZzUyUitBYXk1SHRnQVJYRk15M3Jp?=
 =?utf-8?B?VmxJdVdjUmxzTFJ1dy9TdWZ1TTk4eGpOQ1c1MVdwTTVlcnhUZmFiMXJmNWNT?=
 =?utf-8?B?OXZoZ042cHBjYnc3NklSbjVRYjJwbDlEd0x1Y1JpMEdkdXJTNVhIMTBZdEdI?=
 =?utf-8?B?MVVaa3Q1WDdmbnFyVExRTVZvS1ZhQlc5SC9PV1ViRHlVS3B4ZTRWOE8xZlF3?=
 =?utf-8?B?UWRLZG4xODRSR3l0RmQxd3kxcXBsV3BUNXBJZ3g3WUh6RndkM2NsSDM2aVhP?=
 =?utf-8?B?cmt4U0JvVTI4OUc1d3dNWlRmbG9TdUhFMnBXOUFLam1RSHZUQ0FxUnVlUVlS?=
 =?utf-8?B?VEhjNnFvajVQdEFRc1VITEhZVEdLc2FPclJXL0dWeXhlN3F3Y0xDalUxOEFy?=
 =?utf-8?B?QkFsdU05UGNtMjBvY1FyN1BLNkNzT2JqaFpYQTJZbG1EN0N3dk1lYWg2QkFr?=
 =?utf-8?B?NDNlNlYzcWZsWEdWRE8wM3FCNFFpOUk4NzFxb3FlRm9KTWI3YWlRRloramN4?=
 =?utf-8?B?eXlBZnlPcXZNY3lCV1p2dlE4MHhlMW1sZmRzN3AzT3N0Y2tTdzdHMGpobGFP?=
 =?utf-8?B?RTlJeHR1NStOQlpZKzQwazBXT1dyTTZ4SkdKSDJZWXVxRm5aV09hNjArZ3cz?=
 =?utf-8?B?SVY3cWVGTDhLbWd4d2prSXNYV1VUZFBJYy9uYnRmV05neFJMZmxnRGNoUDZr?=
 =?utf-8?B?RU1lYkxtbjlsV3RrVUhqN1VSejJIcjF2ZTl5aUhDdHl1YUFQdnVUZHhCaUJ0?=
 =?utf-8?B?WUo2MTV6eVhLd0VmOHdjVWtVZ3d2eStvZ0h6YTJPdWtDdmpkT3ovZjJQZXVB?=
 =?utf-8?B?QW12cGJSSzlONkk4ZnRBcWI0REwyclFxc0E0eW9oSVJDZFNVZUYvY2UxZDly?=
 =?utf-8?B?ZjhXdUkyL3paWm1UV3NqR1hEYy92UDZMMzIxUVMrUTVVa2RybURHdkhZVVJV?=
 =?utf-8?B?R1BxS3RRV3JRMksxVHVDSVF6RmJyNktsbnBvNmpLYWt3QzFLcVpMZTZsR2RX?=
 =?utf-8?B?ZXVyYnpJWllGVE5lN05xMUJhekxpOTRvYWxrQ2ZTd2tqUlM5eDkyT245VUdt?=
 =?utf-8?B?T0h2Q2NxelcxakFIUlZ5ckRJNFNvY2hHd2kwMUcrbTZieUxUUnlJWTFUQlZN?=
 =?utf-8?B?NXR1NzQza0x6d2lMU2ZkVlBMbGZPK3A5cDhtVWczSWV1RlZFaG9sVnVCZjFx?=
 =?utf-8?B?WDVMV3czUndiMTZ2T21URE4rbnFXZm0vMDI4L2Y1RTMxbzJMampXMXZCa1ZC?=
 =?utf-8?B?dGExYWdsd3V4cHpSZDlaK1lIVXlxVEVsbzJmZE4ydkd1Qzg5d1FUcTAwU2pL?=
 =?utf-8?B?VHlKUWp3bE1sWXhRMWxiOVBBeFNzaEhQNzRseHFvYm5Lc1M1cVNxeTNLS3li?=
 =?utf-8?B?ejBkTmlmRDhqSGIxSW9VUWJueDd0aDlzRkRVbi9qazQ5NTI3SDIxV2g5SGJU?=
 =?utf-8?B?L3ZNQnAwbkFZaFVkdDBEYXI2eGhWOXh3MzZucG9walhlMXVSQ09pL2N1cTlt?=
 =?utf-8?B?N2hwV01RNUtHM3JjNndlRGNMWjdmREpZbDRaN1JNOEdqdzl3OVpCN2ZqeSti?=
 =?utf-8?B?VmllUCtHYWoyYVFmMHkrVkh3U28wcFU3S1JrT2NLRVNsRGtzS1dtT2JJcktn?=
 =?utf-8?B?RTNGOFpOOWljRURUOEJGbzlOV1Bkc2t1Y3dlb3ZXQVlvZDlSRkJuV3g2TG9Z?=
 =?utf-8?B?cXBqN1NpZ3ZQZUhpWEZIMlo3S3ovVkthMGMzMTFsTVBOODVhWHhJUXNIWVRY?=
 =?utf-8?B?RHR2czNROWRRTnN2M3VyZms3TlNPY1Vvd1NmR2tzM3Z1OXBrS0xHL2xNbkx1?=
 =?utf-8?B?NFJ0OW55bWRYUXoyTk45WDRyaTRnYmtPYnBTRkFzZ1Y3a0xzU0w5RWNGL05I?=
 =?utf-8?B?QkgveGZML2xXQ1dKcU8wbk12OC82UFhDbTFhYS9kMDBzK0MvaElLOEJub2Jz?=
 =?utf-8?B?dzBKY2g2dUI2am1EcDdramRLUFZvTE45bnptTk1mNjQyRXpEVnRHd3ZPNDNV?=
 =?utf-8?B?LzV1NDhPS1RFalprcU5VS2xaUzZOTk5DOUZRcHNVY2p6bFN5eVBYVHhnb0dt?=
 =?utf-8?Q?g3j+y6SDfb9mIb004xHHDxsCa?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2483315e-a6f3-4cd9-9a0b-08dc7ad305eb
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2024 02:49:53.6706
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: quK1eRjithx4SfGsWnlmMRRI7TSCJcmWg//eaHdWSKJxXA+K87KBWkqqFh3GWhUC6pliVKPZfwADxi9ndAosxA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8186
X-OriginatorOrg: intel.com


On 18/04/2024 11:48 pm, Kai Huang wrote:
> TL;DR:
> 
> Change to do unconditional WBINVD in stop_this_cpu() for bare metal
> to cover kexec support for both AMD SME and Intel TDX, despite there
> _was_ some issue preventing from doing so but now has it got fixed.
> 
> Long version:
> 
> Both AMD SME and Intel TDX can leave caches in an incoherent state due
> to memory encryption, which can lead to silent memory corruption during
> kexec.  To address this issue, it is necessary to flush the caches
> before jumping to the second kernel.
> 
> Currently, the kernel only performs WBINVD in stop_this_cpu() when SME
> is supported by hardware.  To support TDX, instead of adding one more
> vendor-specific check, it is proposed to perform unconditional WBINVD.
> Kexec() is a slow path, and the additional WBINVD is acceptable for the
> sake of simplicity and maintainability.
> 

Hi Tom,

May I ask how does SME work with kdump in crash_kexec().  Looking at the 
code, AFAICT the crash_kexec() path doesn't use stop_this_cpu() to stop 
all other cpus.  Instead, kdump_nmi_shootdown_cpus() is called to send 
NMI to remote cpus and crash_nmi_callback() is invoked to stop them.

But the crash_nmi_callback() doesn't invoke WBINVD for SME AFAICT.  It 
does call the kdump_nmi_callback() callback where a WBINVD is performed 
for the SNP host:

void kdump_sev_callback(void)
{
         /*
          * Do wbinvd() on remote CPUs when SNP is enabled in order to
          * safely do SNP_SHUTDOWN on the local CPU.
          */
         if (cc_platform_has(CC_ATTR_HOST_SEV_SNP))
                 wbinvd();
}

So if I read correctly, what's the reason the WBINVD is skipped for SME 
in case of crash_kexec()?