Received-SPF: pass (google.com: domain of linux-kernel+bounces-108276-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <256052ee-4e7b-45c5-8399-515fbb529a01@intel.com>
Date: Wed, 20 Mar 2024 13:45:32 +1300
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 2/5] x86/kexec: do unconditional WBINVD in
 relocate_kernel()
Content-Language: en-US
To: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
CC: Tom Lendacky <thomas.lendacky@amd.com>, <linux-kernel@vger.kernel.org>,
	<x86@kernel.org>, <dave.hansen@intel.com>, <bp@alien8.de>,
	<tglx@linutronix.de>, <mingo@redhat.com>, <hpa@zytor.com>, <luto@kernel.org>,
	<peterz@infradead.org>, <rick.p.edgecombe@intel.com>, <ashish.kalra@amd.com>,
	<chao.gao@intel.com>, <bhe@redhat.com>, <nik.borisov@suse.com>,
	<pbonzini@redhat.com>, <seanjc@google.com>
References: <cover.1710811610.git.kai.huang@intel.com>
 <e1d37efb8951eb1d38493687b10a21b23353e35a.1710811610.git.kai.huang@intel.com>
 <tvembdwwh4immxytlfzlhpvd42dlfsz7sddb7msk23kdduhu3t@ogpc66hklorv>
 <38fca2fa-11b2-4eb7-9e59-dc5d524d172e@amd.com>
 <689bbd29-aaf0-452e-a97f-41b8e3aa6224@intel.com>
 <i3nxazyv2dlauias4jmoqwpjixviuduaw6bgtfv4claxtimlm3@54xmat6zqud4>
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <i3nxazyv2dlauias4jmoqwpjixviuduaw6bgtfv4claxtimlm3@54xmat6zqud4>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WU0wM1ZBMkUvVytUNVRwWUU3Vml6MDZEb0xnV1gzbW53blYrUlNhdVNhNGlu?=
 =?utf-8?B?bXlSMGQzQkQ0aVpMYUF3OXV5cDduemdTUnpIcTFETXNWZ1hYZTlpZzhjZng0?=
 =?utf-8?B?S3BDQ3l6T0FLNUxBQ2lGM0x5dDdLS0trQWxuOHhqMWl6ckVUUExPSjNOOHBh?=
 =?utf-8?B?ZkY0R1BGTHMxMHUxcitqL3ZGYnhsNVZOZHlqQmYvdHRqWkRYNVRRSksxWU5i?=
 =?utf-8?B?MDJnbENHYmxMczFhZzE3OUtIMFl5UzB3SHJGT2tvc3lUZU16WkJQY25PNERk?=
 =?utf-8?B?Y2dGZm5HMnMybXVjQ3hKTWg0QmNZRFE5UDRFOWJjQXA5S1V6Ly9XOWlOMlRs?=
 =?utf-8?B?bE0wLzZNenpkMGE2bDBNUDNQNk5GSVRGZUlIQ0JJTEM0bi9WU0NXTkVqem0x?=
 =?utf-8?B?anlDVitvOGh2eTViRGV1WUJDc3FEcmQvcVJCRXZJejNrQU5KWFZmeE1GY2FI?=
 =?utf-8?B?QnZzWlBaaVpjRysxM3gxQSswdWw4blJ5bnN0MndKWSt5bGwxdkxGQjM3RmpW?=
 =?utf-8?B?YTE3OWJRN0IrckVxekxrcndxR3NwRWlUMG41TjYyU1BXZUNIdkZxWnU0Y0Uv?=
 =?utf-8?B?dVJKalFnWlN3ZElUeTRxVHBnaW1aODVodUpEa0pJS3ltTnRxdlhXSVJQZjNr?=
 =?utf-8?B?ZDBCVUc5RDVremU1QmpMT0JXTTNiV1ZlSGkvNFN2TXRubVFYdTkrMTV1T25L?=
 =?utf-8?B?OEF5NWsxRlNoRnVNNDJUOXo5cGQ2ZzZscm9aQUJaWm9CeHZBVVFpd0FPaGlT?=
 =?utf-8?B?VENqQUwwY0xCMWxEWTVNYlpIQXl6Nld4RnZrRSthNm5DQUprZ3ExMmtjYXdK?=
 =?utf-8?B?cGdsR1NNVHlWNFBCRnhENnpMWk5MWUgvYXQwUmhPRjlaeUU0MlN5SDFsZEIr?=
 =?utf-8?B?YXBJWVpUWE9PVVFlR3JHSXFvSXg0UGtubVJ2NytES2dOQTA1blBPUlFLVzJT?=
 =?utf-8?B?Q2R5VStNa2V3QS9Ca2dhOHRsN2lWMW0yc2s1d21JaTJvMnhseGwvWDVnN2FD?=
 =?utf-8?B?SXBaZk11ZHRqWDdiZTBQYzVIUWlGMHpiMXBSMGhkcWtSNGNJNjB0YjZrZlg1?=
 =?utf-8?B?YndJMUJuazREWEczc3M0RzQrYTFoTEUxSGlSM253aktmeitaMHpacFpZcnMy?=
 =?utf-8?B?UXI4aXhuSm9FLzF1QkRyUGVQK05kUGNLZTRpdlRmVXNJNmFUUE5NeWRPV3h2?=
 =?utf-8?B?K21wWnFRb09NVkFySHN1TGZJY0lkUWFIdS9KaEFsSU5nNDdCKzE2ZVllb210?=
 =?utf-8?B?aDNSSmcyeGNSeG5JUFJ0WmdMS0JtN1pUSGo0NVd4SHlvalZQSmZuTE5zVGs0?=
 =?utf-8?B?UEcvajkwc3ZkWUhzaHc0Z0xJeDA3cDNPZGpVNERyMDNQdnpVaDFneGlZVlRN?=
 =?utf-8?B?REZjWlhJVWRkVmo5OHVUSWFjQXAyWisyYlJTaVhkV1ZUaGJ6MDJzQXVsZHRv?=
 =?utf-8?B?SU1xNzlUL0NJVTN2OTVUR2h6QS9JWGlFUlFyR21makovd3BVQkM3blFCdktS?=
 =?utf-8?B?SVV1aCtualgxNXV3Z0tQbGtQbnZ5aVNCYW8zVEMxMTRKOHJqc3JvblpaaEVW?=
 =?utf-8?B?aWZLWVFCek04dldGVlBGWVJyWVJxNXlNWlY3U05NMlVGY3V0SUxLQyt0K0JO?=
 =?utf-8?B?WSs4TWRvMkY1bW1lRjN0eGRJczhoNDBjckFMNnNvQjh4aE1EMXZwdlQ5dUZY?=
 =?utf-8?B?azcxbmlIb2lCMWxGK3ovTDBZay9sSFB2eW0xRGpqUTdCNGdZaDJqdGJqcUVV?=
 =?utf-8?B?YVBoUmY3RUczWHdWSklKTnk4NkJDeGxwbDQzcXVLN1hnT3ZZbmR4ZzN2RExq?=
 =?utf-8?B?dVhCR2N2azZjV09Ya1h1U1JJMndIUXpKdWorUkJEd3ErU1J4S1oyRytsd3Nw?=
 =?utf-8?B?bmRzVkZFckdwMGtDS3FpbEF5bWV1R24yOXE5dy9QVldKY3JkUGlsdkpwNmJo?=
 =?utf-8?B?YnB6QzhyYzRFZGNvWGVhSXp3TTF4MVk4WTY5RU92ME02OGRWaE82R3BCTmV0?=
 =?utf-8?B?a2pYYnhRQVFlYjNtMlYrSFFwaTNGV0NNd3F1SmZ2Y3hnWWs0UHFIVlNROW4z?=
 =?utf-8?B?d1JHRXhUUnZZQ2tqWHA3a0VLSi84SnlOaE9DTVFGQTh5UGdqSTNJeFdqWVBU?=
 =?utf-8?Q?rNF1ta7hkhvxEJWhSmqZeymy3?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 084b6a2d-d0a7-4dc9-c0f4-08dc48771407
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2024 00:45:45.5730
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9v++drJa51Pl1JzPsQvc65lqWGh5dYAY4D6T0l9Yd7e3SanmFgXseMu2QjRppEy92xAiOy6pmzRiej6JYxkhRA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5183
X-OriginatorOrg: intel.com


On 20/03/2024 1:19 pm, Kirill A. Shutemov wrote:
> On Wed, Mar 20, 2024 at 10:20:50AM +1300, Huang, Kai wrote:
>>
>>
>> On 20/03/2024 3:38 am, Tom Lendacky wrote:
>>> On 3/19/24 06:13, Kirill A. Shutemov wrote:
>>>> On Tue, Mar 19, 2024 at 01:48:45AM +0000, Kai Huang wrote:
>>>>> Both SME and TDX can leave caches in incoherent state due to memory
>>>>> encryption.  During kexec, the caches must be flushed before jumping to
>>>>> the second kernel to avoid silent memory corruption to the
>>>>> second kernel.
>>>>>
>>>>> During kexec, the WBINVD in stop_this_cpu() flushes caches for all
>>>>> remote cpus when they are being stopped.  For SME, the WBINVD in
>>>>> relocate_kernel() flushes the cache for the last running cpu (which is
>>>>> executing the kexec).
>>>>>
>>>>> Similarly, for TDX after stopping all remote cpus with cache flushed, to
>>>>> support kexec, the kernel needs to flush cache for the last running cpu.
>>>>>
>>>>> Make the WBINVD in the relocate_kernel() unconditional to cover both SME
>>>>> and TDX.
>>>>
>>>> Nope. It breaks TDX guest. WBINVD triggers #VE for TDX guests.
>>>
>>> Ditto for SEV-ES/SEV-SNP, a #VC is generated and crashes the guest.
>>>
>>
>> Oh I forgot these.
>>
>> Hi Kirill,
>>
>> Then I think patch 1 will also break TDX guest after your series to enable
>> multiple cpus for the second kernel after kexec()?
> 
> Well, not exactly.
> 
> My patchset overrides stop_this_cpu() with own implementation for MADT
> wakeup method that doesn't have WBINVD. So the patch doesn't break
> anything, 

Well, your callback actually only gets called _after_ this WBINVD, so...

I guess I should have that checked by myself. :-)

but if in the future TDX (or SEV) host would use MADT wake up
> method instead of IPI we will get back to the problem with missing
> WBINVD.
> 
> I don't know if we care. There's no reason for host to use MADT wake up
> method.
> 

I don't think MADT wake up will be used at any native environment.

Anyway, regardless whether patch 1 will break TDX/SEV-ES/SEV-SNP guests, 
I think to resolve this, we can simply adjust our mindset from ...

	"do unconditional WBINVD"

to ...

	"do unconditional WBINVD when it can be done safely"

For now, AFAICT, only TDX guests and SEV-ES/SEV-SNP guests are such guests.

And they all report the CC_ATTR_GUEST_MEM_ENCRYPT flag as true, so we 
can change to only do WBINVD when the kernel sees that flag.

	if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
		native_wbinvd();

Alternatively, we can have a dedicated X86_FEATURE_NO_WBINVD and get it 
set for TDX/SEV-ES/SEV-SNP guests (and any guests if this is true), and do:

	if (!boot_cpu_has(X86_FEATURE_NO_WBINVD))
		native_wbinvd();

It seems the first one is too generic (for any CoCo VMs), and the second 
one is better.

Any comments?

Hi Boris/Dave,

Do you have any comments?