Received-SPF: pass (google.com: domain of linux-kernel+bounces-197241-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Fri, 31 May 2024 10:17:19 -0700
From: Isaku Yamahata <isaku.yamahata@intel.com>
To: Kai Huang <kai.huang@intel.com>
Cc: linux-kernel@vger.kernel.org, x86@kernel.org, dave.hansen@intel.com,
	bp@alien8.de, kirill.shutemov@linux.intel.com, tglx@linutronix.de,
	mingo@redhat.com, hpa@zytor.com, luto@kernel.org,
	peterz@infradead.org, rick.p.edgecombe@intel.com,
	thomas.lendacky@amd.com, ashish.kalra@amd.com, chao.gao@intel.com,
	bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com,
	seanjc@google.com
Subject: Re: [PATCH v4 5/5] x86/virt/tdx: Add TDX memory reset notifier to
 reset other private pages
Message-ID: <20240531171719.GA380706@ls.amr.corp.intel.com>
References: <cover.1713439632.git.kai.huang@intel.com>
 <c56350fc0d79724a1206708a436cd2847c3b6ee7.1713439632.git.kai.huang@intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <c56350fc0d79724a1206708a436cd2847c3b6ee7.1713439632.git.kai.huang@intel.com>

On Thu, Apr 18, 2024 at 11:48:05PM +1200,
Kai Huang <kai.huang@intel.com> wrote:

> TL;DR:
> 
> To cover both normal kexec and crash kexec, add a TDX specific memory
> reset notifier to let "in-kernel TDX users" use their own way to convert
> TDX private pages (that they manage respectively) in tdx_reset_memory().
> 
> Long version:
> 
> On the platforms with TDX "partial write machine check" erratum, during
> kexec, the kernel needs to convert TDX private memory back to normal
> before jumping to the second kernel to avoid the second kernel seeing
> potential machine check.
> 
> For now tdx_reset_memory() only resets PAMT pages.  KVM will be the
> first in-kernel TDX user to support running TDX guests, and by then
> other TDX private pages will start to exist.  They need to be covered
> too.
> 
> Currently the kernel doesn't have a unified way to tell whether a given
> page is TDX private page or not.  One choice is to add such unified way,
> and there are couple of options to do it:
> 
>  1) Use a bitmap, or Xarray, etc to track TDX private page for all PFNs;
>  2) Use a "software-only" bit in the direct-mapping PTE to mark a given
>     page is TDX private page;
>  3) Use a new flag in 'struct page' to mark TDX private page;
>  4) ... potential other ways.
> 
> Option 1) consumes additional memory.  E.g., if using bitmap, the
> overhead is "number of total RAM pages / 8" bytes.
> 
> Option 2) would cause splitting large-page mapping to 4K mapping in the
> direct mapping when one page is allocated as TDX private page, and cause
> additional TLB flush etc.  It's not ideal for such use case.
> 
> Option 3) apparently contradicts to the effort to reduce the use of the
> flags of 'struct page'.
> 
> None of above is ideal.
> 
> Therefore, instead of providing a unified way to tell whether a given
> page is TDX private page or not, leave "resetting TDX private pages" to
> the "in-kernel user" of TDX.
> 
> This is motivated by the fact that KVM is already maintaining an Xarray
> to track "memory attributes (e.g., private or shared)" for each GFN for
> each guest.  Thus KVM can use its own way to find all TDX private pages
> that it manages and convert them back to normal.
> 
> For the normal kexec the reboot notifier could be used, but it doesn't
> cover the cash kexec.

If we do so, KVM needs to traverse complex data structures.  Anyway, we
can miss TDX private pages in a transitional state between private-shared.
Also, we have to maintain the logic if the data structure changes, both
the KVM code change or TDX module data structure update for future features.

Here are other options.  What do you think?


Option 5) refuse normal kexec when TDX module is initialized.
Enforce the user to go system reset.  For kdump case, we know that TDX
private page resides in the area of the crashed kernel.  Kdump kernel
will use reboot with BIOS after capturing crash image.

Pro: No need to traverse complex data structures
Pro: Keep room for future client support without complication
Con: LINUX_REBOOT_CMD_KEXEC can fail


Option 5.1) refuse normal kexec if TDX guests are running.
The admin needs to kill the TDX guest be kexec.  Perhaps update
systemd unit file.


Option 6) Clear the all pages instead of selected ones.
We can use the e820 entries passed to the second kernel.

Pro: No dependency on in-kernel TDX users.
Pro: Work for possible future TDX module data structure change
Con: It may take long time if the memory size is large.
     If TDX guests use most of the available memory, this is inevitable.
-- 
Isaku Yamahata <isaku.yamahata@intel.com>