Received-SPF: pass (google.com: domain of linux-kernel+bounces-181702-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <ffd24fa5-b573-4334-95c6-42429fd9ee38@intel.com>
Date: Fri, 17 May 2024 14:36:43 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 10/16] KVM: x86/tdp_mmu: Support TDX private mapping for
 TDP MMU
References: <20240515005952.3410568-1-rick.p.edgecombe@intel.com>
 <20240515005952.3410568-11-rick.p.edgecombe@intel.com>
 <12afae41-906c-4bb7-956a-d73734c68010@intel.com>
 <1d247b658f3e9b14cefcfcf7bca01a652d0845a0.camel@intel.com>
 <a08779dc-056c-421c-a573-f0b1ba9da8ad@intel.com>
 <588d801796415df61136ce457156d9ff3f2a2661.camel@intel.com>
 <021e8ee11c87bfac90e886e78795d825ddab32ee.camel@intel.com>
 <eb7417cccf1065b9ac5762c4215195150c114ef8.camel@intel.com>
 <20240516194209.GL168153@ls.amr.corp.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
To: Isaku Yamahata <isaku.yamahata@intel.com>, "Edgecombe, Rick P"
	<rick.p.edgecombe@intel.com>
CC: "kvm@vger.kernel.org" <kvm@vger.kernel.org>, "pbonzini@redhat.com"
	<pbonzini@redhat.com>, "seanjc@google.com" <seanjc@google.com>,
	"isaku.yamahata@gmail.com" <isaku.yamahata@gmail.com>, "sagis@google.com"
	<sagis@google.com>, "Aktas, Erdem" <erdemaktas@google.com>,
	"dmatlack@google.com" <dmatlack@google.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, <isaku.yamahata@linux.intel.com>
In-Reply-To: <20240516194209.GL168153@ls.amr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Rk9vU290OXBtb3drR0tGK1U1ZjdhN3pPTForb3pYbytnMjJUMk15eHZlZzVL?=
 =?utf-8?B?ZERkc1JmSDhLeWVFZUtRc2E3a3RZZUJDVVl1aXoycU1iZlY4RkJSNWw2cWtq?=
 =?utf-8?B?ak4zSW9rSG82YVY0cGg1dEs0Wm9mNXFhR2xKdFhLcW9SRHI4Y29xckpjTkdV?=
 =?utf-8?B?RlZKcmNhVGpJNXVvME1ITDZQcWt4TDA1dUYyMTRWV24rdm1PcHBFZmlFU3dY?=
 =?utf-8?B?ZEE0RHBReVpVMmhHVENDbDRXOHJXZVRJaWs3WVpRV2tBYU1XUDgrS3JCckc5?=
 =?utf-8?B?L0ZlQXAxMDlmeVhQaWJYMGU3UlJta1FCNjJRQ1NKL1dTR3hOMTV6TFNRM2RP?=
 =?utf-8?B?MDkzVHY4bWhFaHVRNW9TMDlXMDhrS3ZLMEgwcWQ1dW1tOUVuSDBIMk51eFlw?=
 =?utf-8?B?ZUJhK05hWVNCNy8xSml0R0pObTV2SjRXdGluaC9BZ0p1MU1PMWE1Y1V2Kzkr?=
 =?utf-8?B?a0wzSStCd1FtZW5EM2NBVlZFVnd2Z2JxTjB1WWZzaWJoc0RLTjk5aDVwendJ?=
 =?utf-8?B?c0cvZzB1NlpkYUliV2h6SGhObUpzSlNyYThOa3BGMThqMmlUbkdxNnM1NCtQ?=
 =?utf-8?B?SGhyQWNNZlVpR0EvOXZYalltYzJHRzNUMUpoYWxLSHhsOVRFTU13elMwblIv?=
 =?utf-8?B?V1kzbWhhK0JVTVNLOHJMNUJ5RXhRV3F4NjhVYjJRUVpranNCaWlwY1dXZXdx?=
 =?utf-8?B?QzNZT3VTVUdvajFnd09lSVQvODloamh4SVFiZUFNejBFczJyM2R2cUpqUTFv?=
 =?utf-8?B?c2diQ08wVWtrZlR1SmdkcmZkN21zeld0V3daQWpQRUFrRlRjK3JvVEo1Tjh2?=
 =?utf-8?B?blBlQUdUQWtWM2dBMVJLNkgrUklWTS9DMWViY1E3N1YwQmtCZ295ZkZCcFIx?=
 =?utf-8?B?bzcvazFKL1E0WEJWYTZHak5HWjh3YWV5MmdqR1gwSWFMSHpaVTF1UFV0clY0?=
 =?utf-8?B?dmtaa0EraktoaTVaOTlkakl2U0hpTGd6cXptNTZvZ0ZqSmptL01QcEw4alVl?=
 =?utf-8?B?eDlmUDJzZDBTamNHTC9ra2YwNUlxVWlCUlpVTWVrVlRWZks4dkVkYzlUUmxR?=
 =?utf-8?B?eThRSDFaejE5bHVpK0NLSnhxbjR5M1V4bWpnRmF0U21uK1psNDdNMUc5WWZN?=
 =?utf-8?B?V1hxZDlKdkV2djJiQXM3KytiUEtoVWo4MmMxR3NncGhqTzgwREdHMkFtODBG?=
 =?utf-8?B?YS9mZlE3c1EybnhiMUdwVjFldkVQV1RlYkxvMisvR3ZvSExvbE1ZU1EzdEJU?=
 =?utf-8?B?TjlXZU1rWWlpUklvVWc0V2NQRTRkTDBwN3RpRG9DTEFjVVhEcnFmbm9yeXlO?=
 =?utf-8?B?Z1ZDb1hFbnZMVGp3OXRlQ01zanpnSjBQelNDQUE4N2VteHRNclZ6STQ3VXBW?=
 =?utf-8?B?TDVDZ3Z4ZitPMTEya21MNGFqMkRMREZlZnlEcFNpSnhtTTNCeXI5R1o1SXFF?=
 =?utf-8?B?UFhkUSthU3QxSHdURHNuTmxhMnNLTm9wVjZlcE5Geng5UklCQkQ1bFlZYUVV?=
 =?utf-8?B?WHRFRFJBN09yVERmaTdvY1dPd2ptQVVudEhucEJIeC90VGVBZVdnaE1vS1A1?=
 =?utf-8?B?a3FVU1NvTmJQWmUvdVY0TFZPVENYSFlOQzcwbFNNUjg0WmJETmhweXBJaEFi?=
 =?utf-8?B?clNsblZoZlR6cGtxWGNPNHVUdlBNbjJ2MHdOZFU4b3ZuNm11VFE2bFIzQVFL?=
 =?utf-8?B?U1RSeWtYeW9kQXg3RFJOekE4L1MxNSt6NGJod1IwU2lKNDlqMzFackl5dFI0?=
 =?utf-8?B?WHNyVlVFSk5FMStNSldrUjg5SW9XY3hYT1A5ZUM1Yis2ajdRMG1tZlFNNTN1?=
 =?utf-8?B?MjBrN1o3NHBtZTRjUXcycC9Nd01JZ2RSSXUzNzlUaklvcS9xYy9nbjVYUzFS?=
 =?utf-8?B?ZU44b0dDMlVaOFpNR1JBeEpLNHdhSG5sMU1rRzJacm9qbXRicGl1d3pEL2Mw?=
 =?utf-8?B?MFJUQWRXTXppUVZSU3dWMjVOVHFBQUNsVjYvSS80YTNLNDB4bjAwcmhUZUZk?=
 =?utf-8?B?NTFJSWxjRDJCKzdVVVdXb2hJZldNd1Yva0F2aDcyb0NTZDFsL29Nc1RWN1lM?=
 =?utf-8?B?aWoycXBFS2NZTFkwWEhhNWpxNnJBcUl5WFhwT2JnaUxFaG5JMzNpYVJTNUoy?=
 =?utf-8?Q?BLGJd26gIGXKokVUsuaCw1Ru8?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9d5a1f09-1b40-409b-350b-08dc761a36a9
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 02:36:53.9112
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UdFWcRBqrqEizvI8nk8G1QfsJYH8vmBNaWDfV6uNr4YnlsdFLEBMdw4W35dPETEAOPAMUdqH3MoRF/ZGlAkFUA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7523
X-OriginatorOrg: intel.com


On 17/05/2024 7:42 am, Isaku Yamahata wrote:
> On Thu, May 16, 2024 at 04:36:48PM +0000,
> "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> wrote:
> 
>> On Thu, 2024-05-16 at 13:04 +0000, Huang, Kai wrote:
>>> On Thu, 2024-05-16 at 02:57 +0000, Edgecombe, Rick P wrote:
>>>> On Thu, 2024-05-16 at 14:07 +1200, Huang, Kai wrote:
>>>>>
>>>>> I meant it seems we should just strip shared bit away from the GPA in
>>>>> handle_ept_violation() and pass it as 'cr2_or_gpa' here, so fault->addr
>>>>> won't have the shared bit.
>>>>>
>>>>> Do you see any problem of doing so?
>>>>
>>>> We would need to add it back in "raw_gfn" in kvm_tdp_mmu_map().
>>>
>>> I don't see any big difference?
>>>
>>> Now in this patch the raw_gfn is directly from fault->addr:
>>>
>>>          raw_gfn = gpa_to_gfn(fault->addr);
>>>
>>>          tdp_mmu_for_each_pte(iter, mmu, is_private, raw_gfn, raw_gfn+1) {
>>>                  ...
>>>          }
>>>
>>> But there's nothing wrong to get the raw_gfn from the fault->gfn.  In
>>> fact, the zapping code just does this:
>>>
>>>          /*
>>>           * start and end doesn't have GFN shared bit.  This function zaps
>>>           * a region including alias.  Adjust shared bit of [start, end) if
>>>           * the root is shared.
>>>           */
>>>          start = kvm_gfn_for_root(kvm, root, start);
>>>          end = kvm_gfn_for_root(kvm, root, end);
>>>
>>> So there's nothing wrong to just do the same thing in both functions.
>>>
>>> The point is fault->gfn has shared bit stripped away at the beginning, and
>>> AFAICT there's no useful reason to keep shared bit in fault->addr.  The
>>> entire @fault is a temporary structure on the stack during fault handling
>>> anyway.
>>
>> I would like to avoid code churn at this point if there is not a real clear
>> benefit. >>
>> One small benefit of keeping the shared bit in the fault->addr is that it is
>> sort of consistent with how that field is used in other scenarios in KVM. In
>> shadow paging it's not even the GPA. So it is simply the "fault address" and has
>> to be interpreted in different ways in the fault handler. For TDX the fault
>> address *does* include the shared bit. And the EPT needs to be faulted in at
>> that address.

It's about how we want to define the semantic of fault->addr (forget 
about shadow MMU because for it fault->addr has different meaning from TDP):

1) It represents the faulting address which points to the actual guest 
memory (has no shared bit).

2) It represents the faulting address which is truly used as the 
hardware page table walk.

The fault->gfn always represents the location of actual guest memory 
(w/o shared bit) in both cases.

I originally thought 2) isn't consistent for both SNP and TDX, but 
thinking more, I now think actually both the two semantics are 
consistent for SNP and TDX, which is important in order to avoid confusion.

Anyway it's a trivial because fault->addr is only used for fault 
handling path.

So yes fine to me we choose to use 2) here.  But IMHO we should 
explicitly add a comment to 'struct kvm_page_fault' that the @addr 
represents 2).

And I think more important thing is how we handle this "gfn" and 
"raw_gfn" in tdp_iter and 'struct kvm_mmu_page'.  See below.

>>
>> If we strip the shared bit when setting fault->addr we have to reconstruct it
>> when we do the actual shared mapping. There is no way around that. Which helper
>> does it, isn't important I think. Doing the reconstruction inside
>> tdp_mmu_for_each_pte() could be neat, except that it doesn't know about the
>> shared bit position.
>>
>> The zapping code's use of kvm_gfn_for_root() is different because the gfn comes
>> without the shared bit. It's not stripped and then added back. Those are
>> operations that target GFNs really.
>>
>> I think the real problem is that we are gleaning whether the fault is to private
>> or shared memory from different things. Sometimes from fault->is_private,
>> sometimes the presence of the shared bits, and sometimes the role bit. I think
>> this is confusing, doubly so because we are using some of these things to infer
>> unrelated things (mirrored vs private).
> 
> It's confusing we don't check it in uniform way.
> 
> 
>> My guess is that you have noticed this and somehow zeroed in on the shared_mask.
>> I think we should straighten out the mirrored/private semantics and see what the
>> results look like. How does that sound to you?
> 
> I had closer look of the related code.  I think we can (mostly) uniformly use
> gpa/gfn without shared mask.  Here is the proposal.  We need a real patch to see
> how the outcome looks like anyway.  I think this is like what Kai is thinking
> about.
> 
> 
> - rename role.is_private => role.is_mirrored_pt
> 
> - sp->gfn: gfn without shared bit.

I think we can treat 'tdp_iter' and 'struct kvm_mmu_page' in the same 
way, because conceptually they both reflects the page table.

So I think both of them can have "gfn" or "raw_gfn", and "shared_gfn_mask".

Or have both "raw_gfn" or "gfn" but w/o "shared_gfn_mask". This may be 
more straightforward because we can just use them when needed w/o 
needing to play with gfn_shared_mask.

> 
> - fault->address: without gfn_shared_mask
>    Actually it doesn't matter much.  We can use gpa with gfn_shared_mask.

See above.  I think it makes sense to include the shared bit here.  It's 
trivial anyway though.

> 
> - Update struct tdp_iter
>    struct tdp_iter
>      gfn: gfn without shared bit

Or "raw_gfn"?

Which may be more straightforward because it can be just from:

	raw_gfn = gpa_to_gfn(fault->addr);
	gfn = fault->gfn;

	tdp_mmu_for_each_pte(..., raw_gfn, raw_gfn + 1, gfn)

Which is the reason to make fault->addr include the shared bit AFAICT.

> 
>      /* Add new members */
> 
>      /* Indicates which PT to walk. */
>      bool mirrored_pt;

I don't think you need this?  It's only used to select the root for page 
table walk.  Once it's done, we already have the @sptep to operate on.

And I think you can just get @mirrored_pt from the sptep:

	mirrored_pt = sptep_to_sp(sptep)->role.mirrored_pt;

Instead, I think we should keep the @is_private to indicate whether the 
GFN is private or not, which should be distinguished with 'mirrored_pt', 
which the root page table (and the @sptep) already reflects.

Of course if the @root/@sptep is mirrored_pt, the is_private should be 
always true, like:

	WARN_ON_ONCE(sptep_to_sp(sptep)->role.is_mirrored_pt
			&& !is_private);

Am I missing anything?

> 
>      // This is used tdp_iter_refresh_sptep()
>      // shared gfn_mask if mirrored_pt
>      // 0 if !mirrored_pt
>      gfn_shared_mask >
> - Pass mirrored_pt and gfn_shared_mask to
>    tdp_iter_start(..., mirrored_pt, gfn_shared_mask)

As mentioned above, I am not sure whether we need @mirrored_pt, because 
it already have the @root.  Instead we should pass is_private, which 
indicates the GFN is private.

> 
>    and update tdp_iter_refresh_sptep()
>    static void tdp_iter_refresh_sptep(struct tdp_iter *iter)
>          ...
>          iter->sptep = iter->pt_path[iter->level - 1] +
>                  SPTE_INDEX((iter->gfn << PAGE_SHIFT) | iter->gfn_shared_mask, iter->level);
> 
>    Change for_each_tdp_mte_min_level() accordingly.
>    Also the iteretor to call this.
>     
>    #define for_each_tdp_pte_min_level(kvm, iter, root, min_level, start, end)      \
>            for (tdp_iter_start(&iter, root, min_level, start,                      \
>                 mirrored_root, mirrored_root ? kvm_gfn_shared_mask(kvm) : 0);      \
>                 iter.valid && iter.gfn < kvm_gfn_for_root(kvm, root, end);         \
>                 tdp_iter_next(&iter))

See above.

> 
> - trace point: update to include mirroredd_pt. Or Leave it as is for now.
> 
> - pr_err() that log gfn in handle_changed_spte()
>    Update to include mirrored_pt. Or Leave it as is for now.
> 
> - Update spte handler (handle_changed_spte(), handle_removed_pt()...),
>    use iter->mirror_pt or pass down mirror_pt.
> 

IIUC only sp->role.is_mirrored_pt is needed, tdp_iter->is_mirrored_pt 
isn't necessary.  But when the @sp is created, we need to initialize 
whether it is mirrored_pt.

Am I missing anything?