Received-SPF: pass (google.com: domain of linux-kernel+bounces-180561-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <a08779dc-056c-421c-a573-f0b1ba9da8ad@intel.com>
Date: Thu, 16 May 2024 14:07:36 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 10/16] KVM: x86/tdp_mmu: Support TDX private mapping for
 TDP MMU
To: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>, "kvm@vger.kernel.org"
	<kvm@vger.kernel.org>, "pbonzini@redhat.com" <pbonzini@redhat.com>,
	"seanjc@google.com" <seanjc@google.com>
CC: "sagis@google.com" <sagis@google.com>, "dmatlack@google.com"
	<dmatlack@google.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "isaku.yamahata@gmail.com"
	<isaku.yamahata@gmail.com>, "Zhao, Yan Y" <yan.y.zhao@intel.com>, "Aktas,
 Erdem" <erdemaktas@google.com>
References: <20240515005952.3410568-1-rick.p.edgecombe@intel.com>
 <20240515005952.3410568-11-rick.p.edgecombe@intel.com>
 <12afae41-906c-4bb7-956a-d73734c68010@intel.com>
 <1d247b658f3e9b14cefcfcf7bca01a652d0845a0.camel@intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <1d247b658f3e9b14cefcfcf7bca01a652d0845a0.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UTIrcjU2Tk9mSDExNnNmbFVCR2g1d3oyWFdpZCtjVThnZDd3eWJSK2JSekVx?=
 =?utf-8?B?TlpWenl2czBOSXNLclNGTEkveUMrUk5oVWJiMFFtRStWS3ZoNlBLbEx6WXVX?=
 =?utf-8?B?U2RkRXpKNEZ0SkZGMHdDeVY3d1lsV2g1TnFUaFRrTXZ6ZWRJeWp1KzdrREQ1?=
 =?utf-8?B?OGVkb1FmRFd4Sk1maS8wQW9naTZGN2lmZTVOZ1JmalY1V21ZT09meUpaRTly?=
 =?utf-8?B?dnd6djZCL29vOTFWb1A1enhuZTg1Y1pjMlRsWGh5VnFLelYxRjlvZENRdEdM?=
 =?utf-8?B?S1MzZFNaYTk5aU9wZXVoZXVxdXRObHhUSTROcCtNeVZMbExXYzYyb1ZZbTZt?=
 =?utf-8?B?Z0lFY0J3OUpYTmdrTHczWVhRNEZFRDRSU0F3NXJ0Mjk0dUY5WkFyK2sxY3FY?=
 =?utf-8?B?aW5ON2FRbUYzaWlhNVZidGdnUFdSenVVTnNlaXFSRTRpSURUME45dW5zQnNm?=
 =?utf-8?B?MitwNjE5TzNtZ21jdEJYdmp1akp5allpQTRZcnJhSWlhak5IdW5wQ3pIZWJV?=
 =?utf-8?B?QVh4eHB3ZDdjVTU4cmc2cUhhQjZZN3FZQjB1RWpwZFhGYzhYdkpZbm1oL2FU?=
 =?utf-8?B?Ry9YWXpackd6Ti9BU2dtMFhHWW45cFk2dTVGVmFsQVpsNERIdGRlZzc3YmR4?=
 =?utf-8?B?bWNCNEc0MDM1b0ZyM1FSMGh6Nk1MOWVOSDJEd05Eckx6QlhQS1V3eDVvT2tW?=
 =?utf-8?B?UDBsc1JhUEFBNUV1V28ydWFZYnhCL0tPbW9kK25WWndlTnNiVjlnQWxGcDho?=
 =?utf-8?B?NWNVWnc3MUZKTHgwajZrL1lwaFNwYzN0M2FYUGpueEk0Q08yWkhLWng5S3d1?=
 =?utf-8?B?WmRzZE9NS2VmMytnZXY5aEVmZE9YMmlyalcyN1RLZjF0eXlvMnhDK0duWTJX?=
 =?utf-8?B?RVY0Q0syZkZIcTB1Q1VmTjJvU2ZxMnJCL0I3MlJ2bnlZbXJvUWtOdGRuKzFw?=
 =?utf-8?B?RHBqK3J5UjVWemQxUUd1TmJvVE45R3RiT0RmSHhySnpIcjFjWHRCckZpSXFm?=
 =?utf-8?B?UUIxNlZpVWdBS1oyeG5ITFZrOU51SWJ6R212TlB0ck5jWFRUdjVBMHQ3U1V4?=
 =?utf-8?B?ZnlST0QrQVVEK0xQTlhBT3E5S2dPTGZiWVlla3VvT2FaV3hqUkNyT2I4b3hq?=
 =?utf-8?B?Z3hkZkw4a1d3Y3VVNFBHK21HQUhwL2FqckY3SytmNkhSZEJFZ0F4KzkzaDRm?=
 =?utf-8?B?dXA5T3g3Y1U5YVh5Ujd2ZnUwL2tuUjhQTjBlZ1pnZE1pbC85Y2tGS0UzdnJ0?=
 =?utf-8?B?b1JMS0gzOHlPZjdGU2FRUFN1NEFwbW1NZmVFUjNhc1R6UGNZTGtVWDFFbm9K?=
 =?utf-8?B?bDRrN1RWa2tRK3FUWUhFRVZGUE45UjNXRE50emhiZ3Mwc2x2N3VuTEQvcnlX?=
 =?utf-8?B?YmVYZGFVTk5RWHpxa1VyVi9hcFRFb1R6bVJOVldxeWIzUnJLQkRnUkNYd0xH?=
 =?utf-8?B?RzlnSVBoYVJjVlowUFY0WEFWem91YWI5WGdRaVNTS1h1b01Bd2FEYlFIdTFm?=
 =?utf-8?B?Mk5HaXVKN2F2S1JmNWVvQlNEeEJsNGFQN01UcTRwZStuYjVVUnlMRTExeE1G?=
 =?utf-8?B?UWlPZjNuR3NDNUlWV3ZjOWtGbkpSUDlZdUJ1K2JLQXA2bGhFTXl4UlFYbTcz?=
 =?utf-8?B?b2xQR1Mya011TEh5MmZJTlBsTjFYUnEwcDcyS0VqdHpKbDRMSlpVM05mcGYv?=
 =?utf-8?B?Z3cydWhoYmtvUG1qZCtQeUFjWFFETXBmblZRMDBOL0NFYXc4d1czbnJJUjBT?=
 =?utf-8?B?cXZaZU5KRDZ2aGRLK2orbGNLMUpUdUdvbGhPZEp0dFJNajV2d2Fjc3pTZ21z?=
 =?utf-8?B?THBLMVVSK0h6T0Uyc3BHRG1hVUl1MWJlS05DdFF0dUcwWkdJQTlFRkRIRjhP?=
 =?utf-8?B?M0pkdlNHNjZXelVpbzV4YmtDQXhrSGJIeFZoVU5nNG9LMElnRis3TlNRWFZW?=
 =?utf-8?B?Q2pFSWhRT2VURmFTQWlyM3BIeUtSc0t2a3pEUmdBOGxMUzdpWUpobGV6NklI?=
 =?utf-8?B?TFdzY3lZZmFPdmUvUDRKcnJGSExmLzVocDMzdHZkRWlWaGpQZ3dvVGhtS2tu?=
 =?utf-8?B?WTBadFFtUzJRVGlnelE1TmRvNkJXa2xGWFdhSnFaRkw3UDQyT01zR3hvdkJU?=
 =?utf-8?Q?1y9g5y0HOXEsPuPmR+xhCnlkO?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 31e5b551-1887-4e91-fd22-08dc754cf986
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2024 02:07:44.6239
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZeP3VVSfjD005H3yYxIP3f/4IFQVk+Xn33E3fE3czKaTpk9D4Z+4Gek9Vx0cM7CmJwBCw2VGrntgubPAFJ7n4g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB7039
X-OriginatorOrg: intel.com


>>> @@ -470,6 +470,7 @@ struct kvm_mmu {
>>>          int (*sync_spte)(struct kvm_vcpu *vcpu,
>>>                           struct kvm_mmu_page *sp, int i);
>>>          struct kvm_mmu_root_info root;
>>> +       hpa_t private_root_hpa;
>>
>> Should we have
>>
>>          struct kvm_mmu_root_info private_root;
>>
>> instead?
> 
> This is corresponds to:
> mmu->root.hpa
> 
> We don't need the other fields, so I think better to not take space. It does
> look asymmetric though...

Being symmetric is why I asked.  Anyway no strong opinion.

[...]

>>>    
>>> @@ -4685,7 +4687,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct
>>> kvm_page_fault *fault)
>>>          if (kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) {
>>>                  for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level)
>>> {
>>>                          int page_num = KVM_PAGES_PER_HPAGE(fault-
>>>> max_level);
>>> -                       gfn_t base = gfn_round_for_level(fault->gfn,
>>> +                       gfn_t base = gfn_round_for_level(gpa_to_gfn(fault-
>>>> addr),
>>>                                                           fault->max_level);
>>
>> I thought by reaching here the shared bit has already been stripped away
>> by the caller?
> 
> We don't support MTRRs so this code wont be executed for TDX, but not clear what
> you are asking.
> fault->addr has the shared bit (if present)
> fault->gfn has it stripped.

When I was looking at the code, I thought fault->gfn is still having the 
shred bit, and gpa_to_gfn() internally strips aways the shared bit, but 
sorry it is not true.

My question is why do we even need this change?  Souldn't we pass the 
actual GFN (which doesn't have the shared bit) to 
kvm_mtrr_check_gfn_range_consistency()?

If so, looks we should use fault->gfn to get the base?

> 
>>
>> It doesn't make a lot sense to still have it here, given we have a
>> universal KVM-defined PFERR_PRIVATE_ACCESS flag:
>>
>> https://lore.kernel.org/kvm/20240507155817.3951344-2-pbonzini@redhat.com/T/#mb30987f31b431771b42dfa64dcaa2efbc10ada5e
>>
>> IMHO we should just strip the shared bit in the TDX variant of
>> handle_ept_violation(), and pass the PFERR_PRIVATE_ACCESS (when GPA
>> doesn't hvae shared bit) to the common fault handler so it can correctly
>> set fault->is_private to true.
> 
> I'm not sure what you are seeing here, could elaborate?
See reply below.

[...]

>>
>> Anyway, from common code's perspective, we need to have some
>> clarification why we design to do it here.
>>
>>>          free_mmu_pages(&vcpu->arch.root_mmu);
>>>          free_mmu_pages(&vcpu->arch.guest_mmu);
>>>          mmu_free_memory_caches(vcpu);
>>> diff --git a/arch/x86/kvm/mmu/mmu_internal.h
>>> b/arch/x86/kvm/mmu/mmu_internal.h
>>> index 0f1a9d733d9e..3a7fe9261e23 100644
>>> --- a/arch/x86/kvm/mmu/mmu_internal.h
>>> +++ b/arch/x86/kvm/mmu/mmu_internal.h
>>> @@ -6,6 +6,8 @@
>>>    #include <linux/kvm_host.h>
>>>    #include <asm/kvm_host.h>
>>>    
>>> +#include "mmu.h"
>>> +
>>>    #ifdef CONFIG_KVM_PROVE_MMU
>>>    #define KVM_MMU_WARN_ON(x) WARN_ON_ONCE(x)
>>>    #else
>>> @@ -178,6 +180,16 @@ static inline void kvm_mmu_alloc_private_spt(struct
>>> kvm_vcpu *vcpu, struct kvm_m
>>>          sp->private_spt = kvm_mmu_memory_cache_alloc(&vcpu-
>>>> arch.mmu_private_spt_cache);
>>>    }
>>>    
>>> +static inline gfn_t kvm_gfn_for_root(struct kvm *kvm, struct kvm_mmu_page
>>> *root,
>>> +                                    gfn_t gfn)
>>> +{
>>> +       gfn_t gfn_for_root = kvm_gfn_to_private(kvm, gfn);
>>> +
>>> +       /* Set shared bit if not private */
>>> +       gfn_for_root |= -(gfn_t)!is_private_sp(root) &
>>> kvm_gfn_shared_mask(kvm);
>>> +       return gfn_for_root;
>>> +}
>>> +
>>>    static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page
>>> *sp)
>>>    {
>>>          /*
>>> @@ -348,7 +360,12 @@ static inline int __kvm_mmu_do_page_fault(struct
>>> kvm_vcpu *vcpu, gpa_t cr2_or_gp
>>>          int r;
>>>    
>>>          if (vcpu->arch.mmu->root_role.direct) {
>>> -               fault.gfn = fault.addr >> PAGE_SHIFT;
>>> +               /*
>>> +                * Things like memslots don't understand the concept of a
>>> shared
>>> +                * bit. Strip it so that the GFN can be used like normal,
>>> and the
>>> +                * fault.addr can be used when the shared bit is needed.
>>> +                */
>>> +               fault.gfn = gpa_to_gfn(fault.addr) &
>>> ~kvm_gfn_shared_mask(vcpu->kvm);
>>>                  fault.slot = kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn);
>>
>> Again, I don't think it's nessary for fault.gfn to still have the shared
>> bit here?
> 
> It's getting stripped as it's set for the first time... What do you mean still
> have it?

Sorry, I meant fault->addr.

> 
>>
>> This kinda usage is pretty much the reason I want to get rid of
>> kvm_gfn_shared_mask().
> 
> I think you want to move it to an x86_op right? Not get rid of the concept of a
> shared bit? I think KVM will have a hard time doing TDX without knowing about
> the shared bit location.
> 
> Or maybe you are saying you think it should be stripped earlier and live as a PF
> error code?

I meant it seems we should just strip shared bit away from the GPA in 
handle_ept_violation() and pass it as 'cr2_or_gpa' here, so fault->addr 
won't have the shared bit.

Do you see any problem of doing so?

> 
>>
>>>          }
>>>    
>>> diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h
>>> index fae559559a80..8a64bcef9deb 100644
>>> --- a/arch/x86/kvm/mmu/tdp_iter.h
>>> +++ b/arch/x86/kvm/mmu/tdp_iter.h
>>> @@ -91,7 +91,7 @@ struct tdp_iter {
>>>          tdp_ptep_t pt_path[PT64_ROOT_MAX_LEVEL];
>>>          /* A pointer to the current SPTE */
>>>          tdp_ptep_t sptep;
>>> -       /* The lowest GFN mapped by the current SPTE */
>>> +       /* The lowest GFN (shared bits included) mapped by the current SPTE
>>> */
>>>          gfn_t gfn;
>>
>> IMHO we need more clarification of this design.
> 
> Have you seen the documentation patch? Where do you think it should be? You mean
> in the tdp_iter struct?

My thinking:

Changelog should clarify why include shared bit to 'gfn' in tdp_iter.

And here around the 'gfn' we can have some simple sentence to explain 
why to include the shared bit.