Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <41d6ef7c-f184-54b3-d0cf-81a111c2192f@intel.com>
Date:   Mon, 24 Jul 2023 21:50:20 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [PATCH v4 10/20] KVM:x86: Make guest supervisor states as
 non-XSAVE managed
Content-Language: en-US
To:     Chao Gao <chao.gao@intel.com>
CC:     <seanjc@google.com>, <pbonzini@redhat.com>, <peterz@infradead.org>,
        <john.allen@amd.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <rick.p.edgecombe@intel.com>,
        <binbin.wu@linux.intel.com>
References: <20230721030352.72414-1-weijiang.yang@intel.com>
 <20230721030352.72414-11-weijiang.yang@intel.com>
 <ZL410xRbInlQMc5y@chao-email>
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <ZL410xRbInlQMc5y@chao-email>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Z3htQ2pPdG53d09CLzl6OVVmZXB4UXdRV3p0MWhYN3NLMHRMLzgydExCeUNj?=
 =?utf-8?B?Rmx4eG8yQ1dRRDFVLzJIbEduZ3ExS3hLNzdqWkJmODVoUElVWWUrVVV1N2Vs?=
 =?utf-8?B?eGwrZy8vM2pjdE12eXNqZWk2cTl5ZHd3UzRMVmxCa2l4SkJQOVgyN1BLa3JL?=
 =?utf-8?B?djZaaExPcDJVdHlGZ053dFNOTFZ6ZzVTU2RoViswYkt6VlFNd1lyZ1Rka3Z1?=
 =?utf-8?B?MDhZbmx0cU11RWdTa091RWQ3V20xcDRJQmpHckh2R2t0SWFad3NDbThqcEV2?=
 =?utf-8?B?bWkyd0YyMjArbWE4NVZVZTZSMURKWFRSTGEzQ0tGaHgxTXBmaGo1QXJCZXdJ?=
 =?utf-8?B?YmVJTGtPOS9JdVorK0J6MWFTMEU1Z2ExTFg2TGlFOWtRWUpOVGJteG9jazJB?=
 =?utf-8?B?ZEE1SEFjYU9oYkEwOFRQRUFLQUVaNkRVdFB6eUFIdEtjRDlNZm1ST2NSa0JG?=
 =?utf-8?B?alFoMXd0bXNmeExqYWl4OWVTQmJFQnlJMWJTZGVSdmlqdjJ5NGREQ3FOcDFQ?=
 =?utf-8?B?NmErdHdxRHhIaWlNaVd4SDRhcVJSREdMMTdqN2dZMUhENXRWenBTcldrZHFh?=
 =?utf-8?B?REFRSTJ5Ri9ubTBZSHM3QVBOeE4waCtnSi9aaGI5eVg4ajNZVjdjendqU2xN?=
 =?utf-8?B?RUcvaGRncXNxazliU2RxU25hbjFTMjhORkl0anFTWTFhRUFVVy9MRzl0a0ZE?=
 =?utf-8?B?a2VhNWhBK1E2OG9MbzNHK1lIdnpzTHdhdlVhWW1lRDZEN1Q1azVueWxpc1c5?=
 =?utf-8?B?WXlXaGdUbnBGcUFPRzE0ZmZEV25RYVJDYUdkd1N3WDVFYldIbkhUcFNtR0dz?=
 =?utf-8?B?a1hHdE9VMzdRNFczYkNvdUs1SC9US0NLbEowZEtxRFRJM1VlNDAvbi9rei9P?=
 =?utf-8?B?KzJSazJOcEtrQjh4SzdLZnBkUDZScnB5RFM3Y04vWkJncHVCNUowQnNKVFZs?=
 =?utf-8?B?SXRvbUJ2WjNtTzZzM0Roc09rRVhrOFdtMStHeWEycVZjNGFyNEQwWEU4ZGxp?=
 =?utf-8?B?WTk0Vno4dUpzWDFQOXBFL293UEtYTjN1cWdDRjZnYm9ETGxkSUIwRjN6c2Yv?=
 =?utf-8?B?ekZtRUcxYlJUcGdrd01sM0E4aHZyUUFsS0ZPV3REd3NVczFNZkg4MmFDb2o2?=
 =?utf-8?B?bStnN3UrVlJBTS9NKzQ3RmVwOU85bzBOOE14OVFBQTFDUEk4MVFjZlY4QTBM?=
 =?utf-8?B?OHhML25FeVdEUnAzMlRxNHRVN21aUWhLeVpING1XbHRZSG5oQzhPaEtiODZT?=
 =?utf-8?B?SFk2dzFvck52L3B4WDdkYTZ6bnN6Vk11N0Zzd29WaVdJSXU3MFk5YnNmV3Bj?=
 =?utf-8?B?N3lwTDJMSHM3b1pJUkF0aVRJWjUxV09VbG1BdnF5R3NuUmJBNjdDL1F4K1lS?=
 =?utf-8?B?d0V4UVF2Ykx4OU9PL1NWV3QzMHQyNU5xMnA2MVhyYmpxM25oeTBsWFQxSjBR?=
 =?utf-8?B?UHJjNkxFRlljb3k0Skl5ZEhJdi9hcnVkR29zRVE5U3lQaEc5SEw5UGJoOE8x?=
 =?utf-8?B?eGJkQ01CV3ZXZWx4YmFqR2hNZWhhWkloQ0NVVDhSWEh4aDBYaDJWNGZETzBh?=
 =?utf-8?B?SDF1dTAwTTRPYWJVQUZsL0VUMHRwdHB5NVFqY0hFTFNYSVpMaDMwa3FBUHBn?=
 =?utf-8?B?NDlxR1ZjbGlEUE13Nk1Kd3NycFR3RGdqZ255ZEJxdnVXQi9sbTFIUVdCSm5K?=
 =?utf-8?B?WklMVE1oNVllWmE1SG10U1Q5cWJVSDZoV1BLakduQU4ycUxEWHhldXFQSkdB?=
 =?utf-8?B?SjU4eE1VUzE0eDh2ZUtYQldYMElVKzZPOVVsM0tJVFBDZWN6R3ZDSHFHZVF6?=
 =?utf-8?B?N1Y2ekJsZ2dVSkpodjk3THNoOHRlWnhmc21OcEpUSG95UVc2bDVtbXZHSFMr?=
 =?utf-8?B?c0NBWmFOWUE4ZWE4aFJHWGx0bkhKQ1ZMZGsvSXhhT0VYK3BNTEJ2cnJmQmVz?=
 =?utf-8?B?YVd5UURHbXFOb0xKRXRic25sZmxaYzdoejRRbnFrYkRIV0ZSWGZXdVpWVTVk?=
 =?utf-8?B?TkdGZEM3TzVqbnZ5cVpYRGlBR3FJOHdLeU5xbGtnZWNNVnpTc3FTNS9wdWd3?=
 =?utf-8?B?aVkzRm5qNVY3QnZOSkRVem5DOGUxbWZBbHlwdStjQVdoSGlWeGVBWHJ4Y3Nl?=
 =?utf-8?B?dXBRK2FwajF6dDMrQWYvWEpQS3FXcXY4WXhvQzRuRE9DaHJHd0hUeE0vM2d2?=
 =?utf-8?B?L3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a68147b3-f592-46e6-4e56-08db8c4cf253
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2023 13:50:31.7555
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YTGgMx8+1D83oBs3VHN+VwJampp1EhPZIBaAlgKqJRD6tM+SDJ3GYBjUPiSL7acNGH7RN7lvJ49Q3s3GKWTX0w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB5044
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 7/24/2023 4:26 PM, Chao Gao wrote:
> On Thu, Jul 20, 2023 at 11:03:42PM -0400, Yang Weijiang wrote:
>> +static void kvm_save_cet_supervisor_ssp(struct kvm_vcpu *vcpu)
>> +{
>> +	preempt_disable();
> what's the purpose of disabling preemption?

Thanks!

These preempt_disable/enable() becomes unnecessary due to the PLx_SSP 
handling

in sched_in/out(). Will remove them.

>
>> +	if (unlikely(guest_can_use(vcpu, X86_FEATURE_SHSTK))) {
>> +		rdmsrl(MSR_IA32_PL0_SSP, vcpu->arch.cet_s_ssp[0]);
>> +		rdmsrl(MSR_IA32_PL1_SSP, vcpu->arch.cet_s_ssp[1]);
>> +		rdmsrl(MSR_IA32_PL2_SSP, vcpu->arch.cet_s_ssp[2]);
>> +		/*
>> +		 * Omit reset to host PL{1,2}_SSP because Linux will never use
>> +		 * these MSRs.
>> +		 */
>> +		wrmsrl(MSR_IA32_PL0_SSP, 0);
> You don't need to reset the MSR because current host doesn't enable SSS
> and leaving guest value in the MSR won't affect host behavior.

Yes,  I just want to make the host PLx_SSPs as clean as possible.

>
>> +	}
>> +	preempt_enable();
>> +}
>> +
>> +static void kvm_reload_cet_supervisor_ssp(struct kvm_vcpu *vcpu)
>> +{
>> +	preempt_disable();
>> +	if (unlikely(guest_can_use(vcpu, X86_FEATURE_SHSTK))) {
>> +		wrmsrl(MSR_IA32_PL0_SSP, vcpu->arch.cet_s_ssp[0]);
>> +		wrmsrl(MSR_IA32_PL1_SSP, vcpu->arch.cet_s_ssp[1]);
>> +		wrmsrl(MSR_IA32_PL2_SSP, vcpu->arch.cet_s_ssp[2]);
>> +	}
>> +	preempt_enable();
>> +}
> save/load PLx_SSP in kvm_sched_in/out() and in VCPU_RUN ioctl is sub-optimal.
>
> How about:
> 1. expose kvm_save/reload_cet_supervisor_ssp()
> 2. reload guest PLx_SSP in {vmx,svm}_prepare_switch_to_guest()
> 3. save guest PLx_SSP in vmx_prepare_switch_to_host() and
>     svm_prepare_host_switch()?
>
> this way existing svm/vmx->guest_state_loaded can help to reduce a lot of
> unnecessary PLx_SSP MSR accesses.

Nice suggestion! It looks workable. I'll try this,  thanks!

>
>> +
>> int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
>> {
>> 	struct kvm_queued_exception *ex = &vcpu->arch.exception;
>> @@ -11222,6 +11249,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
>> 	kvm_sigset_activate(vcpu);
>> 	kvm_run->flags = 0;
>> 	kvm_load_guest_fpu(vcpu);
>> +	kvm_reload_cet_supervisor_ssp(vcpu);
>>
>> 	kvm_vcpu_srcu_read_lock(vcpu);
>> 	if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) {
>> @@ -11310,6 +11338,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
>> 	r = vcpu_run(vcpu);
>>
>> out:
>> +	kvm_save_cet_supervisor_ssp(vcpu);
>> 	kvm_put_guest_fpu(vcpu);
>> 	if (kvm_run->kvm_valid_regs)
>> 		store_regs(vcpu);
>> @@ -12398,9 +12427,17 @@ void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu)
>> 		pmu->need_cleanup = true;
>> 		kvm_make_request(KVM_REQ_PMU, vcpu);
>> 	}
>> +
>> +	kvm_reload_cet_supervisor_ssp(vcpu);
>> +
>> 	static_call(kvm_x86_sched_in)(vcpu, cpu);
>> }
>>
>> +void kvm_arch_sched_out(struct kvm_vcpu *vcpu, int cpu)
>> +{
> @cpu its meaning isn't clear and isn't used and ...
Yes, I should have removed it.
>
>> +	kvm_save_cet_supervisor_ssp(vcpu);
>> +}
>> +
>> void kvm_arch_free_vm(struct kvm *kvm)
>> {
>> 	kfree(to_kvm_hv(kvm)->hv_pa_pg);
>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>> index d90331f16db1..b3032a5f0641 100644
>> --- a/include/linux/kvm_host.h
>> +++ b/include/linux/kvm_host.h
>> @@ -1423,6 +1423,7 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
>> int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu);
>>
>> void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu);
>> +void kvm_arch_sched_out(struct kvm_vcpu *vcpu, int cpu);
>>
>> void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
>> void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu);
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 66c1447d3c7f..42f28e8905e1 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -5885,6 +5885,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,
>> {
>> 	struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn);
>>
>> +	kvm_arch_sched_out(vcpu, 0);
> passing 0 always looks problematic.
Can you elaborate? I have no intent to use @cpu now.
>> 	if (current->on_rq) {
>> 		WRITE_ONCE(vcpu->preempted, true);
>> 		WRITE_ONCE(vcpu->ready, true);
>> -- 
>> 2.27.0
>>