Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <9307c734-3473-0bdc-57be-c39e96bca4d8@amd.com>
Date:   Mon, 2 May 2022 21:07:44 +0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.8.1
Subject: Re: [PATCH v2 08/12] KVM: SVM: Update AVIC settings when changing
 APIC mode
Content-Language: en-US
To:     Maxim Levitsky <mlevitsk@redhat.com>, linux-kernel@vger.kernel.org,
        kvm@vger.kernel.org
Cc:     pbonzini@redhat.com, seanjc@google.com, joro@8bytes.org,
        jon.grimm@amd.com, wei.huang2@amd.com, terry.bowman@amd.com
References: <20220412115822.14351-1-suravee.suthikulpanit@amd.com>
 <20220412115822.14351-9-suravee.suthikulpanit@amd.com>
 <abb93e2d73b7ada6cbabcd3ebbf7b38e4701ec57.camel@redhat.com>
From:   Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
In-Reply-To: <abb93e2d73b7ada6cbabcd3ebbf7b38e4701ec57.camel@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YVFhV3FyKzBkWmZ3YWZtNUFvWXhoVzQrak80ZUdFTE9sN0haZDdkMHBSWlla?=
 =?utf-8?B?alNOU2N5N2ZMT0pHdWJuMk55dkI3ZUkzZ3huNEN4TzRoZ05KMzd1Vm5vNWdY?=
 =?utf-8?B?NjVIeEdrNzZGVFVLVDJNR21mNDkzbmhOc2xHa1A5OU9FWkJoYnJmV0R0cnEx?=
 =?utf-8?B?cEpWdnNHR2czNTB5QjhFVDlLMkYzT2YrclZqRDl4VHdPbHVNS3VBVnFuR3NB?=
 =?utf-8?B?Vkp0K0RBdzlRMytUYW1MV0QveFpZNjZ3bTlTY3MxWlhvZVVna2dXVU5DLzM5?=
 =?utf-8?B?Znl1UDBMT0JEOFFxZU5FaHNCOXdVZ3dzRDlhV25oRlJuZ3N3bW5PNkRhTzRY?=
 =?utf-8?B?d2VZMllOSkt2V1dhT1h1RWN0eXNNWTU1YnFPSFZvejltemZweUxUTGRVVnhj?=
 =?utf-8?B?dWE2NWhBSjVXeEIxTjZHVnJVMzg5dHhCcVh1cmlNeXRUSnlUaklsU0VXbXM2?=
 =?utf-8?B?RGdtTHR5cXdxeVF4SnBiakVETHpTbTVFWWZBeVdDNjh1bTBWTFAxZ2RzdWFU?=
 =?utf-8?B?MitVR3lXSndmcGNDOC94VnZ3TWJDaXhzZHZEbVk0eWFkYzdHcFcreGNoQzd1?=
 =?utf-8?B?VzIrd0pWc3NiQ21nQktoWW5KSUVDb0JNV0JCUW9ZVURJQmpLU3F6STFuMjFa?=
 =?utf-8?B?NjlmUERuNU80ck5jNDl2MEdZRitHQllkSTI3b2RMY3hnZVZZd0h3ZHppRTF6?=
 =?utf-8?B?V21PNWxLSVBGTkQ4SERSbVVBQ2YzaXFaWG54eGg3R3h5Y1dqM0F2RkxIeHI1?=
 =?utf-8?B?c3BpSlB0UHQwS0NiakpZVzVUSW93ZFJIWXlacEVURWNZeUk4ODE2dzFrZFpT?=
 =?utf-8?B?TmR2NGVRUW1ycjMzRnFoYytrMERvUjUwK1llVk1ISGUxbEVuWUdUeTkxVXc4?=
 =?utf-8?B?bjdhQVZjZnRXU0VEN04rcmJnQWRReVJ6eHdic21ISUpsbVE4MnBvOUtIdTRj?=
 =?utf-8?B?WlV6U3V2M0k0ZVVCOVo3VGo0L3d2MmlDR2s0ZkhRRU91UnZDWTg4cE1Rekls?=
 =?utf-8?B?Tk1BVGZKcjdkVGRsTU1aanBEaFMrNTEyVHJkR1dZcHRRY1B0TFljRWU5QjdY?=
 =?utf-8?B?TUxTYUZQL3IzYmdRcTdza2VlODBvbzRZTkRRUUN0emVFTWVOOUNFTTlJekMx?=
 =?utf-8?B?U016djhSTnRNaGY3S0ZHV09ZNDFTTWNEaStoYVZ3TC9MbTRnbGdFL3dHbUVE?=
 =?utf-8?B?Qi9JNGQvOUorZ0xnbEhPRy9UOFpPZ2V6bmZRSm5CT3Z5UFRQUzZlMXRDVmo4?=
 =?utf-8?B?V3RlajNKZDN6YzVhdFdRbWpvNjRtazh6d2M5QzZXYlh5NkU4bGJCbnZVQWZ4?=
 =?utf-8?B?QXhlRU5PRmJVeFNZN0dtZTF0Z2J0NTBuTW01TzBSV0FZUEpHYlpvdmhPbS9T?=
 =?utf-8?B?bEo1a1VMemRTaUxzc0Uwam00N2pvQ2owSTdQaUZnaTNyb1I3Q1NvNlpQajdw?=
 =?utf-8?B?WmplOW1UNjEvT1JHTXJOUlFibWIwMWNOdE1kM0hyOUVKVllJMFg5bkhVa3Ra?=
 =?utf-8?B?R3JiUG1QSWh3aXNxMi9VZ08xRFBwMTR3SFFINmZ6VE1TcWZmUjhUaEdNQ3p4?=
 =?utf-8?B?NWFNZ010NUZHcDE0MVBDeEtsQzZFWCtqWEduazdyNHF4M1NEaDMxczd0Rmdy?=
 =?utf-8?B?ZVJ5bTZIUWJGYVhoa3hSY24zMElNblowdUxjdkdieVBpVllxL2s2cGRHaXZr?=
 =?utf-8?B?TG5kTkRHMXlBc1lxRkxiU0crNytQb1Q4eDAwUzcwMTU3MUFDZlVVd0RGWXda?=
 =?utf-8?B?U2xlMVdGZnpWRHk3cVF1TUFsVU5SV3VzQ1NaVnM4MHZ2RGxiZStxQUp5U0Vj?=
 =?utf-8?B?SW05blNCWHNybWNnaHA1MUtVMlZmNWNHdUFtZFlIaE5IZWZmblkxL0x0Ynlm?=
 =?utf-8?B?TUdwVGxSZDlTUit1R0FYZ3FGRzlXcHJHRG5Kdy92NkRSMkM1UkZlZEtJa1d6?=
 =?utf-8?B?aGMrTW82Z2FWa3htam40MEtqYkd0bmE3SXF3Mjl0dW5aUkVsMlRoaUorTjlL?=
 =?utf-8?B?N3kvMnRHakJKUTVGbE96Z05WUFR6c2g5SWdtTUYxdUVoazVTYjJrVEtiaEdM?=
 =?utf-8?B?WE9vWEtQeE5IMlN6QitXaGdGUTdFcThpWW02K2ZNY0VRY3pVczJQREt5dUtm?=
 =?utf-8?B?MEpJSkE4aDZSNEdEYVBQemtMN1l1Nkhpd0g5UXQ0M2dId0N5cTRneFNWclhS?=
 =?utf-8?B?NFhvWk9vUU52dWI4UlVHUmVUcGdrRlpXeTEwOHRlK3piMmhpaFpxTldTVW92?=
 =?utf-8?B?S2w4SVg1NFpiTGhYWXFLRmFDUDA5ZjRaWnQwelp0QU5ocWZLeGpIb0lBUkZr?=
 =?utf-8?B?TVNNNUNjc29JeDBVKzFGUXMwQmhRbjI2VWVSZWs5aFVlWFUrZks1ZWY1QW0w?=
 =?utf-8?Q?hzH7OfUhnpp7Xcwbp0knIQTVM/3cpUcoH+jjRNpF4nGKM?=
X-MS-Exchange-AntiSpam-MessageData-1: wkvnfTQJ0m0pGw==
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ae16e21c-a637-431f-cd80-08da2c4525cd
X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2022 14:07:52.8250
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yEFnjEtK8/HHdSPRNxuE4qAQVVupoTo14OJgcNkwmxyHlPpOd6p+oOos6vhwQUAvJSH6hbp9vOuPoeNw5VgSmQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5488
X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Maxim, Sean

On 4/18/22 7:55 PM, Maxim Levitsky wrote:
> On Tue, 2022-04-12 at 06:58 -0500, Suravee Suthikulpanit wrote:
>> When APIC mode is updated (e.g. disabled, xAPIC, or x2APIC),
>> KVM needs to call kvm_vcpu_update_apicv() to update AVIC settings
>> accordingly.
>>
>> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
>> ---
>>   arch/x86/kvm/svm/avic.c | 15 +++++++++++++++
>>   arch/x86/kvm/svm/svm.c  |  1 +
>>   2 files changed, 16 insertions(+)
>>
>> diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
>> index 22ee1098e2a5..01392b8364f4 100644
>> --- a/arch/x86/kvm/svm/avic.c
>> +++ b/arch/x86/kvm/svm/avic.c
>> @@ -616,6 +616,21 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu)
>>   	avic_handle_ldr_update(vcpu);
>>   }
>>   
>> +void avic_set_virtual_apic_mode(struct kvm_vcpu *vcpu)
>> +{
>> +	struct vcpu_svm *svm = to_svm(vcpu);
>> +
>> +	if (!lapic_in_kernel(vcpu) || (avic_mode == AVIC_MODE_NONE))
>> +		return;
>> +
>> +	if (kvm_get_apic_mode(vcpu) == LAPIC_MODE_INVALID) {
>> +		WARN_ONCE(true, "Invalid local APIC state (vcpu_id=%d)", vcpu->vcpu_id);
>> +		return;
>> +	}
>> +
>> +	kvm_vcpu_update_apicv(&svm->vcpu);
> I think it makes sense to call avic_refresh_apicv_exec_ctrl directly here.
>   
> I am not sure that kvm_vcpu_update_apicv will even call it
> because it has an optimization of doing nothing when inhibition status
> didn't change.
>   
>   
> Another semi-related note:
>   
> the current way the x2avic msrs are configured creates slight performance
> problem for nesting:
>   
> The problem is that when entering a nested guest, AVIC on the current vCPU
> is inhibited, but this is done only so that this vCPU*peers*  don't
> try to use AVIC to send IPIs to it, so there is no need to update vmcb01
> msr interception bitmap, and vmcb02 should have all these msrs intercepted always.
> Same with returning to host.
> 
> It also should be checked that during nested entry, at least vmcb01 msr bitmap
> is updated - TL;DR - please check that x2avic works when there is a nested guest running.

In the kvm/queue branch, I found a regression on nested SVM guest, where L2 guest cannot
launch. The bad commit is:

commit a4cfff3f0f8c07f1f7873a82bdeb3995807dac8c (bisect)
Merge: 42dcbe7d8bac 8d5678a76689
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Fri Apr 8 12:43:40 2022 -0400

     Merge branch 'kvm-older-features' into HEAD

     Merge branch for features that did not make it into 5.18:

     * New ioctls to get/set TSC frequency for a whole VM

     * Allow userspace to opt out of hypercall patching

     Nested virtualization improvements for AMD:

     * Support for "nested nested" optimizations (nested vVMLOAD/VMSAVE,
       nested vGIF)

     * Allow AVIC to co-exist with a nested guest running

     * Fixes for LBR virtualizations when a nested guest is running,
       and nested LBR virtualization support

     * PAUSE filtering for nested hypervisors

     Guest support:

     * Decoupling of vcpu_is_preempted from PV spinlocks

     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

I am still working on the bisect into the merge commits.

Regards,
Suravee