Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d02d0b30-f29b-0ff6-98c7-89ddcd091c60@oracle.com>
Date:   Tue, 27 Sep 2022 23:15:25 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.0
Subject: Re: [PATCH v3 05/28] KVM: x86: Don't inhibit APICv/AVIC if xAPIC ID
 mismatch is due to 32-bit ID
Content-Language: en-US
To:     Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>
Cc:     kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        Maxim Levitsky <mlevitsk@redhat.com>,
        Li RongQing <lirongqing@baidu.com>
References: <20220920233134.940511-1-seanjc@google.com>
 <20220920233134.940511-6-seanjc@google.com>
From:   Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <20220920233134.940511-6-seanjc@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Uy9qMnI5dTRrYXVXQ0hLZFIwV1J0d3V1d2FrMUMveFFjNnZmVW0wUEkxMDZn?=
 =?utf-8?B?NCtoRlhNR2Vjd0NrM1hSVU9oKzlGRE9SRlFyWUVJOEVtM3I0bGpzdENwSXdY?=
 =?utf-8?B?R08rT28yNWZTWm5TcUFYL0Zsb1dEbkxqM3doelpqVGdrWFJxR1NhcS9Vb2Rs?=
 =?utf-8?B?UHJ6S3BYVHZuVGFVeStzY0ZaSjdoaFR5cUg4dE56YXBCRzhRVllaWHBPVENV?=
 =?utf-8?B?cEx1N3NTYWdjWFRqWGF5TUJmS0ZvNzBqbzM3aldoSXJZVUp6OVI4czVNOENV?=
 =?utf-8?B?VDEydk5UN1VRdEZjMmtueVZzdGR4d0VZeWROSlVwK0lNWkFzVXBDZHBONDlR?=
 =?utf-8?B?TGFTSmx2SVJSSHhIME1YZHpxZGlkVXB3aXg3cHJqZ1orZHJaYjF0dm5oaVBr?=
 =?utf-8?B?MXdNZ0kwRXJzM3cxTlpnR2tDZ3ZTaG1YQ0lwNDFMZkNVZENocnUwTG1TSzQ4?=
 =?utf-8?B?Q0hEMzdUWko1cG5iUmJtV3o1QUpNZ2xCZWJkeUoyLzFjTE40RlR0Z0QzQlpD?=
 =?utf-8?B?S2pOaHh0ZGpYQ0xPbkFpakJiOHJpVC9MaUlHUUJndlpGUjJnUzIwZE5wUW1M?=
 =?utf-8?B?OGlFRk84clA3YTc0OGpQa0VYaHhDYjJySzMyeWdXeGFhdlRKM2xMRDB1OWFx?=
 =?utf-8?B?OFM0bTlWQ3d6SWJ0U3RETmNIZTBmaUJqOVRvVVI1dmdpOEQ3RCtkZ1FHTita?=
 =?utf-8?B?cUluME94TUlWZzRnRG9NaFgxajhrNDJTUHJxMHoyeUtDN1ZERThxMkRnaEln?=
 =?utf-8?B?Qkhxa3hNRjNaVHQ0QmlNSDZTU0h6L09EaTYxZ0d4TVlwNzNSaDJRcUFvdmY3?=
 =?utf-8?B?SkphQytlNWFaV093R3FSM1ZIM3VXeUtad1JMR2dPMUJ4U1lKQi9PNUJVUVNL?=
 =?utf-8?B?ZFZCOStmKzhmbm1Mb1ZIK1BERWJpNzh3bmxMTjZsc2VYRDcrbWV0dW5aejU4?=
 =?utf-8?B?L1FVazh4S1VzOUc0VHJuQ2JLVlc4bzZxbVFweGgwUDg5QUpJdFB2cDhMU01p?=
 =?utf-8?B?UlNBWnBUd1VLQjBLc0hlZjlma1Zrb3RONzlndWViWnNGOG4reUpscnREbHpZ?=
 =?utf-8?B?N0dFS3J0aUZXZXJBU2hHQmZucHB5cW54WW5obTZGcGI4YUtVWkRydHNPbGZw?=
 =?utf-8?B?WVFVbk1BdFN3UWlCMVNHT1VxVDV6MURoK2tGS0Q3aHhubnRrTmIzUkd5K2Rp?=
 =?utf-8?B?N1BEOWJiR0t0Mm1Td2tJeXVMMjA4eUppWUdVSVRQTjdQZ1U2MTgrR2dib1Vz?=
 =?utf-8?B?ODd3YXNuWnluSk4ySm1PSysvSUlQK0xNQmRhUllrczZYOHh3eWhPUDQxNXpi?=
 =?utf-8?B?YVhxNEU1Z0wybnVpYkE0eWw1Vzhya21QNi9DMDZTMDQwL0RScUMxQXpuOEFn?=
 =?utf-8?B?ekRDK3VuSGN5ejJmNnZzaFVwS2FDRTNQRTlkSFFhdWkxZDJ3R3RpNWtOeFI0?=
 =?utf-8?B?WUQraVFXTXBxaHo2S3Y5aFd4TE9vQ29lZHRpNHpBZFRwdlNiTm1sOTQybXhw?=
 =?utf-8?B?WThkcGZBRlU3M1kxT3BZeUhZOWtYcWFWTysrMGJZenVYWXpNd01Bd2NRVjZu?=
 =?utf-8?B?ZzRTNEFrMXZ4Qm5SZi94Nyt4Z2lqNzZickhaKytBU2I5OGJxQUxIandXaTBI?=
 =?utf-8?B?WVdUNWpXQUJpRG9XMUk0MEFxSjB2MHM4STZycWtZdkNxSXcwMm9nbnEwSWEv?=
 =?utf-8?B?a2Z0Uk10NFpLcUdsalY1WlNjbmVRZzJHNk9USUV3MjR6TnVRbmhlR3duQWtM?=
 =?utf-8?B?NHNxK2lYdGZUZmw1MnpYRTUzZDRjdkRFWllxYUg4SEtaa1c0UjdiTDlGdm84?=
 =?utf-8?B?OWdzaFhvMzgyVkFoZWxjSVpTK05oaVROa2Z4YXlFeDM5bTlEaDJQUEJ4enBC?=
 =?utf-8?B?cnZJNGFyQVRkZ3VHcGJDTCtsYjdiL0tnNjM0US9qTjdqVDY1bmJOa204bHRv?=
 =?utf-8?B?bE1LdVlYT0h1SjNsY1NCRDNucE1jL1kzS3ppWHd2bWx5MmFrc0s2aFRtaUx4?=
 =?utf-8?B?OHNYRE1STklvVHlsYW9wN0ZZN3grazFrdWNLU2o0WDNFMjkzRFNGbW1aVXBw?=
 =?utf-8?B?Wll5eHRXQ0hUZ1ZSblQzY2YvWmJnR2V3VUN6Ny8xS2N6VXQ5VEtsNDBHUndn?=
 =?utf-8?B?SnpFblorTldPS2ZVbU5KT2g2U25PREJqWDhHRkFCMGVZZ1JieE05SDNpa1Nt?=
 =?utf-8?B?REE9PQ==?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a9a611c0-e698-4a17-a596-08daa0ffb15e
X-MS-Exchange-CrossTenant-AuthSource: DS7PR10MB5280.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2022 03:15:28.1020
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WQTPYCPw0p9gwFf1fEWeQJC2I5GkTm6s3B4fS0uGmp+5jPitDZynV5jbhyQuJian6gtwjV6f+VTWV8kKVVGdv6wGppsQf8wpghp6VUFe1tI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4288
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1
 definitions=2022-09-27_12,2022-09-27_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0
 suspectscore=0 adultscore=0 mlxscore=0 spamscore=0 bulkscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2209130000 definitions=main-2209280019
X-Proofpoint-ORIG-GUID: pZ_lqwX2KC2oTdeoAiod0EFh-2DgmNgg
X-Proofpoint-GUID: pZ_lqwX2KC2oTdeoAiod0EFh-2DgmNgg
X-Spam-Status: No, score=-5.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 9/20/2022 7:31 PM, Sean Christopherson wrote:
> Truncate the vcpu_id, a.k.a. x2APIC ID, to an 8-bit value when comparing
> it against the xAPIC ID to avoid false positives (sort of) on systems
> with >255 CPUs, i.e. with IDs that don't fit into a u8.  The intent of
> APIC_ID_MODIFIED is to inhibit APICv/AVIC when the xAPIC is changed from
> it's original value,
> 
> The mismatch isn't technically a false positive, as architecturally the
> xAPIC IDs do end up being aliased in this scenario, and neither APICv
> nor AVIC correctly handles IPI virtualization when there is aliasing.
> However, KVM already deliberately does not honor the aliasing behavior
> that results when an x2APIC ID gets truncated to an xAPIC ID.  I.e. the
> resulting APICv/AVIC behavior is aligned with KVM's existing behavior
> when KVM's x2APIC hotplug hack is effectively enabled.
> 
> If/when KVM provides a way to disable the hotplug hack, APICv/AVIC can
> piggyback whatever logic disables the optimized APIC map (which is what
> provides the hotplug hack), i.e. so that KVM's optimized map and APIC
> virtualization yield the same behavior.
> 
> For now, fix the immediate problem of APIC virtualization being disabled
> for large VMs, which is a much more pressing issue than ensuring KVM
> honors architectural behavior for APIC ID aliasing.

I built a host kernel with this entire series on top of mainline 
v6.0-rc6, and booting a guest with AVIC enabled works as expected on the 
initial boot. The issue is that during the first reboot AVIC is 
inhibited due to APICV_INHIBIT_REASON_APIC_ID_MODIFIED, and I see 
constant inhibition events due to APICV_INHIBIT_REASON_IRQWIN as seen in 
the traces:

qemu-system-x86-10147   [222] .....  1116.519052: 
kvm_apicv_inhibit_changed: set reason=8, inhibits=0x120
qemu-system-x86-10147   [222] .....  1116.519063: 
kvm_apicv_inhibit_changed: cleared reason=8, inhibits=0x20
qemu-system-x86-10147   [222] .....  1117.934222: 
kvm_apicv_inhibit_changed: set reason=8, inhibits=0x120
qemu-system-x86-10147   [222] .....  1117.934233: 
kvm_apicv_inhibit_changed: cleared reason=8, inhibits=0x20

It happens regardless of vCPU count (tested with 2, 32, 255, 380, and 
512 vCPUs). This state persists for all subsequent reboots, until the VM 
is terminated. For vCPU counts < 256, when x2apic is disabled the 
problem does not occur, and AVIC continues to work properly after reboots.

I did not see this issue when testing a similar host kernel that did not 
include this current patchset, but instead applied the earlier:
https://lore.kernel.org/lkml/20220909195442.7660-1-suravee.suthikulpanit@amd.com/
which inspired this [05/23] patch and the follow up [22/28] in this series.

I am using QEMU built from v7.1.0 upstream tag, plus the patch at:
https://lore.kernel.org/qemu-devel/20220504131639.13570-1-suravee.suthikulpanit@amd.com/

Please feel free to request any other data points that might be relevant 
and I'll try to collect them.

Alejandro
> 
> Fixes: 3743c2f02517 ("KVM: x86: inhibit APICv/AVIC on changes to APIC ID or APIC base")
> Reported-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> Cc: Maxim Levitsky <mlevitsk@redhat.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>   arch/x86/kvm/lapic.c | 7 ++++++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> index adac6ca9b7dc..a02defa3f7b5 100644
> --- a/arch/x86/kvm/lapic.c
> +++ b/arch/x86/kvm/lapic.c
> @@ -2075,7 +2075,12 @@ static void kvm_lapic_xapic_id_updated(struct kvm_lapic *apic)
>   	if (KVM_BUG_ON(apic_x2apic_mode(apic), kvm))
>   		return;
>   
> -	if (kvm_xapic_id(apic) == apic->vcpu->vcpu_id)
> +	/*
> +	 * Deliberately truncate the vCPU ID when detecting a modified APIC ID
> +	 * to avoid false positives if the vCPU ID, i.e. x2APIC ID, is a 32-bit
> +	 * value.
> +	 */
> +	if (kvm_xapic_id(apic) == (u8)apic->vcpu->vcpu_id)
>   		return;
>   
>   	kvm_set_apicv_inhibit(apic->vcpu->kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);