Message-ID: <86c79601-2d3d-a6d7-a0a5-baba03e00709@amd.com>
Date:   Thu, 16 Feb 2023 03:15:40 +0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH v2 2/3] KVM: SVM: Modify AVIC GATag to support max number
 of 512 vCPUs
Content-Language: en-US
To:     Sean Christopherson <seanjc@google.com>,
        Joao Martins <joao.m.martins@oracle.com>
Cc:     Igor Mammedov <imammedo@redhat.com>,
        Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
References: <20230207002156.521736-1-seanjc@google.com>
 <20230207002156.521736-3-seanjc@google.com>
 <20230207093350.5db155ca@imammedo.users.ipa.redhat.com>
 <0e12d654-d388-a0f9-e7f9-7e96921786b7@oracle.com>
 <Y+J+dS8ZRX07kgt7@google.com>
From:   "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
In-Reply-To: <Y+J+dS8ZRX07kgt7@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cDVMMXNGYmxoUWdKb1FWUDNoRTJYM1ZZWUFoaEMrTENEN3Vtd2gvdDE3SUFr?=
 =?utf-8?B?cEtqZUtpbjdXa3haeHB4emM1bTNoSDlmaXJRSUREaCtraGFIaHRXMjJPdXhT?=
 =?utf-8?B?N3I0L085QlB6d2dSM2w3bWV2U0N6cFBvV1NpRERMMmRWdVkwcmwwVm10eFpl?=
 =?utf-8?B?R1orS1VFSGpWVjVJd3o2Mkxpd0VPNlJ3Z053U2ZWQ3gyckt1K0pYbEU1aHZQ?=
 =?utf-8?B?U3RIa2c4Z3dvMkN6eHlBaVdyRnBrU0VRL2YvRmNPWTZKdG5rL3F2dXhkNFQy?=
 =?utf-8?B?YW5jWGFIaTc1WmZWR2gzUTJ6OXR5NWVNc3JHV1pKVUFjQW1Pa0NuMitaTEU5?=
 =?utf-8?B?MGVadS9UaDRRNGtkVG9RS216WCsvaXhtanZEZkdiVjY3Mk5qZ3U5VlA3Sktk?=
 =?utf-8?B?S3RlSkFNd01Ea1hOYkxYdVA2YkhuY3V2eDgrTmhzWld1QzBmTmZGVlhoYTk4?=
 =?utf-8?B?WkVHbUN0ZjdRem50SFNyUEkraVp0d2RXWFFkVDdiY3hqZ2VtY2R0UUVTN204?=
 =?utf-8?B?VFR3bG1NdmIwc0pyQ2RCd1M4bDdBWUw2ZWlQeFlVc3hPZ0d2WGtubE8xUEFh?=
 =?utf-8?B?MTh6WGZKTmZjTUtlaW5VTW5hNFlmeEdOanhUTXV5T2l6U1ZXNXVOOXFOREZr?=
 =?utf-8?B?bFJOci9UbVNMR1Azd1FtcE4yd3RuZmVyeCs2U21wV1krN2xjYUdyL0JlRHhL?=
 =?utf-8?B?R2l3dktYRFA3VjN5VHYyTnBtSVFDaGwrTHpjajFES2xvdWdtN3NwQjh2VUZT?=
 =?utf-8?B?NmIzWldhaUdGRitNYVh0aC8rM2x1cnpPei91ZENQS1NiSWZWZ0dmWWw4R2Fs?=
 =?utf-8?B?REFJUE5xTFZ5aUpJZ3dnQlVnek4waWV2dXRiWGtPZUJudVk0VWV5TXlwMFgy?=
 =?utf-8?B?Ti9YWFZUUWtueTNuRkZUcFdxb2tHNnBzMzk5RjBmQmlzcUxlMnMwWHlnV3hP?=
 =?utf-8?B?bTJtMW51YjgweUZlOXNwbTVYdjNSOTlCZVdBLzg4N2FJV2gzV3lhQXBCWDhm?=
 =?utf-8?B?cXhpRFpTT0hwdGc5TEpoRUVzQkVybU1Gb3ZhRzdkdUxVQ3ZxM2FIMHNpK1lK?=
 =?utf-8?B?WkZzb2ZPc2ZhNitCS1hrWmdIbkJKeXlsQjZKbFZsMGkzZUtjTWJzVm93MFhO?=
 =?utf-8?B?TEVJV2FTZ2hSWHNWN09jREl2TUVFTFRTdDI3SzVtOEY5c09MUHlmL3VUWTJI?=
 =?utf-8?B?YWsyejV5ZDMyLzZBaTV1VGJ2RnRSVU1SNWNLWXd4cm82ZmUzZmh6N2NrdGR1?=
 =?utf-8?B?aFJxdUtPKzJFYmkzVkJ1Q0RIaDEvMHBFaTU1ZDVpcEZ0R0dFVDNtNWNtYzRj?=
 =?utf-8?B?Uk9COGhiN3p2UWwzd2FhU0ZiWUJHaTNScFA2T1hpNm96QTNuckdjSlR4dUkx?=
 =?utf-8?B?dHYwMjltUzgyOHdENUpFL0FZZlZiQldxNkQ1TkNsWDhaajJBVG9Hdzc1MTNX?=
 =?utf-8?B?QnNzeEV6ZmRVbUM3Yk9sMlFQUUc1SkVvS2Zta1dIaE1RNGoyb2ZYQjBwVCsz?=
 =?utf-8?B?dUptVDgxMWRyaU5WY0RiRDRaRHVVRmF0TzhORE1qbG9UMXpiNm54TUJnR3hj?=
 =?utf-8?B?V09Ka29uV0lMWWp6bXdxQVZpbUNPdC9VNXVvQUFvTGc0R2UxbFV0WTF4K1dZ?=
 =?utf-8?B?OWdSYmg1VU9ramxZSUlsTkk1SFpmSVBxdU9sKzd3RjBjK0h5bGFpMFhaSWdU?=
 =?utf-8?B?SGFsejR1ZWlWVE1IeVNWNjR3NlE5dGFCZzJiVmNHdEFITWR2RUw2M3J3ZXc3?=
 =?utf-8?B?UmY2YWpUZW9VanlPdmZnYk5MZFpJVWUvY3BqMEJYOWtLNForblZWTWhKTzRS?=
 =?utf-8?B?NEhMcDdjMHF5MVlnRWRsenZsQ3hpcDhEcVFRSDErRW1PbjNpMDdtZjhXS2JG?=
 =?utf-8?B?WmVIOHo5WEpieHVyOGlWWUtIVTRYQzM1UTZ1QmZibXg0MTFlN3A5cVN3NUNw?=
 =?utf-8?B?UzFDVnEzVmlHRlJ3VEw2ektNZXJoSFBXV2RVT2QzVFhFNiswVWQwbmVraHcx?=
 =?utf-8?B?dDNXMndYMVBiWWJvYW5DUmJhK3ZRTytzamQwdC94MFpCbDVub0o4Sm1PYjB5?=
 =?utf-8?B?N0tvQjYrT1dESnNmSjh5Nm1IL2RyOVpTMEcyeEFuTEJBMkdCQ3A4d3lHZkV4?=
 =?utf-8?Q?zu/VCEB//Ezyt70cuH3hbtl8W?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4dfa35eb-e3b2-4d8a-bad6-08db0f916e6e
X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 20:15:50.4054
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FIEebas+jTdAfGSXa/EbtxcTyUYsp/2yQFp+ryhPreHvSMO7P8VSKjYnb40kmMHXRMayY24xgrXiWV2Q8vaK4w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB7082
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 2/7/2023 11:38 PM, Sean Christopherson wrote:
> On Tue, Feb 07, 2023, Joao Martins wrote:
>> On 07/02/2023 08:33, Igor Mammedov wrote:
>>> On Tue,  7 Feb 2023 00:21:55 +0000
>>> Sean Christopherson <seanjc@google.com> wrote:
>>>
>>>> From: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
>>>>
>>>> Define AVIC_VCPU_ID_MASK based on AVIC_PHYSICAL_MAX_INDEX, i.e. the mask
>>>> that effectively controls the largest guest physical APIC ID supported by
>>>> x2AVIC, instead of hardcoding the number of bits to 8 (and the number of
>>>> VM bits to 24).
>>>
>>> Is there any particular reason not to tie it to max supported by KVM
>>> KVM_MAX_VCPU_IDS?
>>>
>>> Another question:
>>>   will guest fail to start when configured with more than 512 vCPUs
>>>   or it will start broken?
>>>
>>
>> I think the problem is not so much the GATag (which can really be anything at
>> the resolution you want). It's more of an SVM limit AIUI. Provided you can't
>> have GATAgs if you don't have guest-mode/AVIC active, then makes sense have the
>> same limit on both.

Correct.

> Yep.  The physical ID table, which is needed to achieve full AVIC benefits for a
> vCPU, is a single 4KiB page that holds 512 64-bit entries.  AIUI, the GATag is
> used if and only if the interrupt target is in the physical ID table, so using
> more GATag bits for vCPU ID is pointless.

Correct.

>> SVM seems to be limited to 256 vcpus in xAPIC mode or 512 vcpus in x2APIC
>> mode[0]. IIUC You actually won't be able to create guests with more than
>> 512vcpus as KVM bound checks those max limits very early in the vCPU init (see
>> avic_init_vcpu()). I guess the alternative would an AVIC inhibit if vCPU count
>> goes beyond those limits -- probably a must have once avic flips to 1 by default
>> like Intel.
> 
> I don't _think_ KVM would have to explicitly inhibit AVIC.  I believe the fallout
> would be that vCPUs >= 512 would simply not be eligible for virtual interrupt
> delivery, e.g. KVM would get a "Invalid Target in IPI" exit.  I haven't dug into
> the IOMMU side of things though, so it's possible something in that world would
> necessitate disabling (x2)AVIC.

SVM-AVIC is independent of the IOMMU-AVIC. We can enable SVM-AVIC, and 
use the legacy IOMMU interrupt remapping mode IRTE[GuestMode]=0.
However, I have not explored the case of combining of the two modes. I 
can look into it and experiment with this case.

Thanks,
Suravee

>> [0] in APM Volume 2 15.29.4.3 Physical Address Pointer Restrictions,
>>
>> * All the addresses point to 4-Kbyte aligned data structures. Bits 11:0 are
>> reserved (except for offset 0F8h) and should be set to zero. The lower 8 bits of
>> offset 0F8h are used for the field AVIC_PHYSICAL_MAX_INDEX. VMRUN fails with
>> #VMEXIT(VMEXIT_INVALID) if AVIC_PHYSICAL_MAX_INDEX is greater than 255 in xAVIC
>> mode or greater than 511 in x2AVIC mode.