Received-SPF: pass (google.com: domain of linux-kernel+bounces-147644-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <4cdf71ca-c4e4-49a5-a64d-d0549ad2cf7b@oracle.com>
Date: Tue, 16 Apr 2024 17:29:21 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC 0/3] Export APICv-related state via binary stats interface
To: Paolo Bonzini <pbonzini@redhat.com>,
        Sean Christopherson <seanjc@google.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        joao.m.martins@oracle.com, boris.ostrovsky@oracle.com,
        mark.kanda@oracle.com, suravee.suthikulpanit@amd.com,
        mlevitsk@redhat.com
References: <20240215160136.1256084-1-alejandro.j.jimenez@oracle.com>
 <Zh6-h0lBCpYBahw7@google.com>
 <CABgObfZ4kqaXLaOAOj4aGB5GAe9GxOmJmOP+7kdke6OqA35HzA@mail.gmail.com>
Content-Language: en-US
From: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
In-Reply-To: <CABgObfZ4kqaXLaOAOj4aGB5GAe9GxOmJmOP+7kdke6OqA35HzA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?K0h2Tk9OMThUV1c3MkxWdTFQU3hHSSsyek1UK2p6UXFzN013UUJvaVJDeUE3?=
 =?utf-8?B?aDVrbmZ6dG5td1oxQ0FvYzF6blk1L2lZTi9NVVlXbll3aU9KM3Y0dWdJYjhF?=
 =?utf-8?B?Z3ZsUmQ5QjI0N2JkSEIvWXFsV0xzbnNkZ1NJT1pncnNNdzlYZ3dEbnNlVEJl?=
 =?utf-8?B?RFRxRHk0bHEyeXlsZWhxNFVoVnlYZE1vZ2hZZGl4ZzhhdGNhTzVGY3B1NXEx?=
 =?utf-8?B?ZzRaS24wRUhJVUFtYzcrOHRaZlh4azMrZGZMM1NoQlNJdDd4U0wwMkNDNWtU?=
 =?utf-8?B?NFFyUUJLZ2gxOWRKOU1RY05TVkErRHZEczVFWHlqZnpCbEtyb0xOMU1BVk00?=
 =?utf-8?B?Mlk1MkNjaVhPbFoyUEhnbnJxSmkrTXRZS3hlVFFUK0JLZGJGbWFYT0d5Tkdm?=
 =?utf-8?B?a2RrKzE3SXRkYnVPZ1c3bmxVNllMNWkvZmFPYmFMZnh3eVNzdXV5azFuS2dB?=
 =?utf-8?B?amRENUJNOGtNR1NvUW1oK3NJL1RFN01GaU5lQVkwODBjOUFmS3lleUZVNU1a?=
 =?utf-8?B?Z3FSWGtuSjgvdGVjelNsZzlQcXJ4SzNxNnJPYTZjRXJLd1ZpZTEySHc3QWF6?=
 =?utf-8?B?M0l2VklabDhnUnNWaDdWM2RVbURReVRJT25nTEtPNVBQZlY1MmhEdjh0c0VU?=
 =?utf-8?B?Q1JpRy94QmU5a08weFcrL2hUQlBRVnZuZ29rSnh4SldEQWJ1M2E5MjJwaGx1?=
 =?utf-8?B?aHBVZWhtaWc1VDlvalBEU21mT05HV1RpWW81UFQrKzB6UzZTV1FlT0hHdWx0?=
 =?utf-8?B?TitBRHMralFmeUxhYnVWdVRQVDlXZWR4dlJ4Qm1kQXFrM3pOdHBUb0hrc0VL?=
 =?utf-8?B?K2dvRUdyempQWlA4b2FHQ3A4UmR6M29WeW91VmNuaEptUklaOElLS0ZEV2gw?=
 =?utf-8?B?L01LOUYzVWV1UnhzZXVrU3dXSnNKQkR2ZWRpZE9QRjA3TWpUTWFIZkdxWEY0?=
 =?utf-8?B?Q25zSXN5MWZJU0pjR2FHRjhBcUN0eitMc0lub3d1ZzlBQUtEZkJIK1JhK3gr?=
 =?utf-8?B?RGlJY3Zva0FEbUJyM0htMW5IS29uN2RpcjBPSEw2dU5lTkFBVHo2RUluZERN?=
 =?utf-8?B?S2RtZ2ZDZzh3K1ZncGdaaFNhSk1EYTN2NVBnV0ptRVJ0bTU1WW5idTRxQ0VC?=
 =?utf-8?B?ZlUrMzM1b0Qvb2hDRno1bFR2WXJoTFpFa1VtNk8vRkh1dDkySHJpM1VFVDhK?=
 =?utf-8?B?VFAxL0ltUVUwU0w4b3M3T2JvNnJ2a3JrVWFCeXU4RWZHTzd2ZktNbURpREhK?=
 =?utf-8?B?UFFlL2VhRDlMM1pxeW9Wb1o1STQrSkIzTGVNaTRoaGdCVXZHQzdoa0xVWStk?=
 =?utf-8?B?TkM1UDRyN1JiTGlldTFMQUxJbjFjdDNtNThsN1NnZDJyVVZQYUMzWjBxVGJ5?=
 =?utf-8?B?bkY2b0JiT2dmL2E3ZlJXeldZbDVZcHgySDR1c203YUhLWThrMzBrQkFtWjZu?=
 =?utf-8?B?Zzk5Zm5jWEJZYVA2QlRPbmllUEpDUitRWUdmakJpbER2U3VmQjNKclZHREsr?=
 =?utf-8?B?RG1STVIyQkhqNzcrZlA3cFVZY0xpVHdod1NPTHR2b0ZZRko4STdXZFM2UXJp?=
 =?utf-8?B?YTVTQXM2eVpsNTJ5YVJqemwrNVNXRlNVakdMU0JnSlZZSGFHNXNuaERkME53?=
 =?utf-8?B?MFp6aEFzRGZPSlh6SExFNjAvTVpFc1pPdGw1NkJVbGx0WW1RSktkSGNIVUJK?=
 =?utf-8?B?M2lLa2dEdTA0Q21Yb2tONGJlQXNoM1pFbWhucys3Qitsa0UxaUlVdW8xY3Ba?=
 =?utf-8?B?ang1aXJvZU1rT1BtUWovYW1VTmc3WnY4d0JpVnhUZDZGNjh6SVc4R0Y5YjBE?=
 =?utf-8?B?MmFBNGFZblo0amxtb0VlUkFCVk9NWEprM2ZKOGNHcWhydGFmcUtOZmlUVlJP?=
 =?utf-8?B?VHIrZFB0cEpVRDl0N1dGMndRcVluekZaeW5xd3U1NXZoRVB4MExjL2c2bXY0?=
 =?utf-8?B?N0ZsMXY1MzYrdzE0SFlnMndoVnlhZEp0ZWxOMmxvbEZNL2FxczJya1lTTysx?=
 =?utf-8?B?U3ovOWtMeVUzQ0FDNnU5UHJQelFTMEVoVTk2Q1RzYzJQdFpuaDNmWUduTWxT?=
 =?utf-8?B?amdPWWNkb0w1SVJuRjFJaDl4QlRkNFJtVmVCOU9GVFA3Z282RE5GVjRMeEFq?=
 =?utf-8?B?NnJBQnZpZlFYckxadEl0OU5zemNKWFZLMTVGdTIvTHdTQnV0MEdZWnNHYXlJ?=
 =?utf-8?B?ZUE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	BG3RkUeYnqd8HJdSKBFA1+6zSMaLT/u37VcCol2wSswVUtN7g6finHbSgcSxZ7UQyeMXsdt9xTuNgwxPRDvodtfxStbCyFerKYjCiH2+VI9wOq9U9a9LyGmOI2DeL5yPv+AtPfiW+AfhsgonJpZJX6XHeP1imIlcwUELF1KxwAwteUqjjqtuh/uLV8C8w/f9YmKFHfnqObT4RHSA0Y6LSp/xFuXvP7sIIZBVaDcnjEPlwvqMsvIZIP/OjI2MQruFNOUladG2PRmlnwGGZ3KE8dyUn7bqDpHfzXFSwkHDewrhaNyNFiORf7m3xUfZ6xXQ9whLKiOe5mwdMMX/GoLJiW3E3fzrnT1e1iJl2KK9QZDhbNtOvWXhBNC8l4D7Fe4Jj5pwrqq6EZXrhITgrM1NoewvldrR9+18o3RVEHD4bissLSEpIa84ioTEGPxsItKdIjeYeoKtQDQ+mtkiV01TaXbPX8pFlcDWkJuiOpWQ56DwO143FGyjVuuxYG/OMip4j9SUETS2GbVlChGQtsF4Vj3ahCfvJV5NJjT+o9LeywuzQ3USGXArCrRxPkaVMmTMtHWcktIG7eBwzk9VIU+lZMTQ0sTkTC75SJ3eQI5QZpo=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9000b1b2-4d89-44fb-e426-08dc5e5c4928
X-MS-Exchange-CrossTenant-AuthSource: DS7PR10MB5280.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Apr 2024 21:29:23.9874
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 64MpaGQMvY6p8OfRwmOw5QB8IciLJ8wPpfHr5i0sV2260NB7w2q5adp4Ecd1s0sAkBEp3ckQJecFD/b0GQGdCgrOwH6jivP4tNcNXKNlTgU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4484
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-16_18,2024-04-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 mlxlogscore=999
 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000
 definitions=main-2404160138
X-Proofpoint-ORIG-GUID: u0FpoYFCb9uvB1ybkR4aq4e3BYrOBjED
X-Proofpoint-GUID: u0FpoYFCb9uvB1ybkR4aq4e3BYrOBjED


On 4/16/24 15:51, Paolo Bonzini wrote:
> On Tue, Apr 16, 2024 at 8:08 PM Sean Christopherson <seanjc@google.com> wrote:
>>
>> On Thu, Feb 15, 2024, Alejandro Jimenez wrote:
>>> The goal of this RFC is to agree on a mechanism for querying the state (and
>>> related stats) of APICv/AVIC. I clearly have an AVIC bias when approaching this
>>> topic since that is the side that I have mostly looked at, and has the greater
>>> number of possible inhibits, but I believe the argument applies for both
>>> vendor's technologies.
>>>
>>> Currently, a user or monitoring app trying to determine if APICv is actually
>>> being used needs implementation-specific knowlegde in order to look for specific
>>> types of #VMEXIT (i.e. AVIC_INCOMPLETE_IPI/AVIC_NOACCEL), checking GALog events
>>> by watching /proc/interrupts for AMD-Vi*-GA, etc. There are existing tracepoints
>>> (e.g. kvm_apicv_accept_irq, kvm_avic_ga_log) that make this task easier, but
>>> tracefs is not viable in some scenarios. Adding kvm debugfs entries has similar
>>> downsides. Suravee has previously proposed a new IOCTL interface[0] to expose
>>> this information, but there has not been any development in that direction.
>>> Sean has mentioned a preference for using BPF to extract info from the current
>>> tracepoints, which would require reworking existing structs to access some
>>> desired data, but as far as I know there isn't any work done on that approach
>>> yet.
>>>
>>> Recently Joao mentioned another alternative: the binary stats framework that is
>>> already supported by kernel[1] and QEMU[2].
>>
>> The hiccup with stats are that they are ABI, e.g. we can't (easily) ditch stats
>> once they're added, and KVM needs to maintain the exact behavior.
> 
> Stats are not ABI---why would they be? They have an established
> meaning and it's not a good idea to change it, but it's not an
> absolute no-no(*); and removing them is not a problem at all.
> 
> For example, if stats were ABI, there would be no need for the
> introspection mechanism, you could just use a struct like ethtool
> stats (which *are* ABO).
> 
> Not everything makes a good stat but, if in doubt and it's cheap
> enough to collect it, go ahead and add it. Cheap collection is the
> important point, because tracepoints in a hot path can be so expensive
> as to slow down the guest substantially, at least in microbenchmarks.
> 
> In this case I'm not sure _all_ inhibits makes sense and I certainly
> wouldn't want a bitmask,

I wanted to be able to query enough info via stats (i.e. is APICv active, and if
not, why is it inhibited?) that is exposed via the other interfaces which are not
always available. That unfortunately requires a bit of "overloading" of
the stat as I mentioned earlier, but it remains cheap to collect and expose.

What would be your preferred interface to provide the (complete) APICv state?

  but a generic APICv-enabled stat certainly
> makes sense, and perhaps another for a weirdly-configured local APIC.

Can you expand on what you mean by "weirdly-configured"? Do you mean something
like virtual wire mode?

Alejandro

> 
> Paolo
> 
> (*) you have to draw a line somewhere. New processor models may
> virtualize parts of the CPU in such a way that some stats become
> meaningless or just stay at zero. Should KVM not support those
> features because it is not possible anymore to introspect the guest
> through stat?
> 
>> Tracepoints are explicitly not ABI, and so we can be much more permissive when it
>> comes to adding/expanding tracepoints, specifically because there are no guarantees
>> provided to userspace.
>>
> 
>