Received-SPF: pass (google.com: domain of linux-kernel+bounces-136245-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <19f634de-7d72-4abc-87ff-599d22e310bd@amd.com>
Date: Tue, 9 Apr 2024 10:39:52 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC 0/3] Export APICv-related state via binary stats interface
To: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>, kvm@vger.kernel.org
Cc: seanjc@google.com, pbonzini@redhat.com, linux-kernel@vger.kernel.org,
 joao.m.martins@oracle.com, boris.ostrovsky@oracle.com,
 mark.kanda@oracle.com, suravee.suthikulpanit@amd.com, mlevitsk@redhat.com
References: <20240215160136.1256084-1-alejandro.j.jimenez@oracle.com>
Content-Language: en-US
From: Vasant Hegde <vashegde@amd.com>
In-Reply-To: <20240215160136.1256084-1-alejandro.j.jimenez@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SkQreWt1WGZrOTBHZWQ4cXJCRzBXbVZzeDJmR0FJaW9rM0ErcFY5QXNzQ2pD?=
 =?utf-8?B?N2ZTdE9VU1FkQXdQZ2pPTjdDaVBUMnJsKzkzWTFLTkk2dFFaVGo2OS9yNVdJ?=
 =?utf-8?B?UVdKYXRYZXk2K2FibloybzdVbDVFM2ZiRG1UYjJYeU9zTndpc084b1pTbVhM?=
 =?utf-8?B?TUZMRG1qVFhZK2lqUWRTb3J1bVhBM1dRT3R6UzA1UUUvaUlwU1VadUhqR084?=
 =?utf-8?B?cmg2R1lYdzR6TmNuYStpcXRyNVdJMk8zTTJOV0ZNVlpaU3JKUjRPY2wyVnBM?=
 =?utf-8?B?bk5SVStPUkhZWVRnN1VmK0MxbVRINklZSFhqSXZVczc1dGNTVVBGd3Voc0tQ?=
 =?utf-8?B?bVZFZFNmajFUdUFOcyt5bTgzc1RINENwZnkrMjRqOHppZ29mQUVOYjd6MjBH?=
 =?utf-8?B?eHlnN2hSYitQdWcxRG5ZU3JhOWpFdVBQTitZU0UzR2VGSTdpM2xWMUdrVk8r?=
 =?utf-8?B?WGJqdzJpRUl1cHZiTFpyUkdlaFVNQ1lmWFFoY0RDZ3JtY1kzSmptTEY5bU01?=
 =?utf-8?B?KzVERE1mRXkrbmtDbEdVbzdLV2NpQmNQZ25Ub0lmZ1ZZaFBGS1BYZU1lVnMr?=
 =?utf-8?B?UEJLa1hRVUhFK25MK0ZuU1J5QTZOLzFYZEYxMVVFenA3NnoyTG5TWTFZRUdV?=
 =?utf-8?B?MFhJNEJ1bXZlYjZuQ1FHTHJScW1oWFNLVTlLYWZaVDNvRkYzVjQ5a1IzaStl?=
 =?utf-8?B?ZTV2QVBobnFDU3k5QUVCUXNFVEJ2VzZnejNEc1d6djJsa3FQOXVRU1lJL0tC?=
 =?utf-8?B?c1BPN0VMbmQwSDdXa0hLUGNTR1JMQmd3Q01PZGtHMzY4cmVjMitzSFN6L09F?=
 =?utf-8?B?OEU0bFZubjJ1TUNlTHhGTTVIcXZVc1MzZ0JhQW85cXNLWFJQS2FkVFlJT29l?=
 =?utf-8?B?T3YvMVkrbWVscTZzL0txVDltbDhYbUE3UGJwUk5CcmZzb1dIaUFYMTErUU9O?=
 =?utf-8?B?ZmEyOGU0Z09WZEsvZ1MwcWJwdzVpTEp6Tnlad3dPWFZRT010QzZ6UnhJWEZL?=
 =?utf-8?B?WjBZRmFTcCtGUGdnL0pKRXhMa2hkUmpuTmFXQmFvQjh3NVM5eFVRYW1uN3E1?=
 =?utf-8?B?UGpZcncrNnFrdTB0c1FWUHg0VVRlNTgyNlA5aGlyU2k4aVcvUk9xVGZwQkZi?=
 =?utf-8?B?dGNWbmE2TjdtTDhXNzgvcVJXOXBsUWFOOVYrZFlSdmRqTlhid0Z6YnpWZmx2?=
 =?utf-8?B?T2czNnZUVEs0aGpIamlBQ3AxZHhENUhFcEt6bGJnZXVuOW1PUU9KK2wzbG9q?=
 =?utf-8?B?NkFueXhkY0NrRnBpQ0E4Q2lUNytZb05wWUJnZjNpdGswUWMzWmVEQWNmdXVL?=
 =?utf-8?B?dTB5ZTZFZFdjNFBWY1E1S29mcHpPcHBrNXYwbi9PRHkzS2tuQTQ1a3doU0FN?=
 =?utf-8?B?QlFRaXhPVWxjZ3JkdUtPb0Zob2lpN2N3dCtWSVp6d0FCbzhscklpQzVYTW9L?=
 =?utf-8?B?VStocGVPeTVYR09ydUJnWnF2TnZnd2FtM2FMaHdXa1NRbkRaNGVySVpSbGtk?=
 =?utf-8?B?YnZDcitjZ2dNOHg5SkZZZFA3SmNJcWYwM1hteXJlU1MvUFg5akI2aEtvdDRO?=
 =?utf-8?B?WS9qZzNkcGdBWXdNYUdRSEtPTjB4WWs1ZWdyNzdBU2dhK01wR1hIL01Jb1Bp?=
 =?utf-8?B?alZwK3R3YVl3aVVwbVZwT3c0L2ZKTG1QUDdQNjRqVGpIV2NtbTFyekxodHNX?=
 =?utf-8?B?SXpOTGtJSVFsUGhJcFkzVGVnV1BCTTN6ZEJISW1UVHlST2pkU0t6bGhyODZV?=
 =?utf-8?B?MEI2dUE0Z05RWHRvbDJoZnY4OHZZaUl3cDEzancyTmF1YlUrL0VXb1YvenB0?=
 =?utf-8?B?S1hMaVE2SGxZaXl5UUl3TENPUTEzU1I0M1RKbFF2RHNkVGlwNWhDdmIrZkpR?=
 =?utf-8?B?VHE3d1FBM0VFZDBLNDJPSEg5SkdUbGc3QTk1NStXcEk0V1Z5YXZ5TjdMUmsz?=
 =?utf-8?B?UHBzd3NFb3pzaVhUdWU2K0xXKzY4ZVdCU29tWG9kemxDUjJjK2NqRW5oMnBx?=
 =?utf-8?B?TWdCeVlNZFdRZXpJc05ydWxuZUk5dnN6UmZKMW1pNjQ0Z0xPWUZlKzV0Y3I4?=
 =?utf-8?B?VThYRWNQaUhKcFRpdm9BNCtJYzVlUmZhZGdyZVdKblN0ZXJHeW9wY1lkRmRG?=
 =?utf-8?Q?uFCrNE2bCJRnfYgrM8ZArJXl8?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c5ef270d-b437-4ec1-4b36-08dc58534ed8
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6048.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2024 05:10:01.3118
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: U69+9jNVvSvsHW4SFr7Xk5ccYom/C7slEoTXg9nK2SlWKkxC6ETA+0huJK1XBtUiAqCVTcez3uSMuHIhGNI6Ag==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8174

Hi Alejadnro,

On 2/15/2024 9:31 PM, Alejandro Jimenez wrote:
> The goal of this RFC is to agree on a mechanism for querying the state (and
> related stats) of APICv/AVIC. I clearly have an AVIC bias when approaching this
> topic since that is the side that I have mostly looked at, and has the greater
> number of possible inhibits, but I believe the argument applies for both
> vendor's technologies.
> 
> Currently, a user or monitoring app trying to determine if APICv is actually
> being used needs implementation-specific knowlegde in order to look for specific
> types of #VMEXIT (i.e. AVIC_INCOMPLETE_IPI/AVIC_NOACCEL), checking GALog events
> by watching /proc/interrupts for AMD-Vi*-GA, etc. There are existing tracepoints
> (e.g. kvm_apicv_accept_irq, kvm_avic_ga_log) that make this task easier, but
> tracefs is not viable in some scenarios. Adding kvm debugfs entries has similar
> downsides. Suravee has previously proposed a new IOCTL interface[0] to expose
> this information, but there has not been any development in that direction.
> Sean has mentioned a preference for using BPF to extract info from the current
> tracepoints, which would require reworking existing structs to access some
> desired data, but as far as I know there isn't any work done on that approach
> yet.
> 
> Recently Joao mentioned another alternative: the binary stats framework that is
> already supported by kernel[1] and QEMU[2]. This RFC has minimal code changes to
> expose the relevant info based on the existing data types the framework already
> supports. If there is consensus on using this approach, I can expand the fd
> stats subsystem to include other data types (e.g. a bitmap type for exposing the
> inhibit reasons), as well as adding documentation on KVM explaining which stats
> are relevant for APICv and how to query them.

Thanks for the series. IMO this approach makes sense. May be we should 
consider adding one more stat to say whether AVIC is active or not. That 
way,
  Check whether AVIC is active or not.
  If AVIC is active, then inhibited or not
  If not inhibited, then use other statistics info.


I have reviewed/tested this series on AMD Genoa platform. It looks good 
to me.

Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>

-Vasant

> 
> A basic example of retrieving the stats via qmp-shell, showing both a VM and
> per-vCPU case:
> 
> # /usr/local/bin/qmp-shell --pretty ./qmp-sock
> 
> (QEMU) query-stats target=vm providers=[{'provider':'kvm','names':['apicv_inhibited']}]
> {
>      "return": [
>          {
>              "provider": "kvm",
>              "stats": [
>                  {
>                      "name": "apicv_inhibited",
>                      "value": false
>                  }
>              ]
>          }
>      ]
> }
> 
> (QEMU) query-stats target=vcpu vcpus=['/machine/unattached/device[0]'] providers=[{'provider':'kvm','names':['apicv_accept_irq','ga_log_event']}]
> {
>      "return": [
>          {
>              "provider": "kvm",
>              "qom-path": "/machine/unattached/device[0]",
>              "stats": [
>                  {
>                      "name": "ga_log_event",
>                      "value": 98
>                  },
>                  {
>                      "name": "apicv_accept_irq",
>                      "value": 166920
>                  }
>              ]
>          }
>      ]
> }
> 
> If other alternatives are preferred, please let's use this thread to discuss and
> I can take a shot at implementing the desired solution.
> 
> Regards,
> Alejandro
> 
> [0] https://lore.kernel.org/qemu-devel/7e0d22fa-b9b0-ad1a-3a37-a450ec5d73e8@amd.com/
> [1] https://lore.kernel.org/all/20210618222709.1858088-1-jingzhangos@google.com/
> [2] https://lore.kernel.org/qemu-devel/20220530150714.756954-1-pbonzini@redhat.com/
> 
> Alejandro Jimenez (3):
>    x86: KVM: stats: Add a stat to report status of APICv inhibition
>    x86: KVM: stats: Add stat counter for IRQs injected via APICv
>    x86: KVM: stats: Add a stat counter for GALog events
> 
>   arch/x86/include/asm/kvm_host.h |  3 +++
>   arch/x86/kvm/svm/avic.c         |  4 +++-
>   arch/x86/kvm/svm/svm.c          |  3 +++
>   arch/x86/kvm/vmx/vmx.c          |  2 ++
>   arch/x86/kvm/x86.c              | 12 +++++++++++-
>   5 files changed, 22 insertions(+), 2 deletions(-)
> 
> 
> base-commit: 7455665a3521aa7b56245c0a2810f748adc5fdd4