Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     "Gao, Chao" <chao.gao@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Kohler, Jon" <jon@nutanix.com>
CC:     "Christopherson,, Sean" <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "x86@kernel.org" <x86@kernel.org>, "H. Peter Anvin" <hpa@zytor.com>
Subject: RE: [RFC v2] KVM: x86/vmx: Suppress posted interrupt notification
 when CPU is in host
Thread-Topic: [RFC v2] KVM: x86/vmx: Suppress posted interrupt notification
 when CPU is in host
Thread-Index: AQHYzyqiVAqlk3tAREG5c6Y0U9+i2630iVbQ
Date:   Wed, 28 Sep 2022 08:29:48 +0000
Message-ID: <BL1PR11MB52712631CBBED273F5E4E1828C549@BL1PR11MB5271.namprd11.prod.outlook.com>
References: <20220923085806.384344-1-chao.gao@intel.com>
In-Reply-To: <20220923085806.384344-1-chao.gao@intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?hIhzpIQuElRboCB//6q+OZo3oachgNGvVf0/9+bLf6TfMSkrSe0V2mYBhgQY?=
 =?us-ascii?Q?07Kcd+DKcEV0fFLHC0FOPg/d/s+8h07x7Tk2Jj2kFLHfL5J1eSU3UNoENtZi?=
 =?us-ascii?Q?+LnpVoQRLiQZxVcJx4qbNBAjzdKM5S2Fe1K3mlTlcMrhwG54biXW7g8BlFRn?=
 =?us-ascii?Q?LM4hDcxtyTg8kxd2URKOum2UoUrTLq5aheu+xaYxlaDEsB9AQgWSqIjreqgx?=
 =?us-ascii?Q?WYIhZHbmvHc24GE/XeibIB4B5pobvaCDXnQt6oJcaA7254uxH5qeN9quOadw?=
 =?us-ascii?Q?wjDJ+195z56ZDUT3N+4rODLBJ/nPuRfCxCdP4IoTd1LmUGQ6FBanBkIzeCZk?=
 =?us-ascii?Q?PElHkY0SBlAjGo3fohnltGa9U7UEO//4EzYXYrRNCcB1XylzDsHmZ5q6I9F0?=
 =?us-ascii?Q?DWl7l6pDLIIryodK5F/ln4zt+ul0exEA8gPujbDC6X6VrRwaQVhmwSNHmTRl?=
 =?us-ascii?Q?QEjZb3Tk4VVYVUBpqhm+K0vm+6xSdZCXS5TAR+M+SUIGf5/DQWfTgycDFoVj?=
 =?us-ascii?Q?n7HxTdrtUrRKwWH1VkTGB1jeqQK/zvhZ6gftnCm0R1Dnhg1IptQHMpOrfjuh?=
 =?us-ascii?Q?YkhTgJZfNSK0kiQJdd6gviIUKsEElaUrOZT7CUDGkroDRFPqa7wuQUJ6XmJG?=
 =?us-ascii?Q?BLl+60h6oDcBHQzgqxPeX8Rc9ib9ZLaz6DySEmYm3XNZmYPiTAxWY4LDXjVY?=
 =?us-ascii?Q?8Etdg4Fyj/rEAcHXTri8yycpLoPQ01DYs2x5O0bf65KHmdoVR9JrO5OuoKoS?=
 =?us-ascii?Q?wgeH7GgxdFO1m+JVvKqcANmWVkk2hsueQopEydhOUMZIp3qnBFg4nuVNX+5b?=
 =?us-ascii?Q?LDHwIgyNCDoPLexhXtwlAHIHYhMG6OIcFoTN5pnhN561r7OSeMtLggPnqh+o?=
 =?us-ascii?Q?WPY5nh0FcfAMCs3bhR1B6jzD3loaucCV2x84CRSrSFXj3y57rtxCu6G8Kkcw?=
 =?us-ascii?Q?/WC/HQ20xpYmjBcb7ctkIGl59252pYAa24yYhxRBOsalpz98oEXUevvxulfy?=
 =?us-ascii?Q?x/9Ds26herqCKupnnvuyuBq+yUrkRHHfyiiwNzkbDhANajdwFXjMxAUPpwyw?=
 =?us-ascii?Q?yE1F6jMWjItbkZ1Rspjdz5zNzfzZlygJcH04y+EDVj7MjgWVJ2KJir61jvRe?=
 =?us-ascii?Q?wBcsp0ivP6kGOuY/QY2xqwWP/1SwiX1GkXmlJnS5pyK544YZDvD4YD6Z8Uhm?=
 =?us-ascii?Q?zoof+6Fo8gtDDhXOGlHiri+LjP+kEjJgebi6oqoC8FZ2a0jxHIOWcSL30WGX?=
 =?us-ascii?Q?OOAzQKngSE1ISlX5PCrk6KCkiszC0pPLs8+QEi6xSIz58QlbL9VdGDmiq7Io?=
 =?us-ascii?Q?D4ZYfPrCOpmavnWOluXQBpN0PJYMOerBfCHyqTAiT6yRkbqc0DgfvSME92fp?=
 =?us-ascii?Q?BGztJ5DrR26z0WAx3lnCtn5oG9Za6hiTG7cCdRYENjJeE5FSpCVRLZv9Zwzb?=
 =?us-ascii?Q?6dmkZ9OXGKlSkrz3oX4gt59T01l63ij0uZzTo/a5W7ZROrmEUqo8GjquF0Uh?=
 =?us-ascii?Q?9AT3uNW/DPubkwA3A0hwRMKpMhcnuuZbdkIead3SJjSnsvPKX08PKCP+cb0v?=
 =?us-ascii?Q?9H/v6AIB2PbYtHSWMCwwvK0YNDwiotCHRhjfzIdQ?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5271.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ee44e2b3-e190-469d-68d1-08daa12b9af1
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Sep 2022 08:29:48.1212
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: eIZBJndAY3WeIsiBaOIg/V15qwH3UZmWJHwFck00fMB3bzpKFp6JDhsn20fH+vD4ipDHSJa5aMToq3mAKN6juA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7046
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Gao, Chao <chao.gao@intel.com>
> Sent: Friday, September 23, 2022 4:58 PM
>=20
> PIN (Posted interrupt notification) is useless to host as KVM always sync=
s
> pending guest interrupts in PID to guest's vAPIC before each VM entry. In
> fact, Sending PINs to a CPU running in host will lead to additional
> overhead due to interrupt handling.
>=20
> Currently, software path, vmx_deliver_posted_interrupt(), is optimized to
> issue PINs only if target vCPU is in IN_GUEST_MODE. But hardware paths
> (VT-d and Intel IPI virtualization) aren't optimized.
>=20
> Set PID.SN right after VM exits and clear it before VM entry to minimize
> the chance of hardware issuing PINs to a CPU when it's in host.
>=20
> Also honour PID.SN bit in vmx_deliver_posted_interrupt().
>=20
> Opportunistically clean up vmx_vcpu_pi_put(); when a vCPU is preempted,
> it is pointless to update PID.NV to wakeup vector since notification is
> anyway suppressed. And since PID.SN should be already set for running
> vCPUs, so, don't set it again for preempted vCPUs.
>=20
> When IPI virtualization is enabled, this patch increases "perf bench" [*]
> by 6.56%, and PIN count in 1 second drops from tens of thousands to
> hundreds. But cpuid loop test shows this patch causes 1.58% overhead in
> VM-exit round-trip latency.
>=20
> [*] test cmd: perf bench sched pipe -T. Note that we change the source
> code to pin two threads to two different vCPUs so that it can reproduce
> stable results.
>=20
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> ---
> RFC: I am not sure whether the benefits outweighs the extra VM-exit cost.
>=20
> Changes in v2 (addressed comments from Kevin):
> - measure/estimate the impact to non-IPC-intensive cases
> - don't tie PID.SN to vcpu->mode. Instead, clear PID.SN
>   right before VM-entry and set it after VM-exit.

One correction here. My comment in v1 [1] was actually close to Sean's
suggestion, i.e. opposite to above description:

--
  I wonder whether it makes more sense to have 'sn' closely sync-ed
  with vcpu->mode, e.g. having a kvm_x86_set_vcpu_mode() ops
  to translate vcpu->mode into vmx/svm specific hardware bits like
  'sn' here. Then call it in common place when vcpu->mode is changed.
--

[1] https://lore.kernel.org/lkml/BN9PR11MB52766B74ADFBAEC0AA205E298CB39@BN9=
PR11MB5276.namprd11.prod.outlook.com/