Received-SPF: pass (google.com: domain of linux-kernel+bounces-59140-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <02fe988e-2b42-9610-6ab5-bd17b0d9fb80@oracle.com>
Date: Fri, 9 Feb 2024 01:00:07 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [v3 0/3] KVM: irqchip: synchronize srcu only if needed
To: Yi Wang <up2wing@gmail.com>, x86@kernel.org, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, foxywang@tencent.com
References: <20240124113446.2977003-1-foxywang@tencent.com>
Content-Language: en-US
Cc: seanjc@google.com, pbonzini@redhat.com, mingo@redhat.com,
        tglx@linutronix.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        bp@alien8.de, wanpengli@tencent.com, oliver.upton@linux.dev,
        anup@brainfault.org, frankja@linux.ibm.com, imbrenda@linux.ibm.com,
        maz@kernel.org, atishp@atishpatra.org, borntraeger@linux.ibm.com
From: Dongli Zhang <dongli.zhang@oracle.com>
In-Reply-To: <20240124113446.2977003-1-foxywang@tencent.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?RzBIQTZmTS9GZTZrSThHeXlsT1EzdzRhWThyOXFRVTR6VzV6aFZTSVRpLzl5?=
 =?utf-8?B?MG5USVhiS1ZzczYrRWNuVXE3bEFoVHNpZWNBN2VjSjRzOGQ0M2lVZS81RUsr?=
 =?utf-8?B?clliTDhoWkhacVhYU2pTMHRaZDNqeEhCcXl0aEx2U05ZNkRiSGYvMTRkTlZS?=
 =?utf-8?B?LzRmRllBZ0VuWXpWN3JKTnJlVFFVUW8yM29EMFVCYTAxSTV3Q0ZueDZSdUpW?=
 =?utf-8?B?S0hPMmpoZnR3WXhPeXNKK2xyRGlRdHNHbTcvL2pVbHBTekwrbmo5RnlXWHo5?=
 =?utf-8?B?Y1YrRjBuUXI2Kzlnd0d2T1JaTGY3ZkVOZVdsQWdyYlhmRmg4ZERDcEtPQmp0?=
 =?utf-8?B?VmFlZ2ZHZk5yOUMvMEpQYzVrdUVwZ3duZ0NWNjVYbjlPc1U5QlV6cFFTZ0hn?=
 =?utf-8?B?bURqNkFVa0hjSE1iUnBvK3JYT3VrckU4NEhjZlB5anIzYWRMSHllbkJTZnhm?=
 =?utf-8?B?SmliQnRBVFdyVW95eUFraDBwSkc0YTM4THhWUHQzZm04cDJvZUhwU1lVeGpm?=
 =?utf-8?B?eFdud2JtQ2VydE1JbFRSWXpYSDN4N1hjMUVsUlBWb29OdCs2bmFXL0FBalQ2?=
 =?utf-8?B?N3VsYWJEaHh3dk1ueUxLejFqZ3pkMHpqaWV3MmtpNjlUYVA5R0dmTFc1ZzMw?=
 =?utf-8?B?eHIxeURCcmoxaU9WQ05saExvUURmQ3d6RzlScHRwNUlMUlEwS09aU1VIWGNk?=
 =?utf-8?B?Y0JIVWlmRkUwMm1NNXhha1lndmVUNCt3Skp5dWhrdnprbFA1ckpVK0JjVVhv?=
 =?utf-8?B?Mk1mVHlkYUVnR0cvNFBCMmZEWVNOUkpVUXowMWZqUlRneDdMU091SWtGTkFw?=
 =?utf-8?B?a3NMNHlnMERrMVlWNTZXS2J4cHl5QVQyZjkxQ0xvK3MwN215U3N5K1ltMzdw?=
 =?utf-8?B?VXBBNDVUcFcvSVYzYjR4L0pEMnRVYm01OVB0aWRYaEo0ZjcvSE53MnI4ODFC?=
 =?utf-8?B?UFdtZWp6Sk9qMDJEY0lmUUs5dm9iK0dkK3lsVEM4YUN4U2ZPMkdNTTVsdG9m?=
 =?utf-8?B?cjBYRmVrNjFMcE9PdTcyVGhhTC9EUVFiMUlHVlYrdk5aL3g1d3Z4TzN2OHpQ?=
 =?utf-8?B?Y0NQbStMVkdRM3dKVVFVZUh0NUdMQ0ZnbFh0a1duYmVEWG9YcU1vcC9vVFE2?=
 =?utf-8?B?WU53MEFQamxOcFlPQW53UTA4WVJob09ObnQrSWpRY0tOR0xjZHdGOFVhQmpP?=
 =?utf-8?B?TWVQaFBTNnp2ek9ndGw0UGxtcGNubjFoV1FzY2NkVUZTekloUzBnZnFFaDBk?=
 =?utf-8?B?TWVZd2p2WVFXN0FnNUI3Q1NnSy81Y2F6eStxV0JsMjFpcWhRMHRnLzlPRERP?=
 =?utf-8?B?cm56SXArQ1pCMW5rTHQ5QTBWSWpsRzQ5SkRhMGFtN2tEd3ltc202UnRzUm56?=
 =?utf-8?B?S205bW03d1RDc0ZKV2VMd2k2Y0FrT3FscE04MGpyU0VwOU5HekxCazhMR1RI?=
 =?utf-8?B?MjF5NlRaeFR5ZjVXR2lUK0twbVczdkZjTGpUSTc3dHE4ZmdsYWszMHZjWlZr?=
 =?utf-8?B?OXIxejl4ZHRzMXQrYjdtMlAwMU9reTNKUlRBTDJEdzBMMkhZaFBsMDVVRmRV?=
 =?utf-8?B?cVRqS2MyVGhWdTJyVTVjcTBSa1hnZnNkSHpscGtmeXFXeUZQS3FrZkpKV1ky?=
 =?utf-8?B?NXVLRmFzRGJwMHRqeUNFR0doVFM4REFIakxoU2REYjNlVzVKVGM3eHZ1aEty?=
 =?utf-8?B?VHQvSHdac3luTFh1R2FIazdiY2hEelJYREdXcDhiOTRTb29WV3BVbStadVFQ?=
 =?utf-8?B?bnozZ1lQY1NRblZSS3lCb1c2ejhxdEgrT0F6dXB5RktIaldHSlBXVmlxbGcz?=
 =?utf-8?B?R2I0bUdmelBUeUx2QWI1OUh1NW9WM3V2RHdQZXdKdkFFRktEWGlqcGlidDVH?=
 =?utf-8?B?M3lTVk9yV0lQbmVENk43RjhLNE9Md2EwT1N0aEs4dTdFenhzSXpZa2JsSWRx?=
 =?utf-8?B?eEpzcnpYZSs3Sk5JWmNNQ3ExaVQxdDZVaXZDRXkxMlU2NmViVVlzWmJGT084?=
 =?utf-8?B?d29KdmxEZUpWS2VTTUNYVFdwNVhTbEVzckRraThpRHUzRjFQdm9vV01yMEJ6?=
 =?utf-8?B?UkZKNXY2bERxeW82TWZQKytuUEFXOEIzN25ENUozaWVUUGtROHhnSGFHZnZX?=
 =?utf-8?B?c2RJMENwQnJaR1hubzBnMUZlNmgzVUZPeXo1cm15N3YyZE1sQ2crWFpOSHNN?=
 =?utf-8?B?UGc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	FEvv7HPHHIfUGE7T7aO68LRyZYmmugSfJYz5yt2CJ0jN0Rb5riDju7WGwjANK+GhyS7i5eNtpeaY4j5jrqSYXhfkKNzDHo0EzKvyRkZSkND92KUCHrENGzF9m9E2dP7v/E3crU45uK5AXD9p7jw0FikX520bFR1RCKbY9LhsTafd1GoG8foGfXQPBWb/SdK+Otu8QMeAlb+M46kfVU0gRH2izhsMiObdlAp2XVZmJbmRq4CxdKRR1iTxAfTNz/o+cjQ4av3dLOuVUhI6iXK9wb0WYsQc7RJwKaIAyKNi4qlEr7hlKnXyakKqrhFBCj9/9flEZ4E60bWUjTjioeuVuGkDZnERnMtL681ErqVy35UYBvjeBYVKBiGFXMc5sHzznR8F1GlJbSx3xXHyTxJ3nBtlfvABA0Xhx4t6HMyCCiXuaGl0dAZpvoVB6rM6M5Q7f5JNQ6UdMQLX9z5Ra4POGNu0/8W48UtK8Dq6jopHdSj9lTnvZtYamCmPc14UKEbsKVCR6zrVrT0sxz+/pahChOcMnySDscAi2g5sabZuFnn1yENSBNLhCkJx3R1nDvjTuQqmjUXF0te5SEL66TigT9KZs9HY7/CK/fCrkEyqusc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: beae8848-5d9c-43cb-771d-08dc294d846a
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 09:00:09.1678
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Wlr0Sj/vdFc2i3zLAqDJ1Tbiym6dQl+bwVvSc7AOcOkiaRu3s0hGOMr7rZOC8lznHHvmJU3YBTujgoak+wuevQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4237
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-09_06,2024-02-08_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 mlxscore=0
 adultscore=0 spamscore=0 malwarescore=0 mlxlogscore=857 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2402090063
X-Proofpoint-ORIG-GUID: cjgBRf6kHdfuyJ7Faw6nbpDZy7Wwp2lc
X-Proofpoint-GUID: cjgBRf6kHdfuyJ7Faw6nbpDZy7Wwp2lc

Hi Yi,

On 1/24/24 03:34, Yi Wang wrote:
> From: Yi Wang <foxywang@tencent.com>
> 
> We found that it may cost more than 20 milliseconds very accidentally
> to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms
> already.

Would you mind explaining the reason that the *number of VMs* matters, as
KVM_CAP_SPLIT_IRQCHIP is a per-VM cap?

Or it meant it is more likely to have some VM workload impacted by the
synchronize_srcu_expedited() as in prior discussion?

https://lore.kernel.org/kvm/CAN35MuSkQf0XmBZ5ZXGhcpUCGD-kKoyTv9G7ya4QVD1xiqOxLg@mail.gmail.com/

Thank you very much!

Dongli Zhang

> 
> The reason is that when vmm(qemu/CloudHypervisor) invokes
> KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and
> might_sleep and kworker of srcu may cost some delay during this period.
> One way makes sence is setup empty irq routing when creating vm and
> so that x86/s390 don't need to setup empty/dummy irq routing.
> 
> Note: I have no s390 machine so the s390 patch has not been tested.
> 
> Changelog:
> ----------
> v3:
>   - squash setup empty routing function and use of that into one commit
>   - drop the comment in s390 part
> 
> v2:
>   - setup empty irq routing in kvm_create_vm
>   - don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP
>   - don't setup irq routing in s390 KVM_CREATE_IRQCHIP
> 
> v1: https://urldefense.com/v3/__https://lore.kernel.org/kvm/20240112091128.3868059-1-foxywang@tencent.com/__;!!ACWV5N9M2RV99hQ!LjwKfBaGVl3u1l9YQSskg_1RU6278h2-fYnYLsoihF9i43aq73eIDqolGzOmeRvO8UlPreQHLqXEL1bAuw$ 
> 
> Yi Wang (3):
>   KVM: setup empty irq routing when create vm
>   KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP
>   KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
> 
>  arch/s390/kvm/kvm-s390.c |  9 +--------
>  arch/x86/kvm/irq.h       |  1 -
>  arch/x86/kvm/irq_comm.c  |  5 -----
>  arch/x86/kvm/x86.c       |  3 ---
>  include/linux/kvm_host.h |  1 +
>  virt/kvm/irqchip.c       | 19 +++++++++++++++++++
>  virt/kvm/kvm_main.c      |  4 ++++
>  7 files changed, 25 insertions(+), 17 deletions(-)
>