Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3951b23f-bafa-2979-f349-232c509a33fb@nvidia.com>
Date:   Sat, 15 Apr 2023 10:49:05 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Subject: Re: [PATCH v3 0/3] Increase the number of IRQ descriptors for
 SPARSEIRQ
Content-Language: en-US
To:     Thomas Gleixner <tglx@linutronix.de>, Marc Zyngier <maz@kernel.org>
Cc:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Michael Walle <michael@walle.cc>, linux-kernel@vger.kernel.org,
        Vikram Sethi <vsethi@nvidia.com>,
        Jason Sequeira <jsequeira@nvidia.com>
References: <20230410155721.3720991-1-sdonthineni@nvidia.com>
From:   Shanker Donthineni <sdonthineni@nvidia.com>
In-Reply-To: <20230410155721.3720991-1-sdonthineni@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cjAxZDg5ODdwQVBHY0hJbm16c3MrOWoxaU9VZFR1MnBMVDVWdTc0Tzg5dGd4?=
 =?utf-8?B?QTV1RDZKMDJYMFlvNmNLL2xNeHBnajFJKzdVWCtQckZpZ1V5NkVCcWR2QWhj?=
 =?utf-8?B?V1ArVWluaFpnMnY3Z2xWYWtCS2szWjZqbkZ1NmZnNVdZNFY0RlI1UVBVMWEr?=
 =?utf-8?B?YTBRTXhET3QwOVJZU1kzTDE0OGE2S3BYa3V3ZlM0VVVrQjNjNkQ5NU5laFQ1?=
 =?utf-8?B?eWd2aTEyOEhOTEVvSm9uNkUyNVBBd3ZPTkdpcUdmTU1kekE0dFhaQ1N4NVMz?=
 =?utf-8?B?R3JvVGR0ajdaTWh0Sk01L0NVZXg0MXlOdlI0Y1p2WTBwZXUxVmZkS0dHQWpl?=
 =?utf-8?B?VnZMZkdKTEZSMm0wcERKdHlNLzhSZjc2SWpXNzZGd2syOTJuU2tGcjA5TUx1?=
 =?utf-8?B?cmNNRjJhTi8vNzd6eFQvWHVJajNTNEdsRk4wdnBmb0xYWUIxTUxaNmtBYmRs?=
 =?utf-8?B?N09mQ0pyYjIybk1LNTRLVFFUdDRIZVhnZ0VDUTJJaFprY2U0QUVrTmtTdDlm?=
 =?utf-8?B?RWkrblFjL3BYZ3BBc3pHNlJWRHFVcDRPVFFJbkxOSUxCSWJkVlhYTFYvaDhO?=
 =?utf-8?B?Mmd4NEJoRTdzcSs2azZlOEN4VDdYR3BFeWtZVEFnZ2lEZEd0b2FCcWwwMkph?=
 =?utf-8?B?cVkvL0V4TzhkTTlQbTJXMXB4Ly8xSVRDSWE4NFltaUpOVjRZMTNndzQvR3BN?=
 =?utf-8?B?bnR1cTRmdTl4OXJKem4yclZQTHBWV2c1bUFvYjNKY3JvZlhBTUxKaDBGKytt?=
 =?utf-8?B?T0NyMUxPeng2QkFYZHBpZmFmd0ljcmxCS3hQSS95U1oybnpwTjE5VGdIMDJH?=
 =?utf-8?B?L2lLcUpnTHc0SnFqNWpwbW5haFh1ZmtBdFY1Y2wxZHFmeENuTFMwTndqVy9E?=
 =?utf-8?B?K0kvZzhkeTdyK0doUklWb2pTTzlNVEYzSlV5eGdwZ00wVHpNd2gxdWkxU3lI?=
 =?utf-8?B?eXhVY3VDOEpyTDVKR1JyV2xLK0tMek5pSEV0M0wyZzVkT3hidHFydnNXVjNZ?=
 =?utf-8?B?c2FBYnBJdHhwa1F2U1RNTTY2M2h2VEtseVJWTXlOazFNUERQUERaTHR1Vkxz?=
 =?utf-8?B?SnN3SWUvMDFlOU03blZPV0VBTGNpaVAwZno3MkdnbVA0S1Q2VDdNOVdLaFdN?=
 =?utf-8?B?eVlNV29YWFhSWFFDL2ZkcmtOTVJsTTAzMVU2Nko1WG04RjZQUUlHT3I2eVJT?=
 =?utf-8?B?WDVxQ3BNTUVSR0hyMExZRjZ5NU13ME93RkdHNTNyc2hZWGx6bnpoa3dhWmY0?=
 =?utf-8?B?bUI4aWsxYWFCWlBsZlYxMVRWbmRYK3ptWFZRdGw0MVlCWVVTQ0NvR2lnKzdR?=
 =?utf-8?B?MUxLSGtKdnJoaTRhaEJrbGNlbnF2WWxUMWlIcjBjUzRzKy84amNPKzV5M3FG?=
 =?utf-8?B?emlVNnU1MU9LRUErc0pzWW1IMmtPbkdVSDdGcXRGZHhtNm5nNnNWY1BwYzVj?=
 =?utf-8?B?VnQxcHkrUnBITXQ3S3JzMldiUGhSK3BpdzJqVjUvaEhndG45T09qQ2ZEZFJu?=
 =?utf-8?B?dlJyaGpFTUxSZnV5SUk3TkxRTXl4OGZnT1BjTWwzQi9mdXYvWGpsYWxXUU1K?=
 =?utf-8?B?aitQNk0weW9DTzNQcm85MEJuQXNTemhkOEVJOW9hTGtpZ0YwTWhzNk40dVls?=
 =?utf-8?B?ditEL1FIdm45UU9OaktucWM0YTlYblMrUkFreUpWM3UvaVVWVGpCS0N3TUt1?=
 =?utf-8?B?Y0pqZmJiTjdFRkwzTEdlZ1lwRXVpTENES2k1V0RrM0VHTnZzZTdDNW9KT2hr?=
 =?utf-8?B?TWozU2F0dVlLTy9jVlRaVTVOdHZGWWpqT25OQlBRSm5tblRnK2VGODZ3dnpU?=
 =?utf-8?B?RktUZG1Jekg3czR0RTh0NGRJbU9oZkE1T2FsZnVjenNIUGFmWTllZlNHdkRL?=
 =?utf-8?B?VzJqY0laRDFkOEo4djN4dW9DSWJwUVdscllOblQvVGtqM3pka1cvYlhBZENn?=
 =?utf-8?B?dmFnSVhxTFZxb1lxNHpWNCtYNE1FT3luQlpJVDNDRkMrZTQ1cVdxYVRRNHFB?=
 =?utf-8?B?bHFZYmNneTZxNzdBZXR1dWhWQUlQOW5IeTFsRlhWZ21tZ0FWYlVrRTdpajBk?=
 =?utf-8?B?ckdFNEtyOEtGYkwxZ29NakN2NmN3Z0Y4VWIyM01lZlNzVFdBcjJwWWwra0JS?=
 =?utf-8?Q?+fcg9LHPxsuyO4bvMAWLTYAV5?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2c720801-b10c-45f3-cdfb-08db3dc8f290
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB5762.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2023 15:49:07.6970
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: bPuGC4mgjtrdQUK1+2GeK5S3Wx4sYRFoYEmDM7QBKvkut2C5pUcSaaAZrtMJ2KRZ1aADsblkVWRnxESN9xZCdg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8711
X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello Thomas,

I wanted to update you that all the review comments have been resolved and
the necessary fixes for the maple tree have been integrated into the mainline.
If there are any outstanding tasks required to consider it for the upcoming
v6.4 release, please let me know.

Thanks,
Shanker

On 4/10/23 10:57, Shanker Donthineni wrote:
> The ARM64 architecture uses SPARSEIRQ with a default value of NR_IRQS,
> which is set to 64. This means that only 64+8192 IRQ descriptors are
> allowed, which may not be sufficient for modern ARM64 servers that
> have a large number of IO devices and GIC hardware that supports
> direct vSGI and vLPI injection features.
> 
> This limitation has caused issues when attempting to launch multiple
> virtual machines with GICv4.1 features, resulting in the error message
> 'kvm_err("VPE IRQ allocation failure\n")'. The root cause of this issue
> is the ~8K IRQ descriptor limit.
> 
> To address this issue, an initial proposal was made to define NR_IRQS
> to 2^19 for ARM64. However, Marc Zyngier suggested implementing a
> generic solution instead of hard-coded values. Thomas Gleixner advised
> to use the maple tree data structure and provided most of the necessary
> functions.
> 
> For more information, refer to the discussion thread at
> https://lore.kernel.org/linux-arm-kernel/20230104023738.1258925-1-sdonthineni@nvidia.com/.
> 
> This patch series converts the static memory allocation to dynamic using
> the maple tree, and increases the maximum number of IRQ descriptors to
> INT_MAX from NR_IRQS+8192. This change has been tested on an ARM64 server
> with CONFIG_SPARSE_IRQ=y, where 256 virtual machines were launched,
> creating a total of 128K+ IRQ descriptors, and IRQ injection was verified.
> 
> Changes in v3:
>    - Edited commit text
>    - Added a helper function irq_resend_init()
>    - Rebased to v6.3-rc6
> 
> Changes in v2:
>    - The patches have been updated to v6.3-rc5.
>    - The patches 2/5 and 4/5 have been removed as they are unnecessary.
>    - The review comments from Thomas have been addressed.
> 
> Shanker Donthineni (3):
>    genirq: Use hlist for managing resend handlers
>    genirq: Encapsulate sparse bitmap handling
>    genirq: Use the maple tree for IRQ descriptors management
> 
>   include/linux/irqdesc.h |  3 ++
>   kernel/irq/chip.c       |  1 +
>   kernel/irq/internals.h  |  6 ++--
>   kernel/irq/irqdesc.c    | 76 +++++++++++++++++++++++++----------------
>   kernel/irq/resend.c     | 47 ++++++++++++++++---------
>   5 files changed, 85 insertions(+), 48 deletions(-)
>