Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 16 Nov 2022 14:36:05 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     LKML <linux-kernel@vger.kernel.org>, x86@kernel.org,
        Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        linux-pci@vger.kernel.org, Bjorn Helgaas <bhelgaas@google.com>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Marc Zyngier <maz@kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Dave Jiang <dave.jiang@intel.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        Ashok Raj <ashok.raj@intel.com>, Jon Mason <jdmason@kudzu.us>,
        Allen Hubbe <allenbh@gmail.com>,
        "Ahmed S. Darwish" <darwi@linutronix.de>,
        Reinette Chatre <reinette.chatre@intel.com>
Subject: Re: [patch 08/20] genirq/msi: Make MSI descriptor iterators device
 domain aware
Message-ID: <Y3UtlTwIka6n0RuT@nvidia.com>
References: <20221111131813.914374272@linutronix.de>
 <20221111132706.500733944@linutronix.de>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20221111132706.500733944@linutronix.de>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?rSq3xpATKlbggA1STQs11Rvt5xFhWUhs8fLyFuweN29KTuV2lRSrIvPWrPcD?=
 =?us-ascii?Q?bZj7MaaMAu13f/jhhpd1XnvB8Bujb3ufUVEgpqEobr1/XjUBb7+cukK0hNEx?=
 =?us-ascii?Q?gdOlzcxIl/Xt0E4sb9a+H9fSVBOZ1Jr8DiAi620g1Xm6/LJ7V/zNFg58H6+R?=
 =?us-ascii?Q?BpxIz7D2ySsHvMnl6FPm8YBjFcOMusemr9r9drPbmMoqYSrZZOI2tRpFsx7r?=
 =?us-ascii?Q?nkbuyUn/pijAjWOEZNjMF8ig9fdVHVIDU8CweZFV1rnOWuS5o85Ml4ZaJ3hp?=
 =?us-ascii?Q?/yQ+0tEhuiRZEj6Scm0lAgF/PdrjZ1EDQatVgP454ylumKIj6WyWOkwTmoe/?=
 =?us-ascii?Q?VMsbu0Y7Yvq6VH1AFQIAKDpAHPsYbtVdh/S+C9JyiFxUL/JMAObtvX1DxiOD?=
 =?us-ascii?Q?SfGV48t2Wrj/3a0Lz+siT6XLwI8MEHXoLxttu/HYoovj8Rkyf2gkXhLXFbWS?=
 =?us-ascii?Q?bf4EatdiWTkaBaNXSJz7zGoGjxrowfCpDde2h+LaZMqFtdtgMBWKGpsDjU8a?=
 =?us-ascii?Q?/63MbC1dDhsPkFS8lZh2tJM0JwqWSsPDEucd56OmkTe/WGb5RiXaG4aEzBgr?=
 =?us-ascii?Q?+TP/e7VfaGELz9v4JIeJZkrF8UGD3uSEMBz1geK03wmKmAIVlWSTHlULPbpf?=
 =?us-ascii?Q?M+4fY9BtsZv7JpXpoT8gfGdeQX14xLxjj+lc0zPs3IdrX+os53r1ha/63OyJ?=
 =?us-ascii?Q?WFKqMH4XM0ASHVrGuXD9yNU590AM3gbcp7ujX38Rxh24uFp26de8l3nQ5jJ8?=
 =?us-ascii?Q?l4/YXLsK326VgmMAonLue4Yf1YHE+N6OIu0AnYqRLtaSAav0S+o8a4H80WUS?=
 =?us-ascii?Q?7nwNJRLd8NoNFpYg6s8QAkB/kEZC2KS6GeRyqxQLLStxBW96rOBP6UcUk/B7?=
 =?us-ascii?Q?Ek4BU8clTOJiCZ3XSx5vzPNOx4wAZYFSnTIWhw2C4w6e72HdUKvyraTqv7vD?=
 =?us-ascii?Q?gEgBi8BKEUO6qoKmfItn1JnIDsJDmNjuMtB2/bbrtPh9+v7tBX+gOjDsloyt?=
 =?us-ascii?Q?Lxnn994rnnx1ELXJDYTVpcTsKwpQBSOEV6KwRC4Pc7eCxgV5CM/39Karb/3Y?=
 =?us-ascii?Q?BUKEKJnUbO8BZVS3n6iXZDT9YIUdG416/xWYDWuVzbihorr6C5PqsDOhByl1?=
 =?us-ascii?Q?q8fSmDiNL7SfRgrH2IDgO7kIeH3uFSItSKZ5+NYewqJu+PnkefMB84u7X/N3?=
 =?us-ascii?Q?P5qtK37cJ4BhwoPsugpwOx4hwMCR2YR4JHqiC7hfq4hUN59PS5csAKgmWFWy?=
 =?us-ascii?Q?xs1xya57/o8cbPzZ2BPha+vwp7JWTspRH6MaWGp7M8g56DhjSbjMSTTXoy26?=
 =?us-ascii?Q?zUOdFsZIBVtFXtIuJFPw8XUXV6l9t9+VJmqFVRA/pAb6fHIyfR0id2IhBgjW?=
 =?us-ascii?Q?0SoJCXEGfPOIga90AEFqmwqFibu31AL/P8390FWQbUk4DAdWdZXjcuTgllw5?=
 =?us-ascii?Q?hasDqgrFbZ3DOO7rgNK9+rJt8yyzb/ByHYiaGbnr3EdpDPabvt9UYlf+rmXk?=
 =?us-ascii?Q?9jH7B31KKyYu4EN8RZC5Xh/bE5ofG44I3Vtp/47hcirTs9sTqkJL6pcO6ULQ?=
 =?us-ascii?Q?GS5He+FxGDtsgIT5gHtxfkPPBTlplZpMhLx2SnxN?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ba09517e-1320-40f7-8466-08dac8016d71
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Nov 2022 18:36:08.5044
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ej+kdpjGroHFyUUOKiwPeXwoXEdDVGbQws+NJCh3RGuzy6uqPiauUwgA3MCejHdC
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7431
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Nov 11, 2022 at 02:56:50PM +0100, Thomas Gleixner wrote:
> To support multiple MSI interrupt domains per device it is necessary to
> segment the xarray MSI descriptor storage. Each domain gets up to
> MSI_MAX_INDEX entries.

This kinds of suggests that the new per-device MSI domains should hold
this storage instead of per-device xarray?

I suppose the reason to avoid this is because alot of the driver
facing API is now built on vector index numbers that index this
xarray?

But on the other hand can we just say drivers using multiple domains
are "new" and they should use some new style pointer based interface
so we don't have to have arrays of things?

At least, I'd like to understand a bit better the motivation for using
a domain ID instead of a pointer.

It feels like we are baking in several hard coded limits with this
choice

> +static int msi_get_domain_base_index(struct device *dev, unsigned int domid)
> +{
> +	lockdep_assert_held(&dev->msi.data->mutex);
> +
> +	if (WARN_ON_ONCE(domid >= MSI_MAX_DEVICE_IRQDOMAINS))
> +		return -ENODEV;
> +
> +	if (WARN_ON_ONCE(!dev->msi.data->__irqdomains[domid]))
> +		return -ENODEV;
> +
> +	return domid * MSI_XA_DOMAIN_SIZE;
> +}
> +
> +
>  /**

Extra new line

Jason