Date:   Tue, 30 Nov 2021 20:17:48 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Megha Dey <megha.dey@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, linux-pci@vger.kernel.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>,
        Dave Jiang <dave.jiang@intel.com>,
        Allen Hubbe <allenbh@gmail.com>, linux-ntb@googlegroups.com,
        linux-s390@vger.kernel.org, Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>, x86@kernel.org
Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Message-ID: <20211201001748.GF4670@nvidia.com>
References: <20211126230957.239391799@linutronix.de>
 <20211126232735.547996838@linutronix.de>
 <7daba0e2-73a3-4980-c3a5-a71f6b597b22@deltatee.com>
 <874k7ueldt.ffs@tglx>
 <6ba084d6-2b26-7c86-4526-8fcd3d921dfd@deltatee.com>
 <87ilwacwp8.ffs@tglx>
 <d6f13729-1b83-fa7d-3f0d-98d4e3f7a2aa@deltatee.com>
 <87v909bf2k.ffs@tglx>
 <20211130202800.GE4670@nvidia.com>
 <87o861banv.ffs@tglx>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87o861banv.ffs@tglx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?aBMJYoXLNHr7c4GBS4Xay0ZJw56EJmXwIcQoWytXdhe5AlwHvpwML/iWG0en?=
 =?us-ascii?Q?p1IdcZYZdO1n30VChxnnG06WnakmquFY+QZEmd3TS37Op5YztaphWRIN3z1f?=
 =?us-ascii?Q?L2GDWNIgqYGwZ/DdMxU2RKVJbpydi3ItORG73e2Q/j/FikifcCcapIsU9tC6?=
 =?us-ascii?Q?Dq6w4DSudHzl06oOeM7bCwFjsSNikW57bEFMg8hnrNT0erQWa0JN0uvAoVXv?=
 =?us-ascii?Q?GkYPlBPE0RHKlUiiq+QPooJQAB1mOiHvUC1X6aA8/5bQJkWnvCi/LeY6awaO?=
 =?us-ascii?Q?a+omHzu/qipJyWclg6RL6D6QfhrQhhEDU5KbYkPQNRtERzHiNoMDlKBEdYtZ?=
 =?us-ascii?Q?SEHUv8KEIYseltU5a3q49XX/5rFTsXXcyDYtyAkXyKTIlUUjkllbX184f9NR?=
 =?us-ascii?Q?U5seszIJP9tZj2RWHHWXEwQoLDiKe0ddNt2KXxdionrtJVvvmuAR+R0Ux6H+?=
 =?us-ascii?Q?qhSqMrImiF1CokPw3coZcKsHxxicuYQ2v18ntjrygBQ3L5UCr4QDleOY6Y01?=
 =?us-ascii?Q?1L5qIpsaX3jyRGpHqswK/cx1dUXy5T4xk1wb6JMae3x85wA9cQcmuhPlpvxI?=
 =?us-ascii?Q?BbOMImT1FKRVmoxIhN8SE0I8BFDqWj89z47XOCkXKKJkU+PGwv7h6GjucsC3?=
 =?us-ascii?Q?eQ9Za6OL4Eey7HEN68zxqDJ4Up3OC/tjfzSxAhOrmfYrDy++m0fDBUJV8PQI?=
 =?us-ascii?Q?pS76Z7FWELi08PLGygKKpanx5EsMi8OHdaJvUdkMnjpV8CQtHNT+DQJ7D7PK?=
 =?us-ascii?Q?rlWYidPJMMKaL2mDGSybE21d1CTNzpjilDAF8G1FN9odfUXKicReGVyA44ow?=
 =?us-ascii?Q?ynWKeOYAJjUOCM5KDH5Dp4f1RPhNFSqReNqavc0l5WPmOcrCg/xPFWSSTpTW?=
 =?us-ascii?Q?ngZ+MD6LG3DulUwIh7gOtpTtpeVtNUY9lpZ7WBPrkbYoFzBnuwMHxw3dN2bK?=
 =?us-ascii?Q?DgWC4yDrwtjI4VYUn+/XsmiRlPKfHbOebk016SLh1ulo7LVhwXzvcCVf1AN7?=
 =?us-ascii?Q?td2jAUnFJL+8duAvDgie33lV2bFk7uL+gp5Q3Mafhlvl5r1CDf7WCm3YB0Lu?=
 =?us-ascii?Q?zhGdHotKSylfjkFpyKBQhV0aDTBkK0ZU6b6B3elLeHzInNzTclfVOo/SfkBT?=
 =?us-ascii?Q?j/T25WgSUu8bfhN1HFJGreqH8/S8Xf5opDDH1mFfZ0YButus0h9fg8TywLOn?=
 =?us-ascii?Q?lSWAlsOchOMGa3+p3+sgK67yUGKb++ouIrdqdENtl7j6V3qR0OGJCl7TjrsN?=
 =?us-ascii?Q?R5J4elnOZsej4CFqfQyOyO8Qx+5K8WyPVKu8nxU6NA3qn/wyvTlvrdZFFHaM?=
 =?us-ascii?Q?c3m/a/8m4u6j8IPlET1W8ni5xK2XNsSLH4tqz/oahww3eCCGl1j0JfsqVwmn?=
 =?us-ascii?Q?bn+ZErwBjkJUgRqqSCs3qEvSLxGureHR62pNIyptKpDwafnq001bcvXor47A?=
 =?us-ascii?Q?g2wjIgsFusXY4/u3SkuBx0I2NeN8qUnp9CTAS3qLSyX15cSVxXi8gTqLSmzR?=
 =?us-ascii?Q?Iou9nYyjX1Y07HS2suYqFdPQjO/C62LaCb22kZGMBqyA4pQBMGvzl9q8K/FR?=
 =?us-ascii?Q?Nz62llL/crGx1clqx1Y=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f3f6dc0b-fa4e-44e2-d3de-08d9b4600247
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2021 00:17:49.9719
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iDE/jBfkytVQm37L9L0hQG0mfQABBDPo/EbT0ND/QFLXRN1BcY5Xu5Sz4RyDFQZP
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5045
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 30, 2021 at 10:23:16PM +0100, Thomas Gleixner wrote:

> > If this doesn't become an irq_chip what other way is there to properly
> > program the addr/data pair as drivers/ntb/msi.c is doing?
> 
> That's not the question. This surely will be a separate irq chip and a
> separate irqdomain.

OK
 
> The real problem is where to store the MSI descriptors because the PCI
> device has its own real PCI/MSI-X interrupts which means it still shares
> the storage space.

Er.. I never realized that just looking at the patches :|

That is relevant to all real "IMS" users. IDXD escaped this because
it, IMHO, wrongly used the mdev with the IRQ layer. The mdev is purely
a messy artifact of VFIO, it should not be required to make the IRQ
layers work.

I don't think it makes sense that the msi_desc would point to a mdev,
the iommu layer consumes the msi_desc_to_dev(), it really should point
to the physical device that originates the message with a proper
iommu ops/data/etc.

> I'm currently tending to partition the index space in the xarray:
> 
>  0x00000000 - 0x0000ffff          PCI/MSI-X
>  0x00010000 - 0x0001ffff          NTB

It is OK, with some xarray work it can be range allocating & reserving
so that the msi_domain_alloc_irqs() flows can carve out chunks of the
number space..

Another view is the msi_domain_alloc_irqs() flows should have their
own xarrays..

> which is feasible now with the range modifications and way simpler to do
> with xarray than with the linked list.

Indeed!

Regards,
Jason