Date:   Tue, 30 Nov 2021 16:28:00 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Megha Dey <megha.dey@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, linux-pci@vger.kernel.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>,
        Dave Jiang <dave.jiang@intel.com>,
        Allen Hubbe <allenbh@gmail.com>, linux-ntb@googlegroups.com,
        linux-s390@vger.kernel.org, Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>, x86@kernel.org
Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Message-ID: <20211130202800.GE4670@nvidia.com>
References: <20211126230957.239391799@linutronix.de>
 <20211126232735.547996838@linutronix.de>
 <7daba0e2-73a3-4980-c3a5-a71f6b597b22@deltatee.com>
 <874k7ueldt.ffs@tglx>
 <6ba084d6-2b26-7c86-4526-8fcd3d921dfd@deltatee.com>
 <87ilwacwp8.ffs@tglx>
 <d6f13729-1b83-fa7d-3f0d-98d4e3f7a2aa@deltatee.com>
 <87v909bf2k.ffs@tglx>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87v909bf2k.ffs@tglx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?i79AMh5yCjE/aw1aYbeO9EILecIeHP50tYqPFWT4cJoGAqRRDousvpXnh02x?=
 =?us-ascii?Q?oQJhnYEdB7Xx5NzQxMcCi5SaJhhWh0TGVhNO0P8pXM1EXbzl2DioTavzdzR1?=
 =?us-ascii?Q?Bb6n2a8xBFDzQBZHFX19NQwgrx46ZSa7gPQnJqh9mQC39TXV49un12D4qtYF?=
 =?us-ascii?Q?yBdrtDxZ9lEcrUhGWkbIfTmVdWU1m53srtSz5OeQ2If5Anq/hY/8d1+DKj2O?=
 =?us-ascii?Q?CdS8LOsgFI0faiv70j8psDLbF9Keq9nXPxx+Fy4vou4PNmZ19eI90GKjKBLK?=
 =?us-ascii?Q?oCkujlL7CUlk/+Yu3UTSVaCCDGlyJAYsOWHpsCcJ35i3vu1o+a8Ss3mv+TmJ?=
 =?us-ascii?Q?KuIYxHnomJNfV65SeiFsNKn/qw+/dbNtJUs2TWD9C7j2a+suSoJ+DuJOcJXw?=
 =?us-ascii?Q?62oJ7XmxWrw3uPv8368xxWORlgISVr2yIYVvMAgBXAyTL23B0XlYQxbd7GRI?=
 =?us-ascii?Q?rUIU9tN/AOG3n9prkNU7cNWEQiKylYZ/faP7N1UUa72YH5M4XDxFIJlfBGlm?=
 =?us-ascii?Q?7Mg3i8VNyLiMMFhG0qdux+5vyW6H9ybPmnyBrgSrVFzx4YyIuvzv+cA2KH9Z?=
 =?us-ascii?Q?AKIc6NhnOHRDO3T7p1XN+V7qygIp0SjzJPdt+7IKYohle+2dN4+T/A58B2/1?=
 =?us-ascii?Q?h6lAw+qPWCvnBjBXZ1Z8Od4G/Z+6OXrgulZpm3JARYIrIAIDqc8j9Tdx1Yi+?=
 =?us-ascii?Q?BrLRRP5ki8VoRS9E2IK1SKSHspxf+DOd1gloWPk8JZ1km7Xh0rkOejcwpOXA?=
 =?us-ascii?Q?fP7YMYKKYei0sMWjx7ldaoC/vP8TS4JzejbzXHL7MREoGrNXdhRrnfEaKLxE?=
 =?us-ascii?Q?a+G5U+yVxnrxLBeQhKZD+yMCYvCfmAZXbk32rtPtdBBtftuFTcABQXee2MWf?=
 =?us-ascii?Q?5oYJOQCOO7NqQoPurc2mIOSTUQEic0G/y5ld7tbnbXlcexbgwSIzdnXzQVkK?=
 =?us-ascii?Q?tnVUZzP2SwmGFffMQlc/hmhnDx41iJhaD1glTSR7FMPafQPAiGrCTbrXvNIH?=
 =?us-ascii?Q?od9Ma2jXbk3LDEO+0BoOgCSwoJYO3U7H+E9LNzVY39rScJX73sbDdL8hkb41?=
 =?us-ascii?Q?iZLHJspvyHTBSwWVwgd6x68dVMt7OQXe5a+I5bwDIGUvKB69OBHzMMeUHlmn?=
 =?us-ascii?Q?KST+Ei7WlD8R56N28IEvd89dczffgwlyRPu1bycozPbhErBYtzu31KwNcFV+?=
 =?us-ascii?Q?kSi9d3AdCTkMnzvkZ41K0ViacuuOv6F5o8Hr7S7j+jxrdRdNamRv66BuncTf?=
 =?us-ascii?Q?AMq5eXyktQyLeq8j++JDFiBI1P1MWoVpjg0FOAu3UbFcIREhR517KRLC+pU7?=
 =?us-ascii?Q?4u3Y6wLfkUbG9lylaU5Ms9kmd3aXLrmisPlDzHE6JItpBEgd4VXr02NBwU7o?=
 =?us-ascii?Q?fHshWSGq6WToBZle/Yx6XYIgZyUF4ujx/JKKGCRjA0+UrQLMEyW+dUE/3REC?=
 =?us-ascii?Q?B+0P9j0RKELMGmrT9rMdXQpkiTyzADQdUEMo0brk6eg8P2cwUkGFObMsTJ+x?=
 =?us-ascii?Q?yXt11LeU/wCckQstLhjOnQncq5NZUWGV+FWj22u9Wa4i1O+DV/cPPH4QqE3M?=
 =?us-ascii?Q?DuT7jWRMz6UeVYNB3rI=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 63537dc7-528d-4a5b-5613-08d9b43fe7f2
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2021 20:28:01.8487
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ucpqPuptML+MhNc9/6VfMxvzAvM6pnkO+/XxCtjpSbCVwp0jjQoK4ZDjGInrO+95
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5240
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 30, 2021 at 08:48:03PM +0100, Thomas Gleixner wrote:
> Logan,
> 
> On Tue, Nov 30 2021 at 12:21, Logan Gunthorpe wrote:
> > On 2021-11-29 5:29 p.m., Thomas Gleixner wrote:
> >> I'm way too tired to come up with a proper solution for that, but that
> >> PCI_IRQ_VIRTUAL has to die ASAP.
> >
> > I'm willing to volunteer a bit of my time to clean this up, but I'd need
> > a bit more direction on what a proper solution would look like. The MSI
> > domain code is far from well documented nor is it easy to understand.
> 
> Fair enough. I'm struggling with finding time to document that properly.
> 
> I've not yet made my mind up what the best way forward for this is, but
> I have a few ideas which I want to explore deeper.

I may have lost the plot in all of these patches, but I thought the
direction was moving toward the msi_domain_alloc_irqs() approach IDXD
demo'd here:

https://lore.kernel.org/kvm/162164243591.261970.3439987543338120797.stgit@djiang5-desk3.ch.intel.com/

I'd expect all the descriptor handling code in drivers/ntb/msi.c to
get wrapped in an irq_chip instead of inserting a single-use callback
to the pci core code's implementation:

void __pci_write_msi_msg(struct msi_desc *entry, struct msi_msg *msg)
{
        if (entry->write_msi_msg)
                entry->write_msi_msg(entry, entry->write_msi_msg_data);

If this doesn't become an irq_chip what other way is there to properly
program the addr/data pair as drivers/ntb/msi.c is doing?

Jason