Date:   Mon, 29 Nov 2021 19:31:33 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Logan Gunthorpe <logang@deltatee.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Megha Dey <megha.dey@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, linux-pci@vger.kernel.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jon Mason <jdmason@kudzu.us>,
        Dave Jiang <dave.jiang@intel.com>,
        Allen Hubbe <allenbh@gmail.com>, linux-ntb@googlegroups.com,
        linux-s390@vger.kernel.org, Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>
Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Message-ID: <20211129233133.GA4670@nvidia.com>
References: <20211126230957.239391799@linutronix.de>
 <20211126232735.547996838@linutronix.de>
 <7daba0e2-73a3-4980-c3a5-a71f6b597b22@deltatee.com>
 <874k7ueldt.ffs@tglx>
 <6ba084d6-2b26-7c86-4526-8fcd3d921dfd@deltatee.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <6ba084d6-2b26-7c86-4526-8fcd3d921dfd@deltatee.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?d8zmZsrlevrOyId2WsxDhY8ZQyvEEmffio4ibRhoF49elGiDkrUTEk9ELLtz?=
 =?us-ascii?Q?94M0UpQqdFK7fyPr1TXCxZodb62l8jzhxPpAuy3RQOMY9NW+w6/9dV5RePzT?=
 =?us-ascii?Q?XVUjPC/4OpR9KyzJZ/KNdvh/LG2O8a3WcihresxeXFRONt0v1wY8I563aA8s?=
 =?us-ascii?Q?Vh7Pi0/gS15oNmz/toxMOjnO8K5NAfLWMyu3AfzKT6o/rV0uou+kYDpP258h?=
 =?us-ascii?Q?04DLXAzZleEIHq1lNldM8OJONR13fjJBgjdCTnhKMCil7oeZWJ4TSsTwMl5t?=
 =?us-ascii?Q?WK6whG9NajHudHXZR20n8Nhcf68OXACRSh27f8C7WJJ6oti969D6gDxktxXg?=
 =?us-ascii?Q?J0g0ecGuF4K36EPZ+l1l1LvBZMb4VBwaiYm9hnh03fgP/tEDrH12rD1YYpLr?=
 =?us-ascii?Q?8W9bKk5pSyLMmQUyvfIhL0/eoUBJmtm3EIiU49nD9kyojCHSUTg30MP9Phd7?=
 =?us-ascii?Q?Y5BWFZcVYXsk84o7lFunjVVynuoE167agf1Nyxs1D4zhPCEXCVOiFqwFTI3u?=
 =?us-ascii?Q?Cisv43lqhW18SM6tp9twWF3QwaYLK8VtCtXgr7UkJDjyqMko1zUDM2Z0W9M+?=
 =?us-ascii?Q?V1O/NoYNx1HMdnSapkEURWHmbrmthDvUSzZevrNY9GX9ZAGyyKFzoEri3ewU?=
 =?us-ascii?Q?YhaW8HNcfh+wm3xtbnN9J9EcO4id0Ksq9NG9Z2vR7MNgRrttlbHveoV4e8LC?=
 =?us-ascii?Q?XENrVKo8Fg8D8Twg056dHxlM2K4z4tgeLdKd+daFHxfuxh0I259l+q34ksVX?=
 =?us-ascii?Q?ukfiAmHhv2C3/6GYlX70AjDpHU38f50U7hovaWFhLmNpfNWou6UG+NpKosYU?=
 =?us-ascii?Q?XQsArJl9M4IOsc7InKb4RytaKNybkdSQw8krIbAYwAfHl8sfYB3TgP0KqVD/?=
 =?us-ascii?Q?probtVtyRYDjfbK+cT+3zz/1o3aAOUziLS2fH95LQ0Qqm8gdqwHIAlX1LCDH?=
 =?us-ascii?Q?5gxDP+XliD6zi00gwf88Y1k3JuMcohKctRzoJn2sxTcOBtT9SBzpvt3HUyak?=
 =?us-ascii?Q?vRY9YonuHUWL4HgQ+Y5uoDKXIH82kj3Qizlo1ATYM5bseKtnBnH6FNA8U2sS?=
 =?us-ascii?Q?dHv1z0CT3D5yWbfGhrYL1EGpPYB+OUoODd32mI1zxJIGpW1XquoOAlYcoLko?=
 =?us-ascii?Q?tUJjsO7NhFeeptBehBbipADoXZvVYB+nSZWQrdwbO2Ic3nIQKbGXesAJloa2?=
 =?us-ascii?Q?LemTMpsz20KcjiKe4vdxlE+uIzNPkUNN7i8gCTt4uZPFfCmkfeD7kBIJjEvc?=
 =?us-ascii?Q?NQYpMXRcNowTgNkWx0gK4AJbSlXyGjKReiAfq/67AoCmsSG1m80reydi5M/C?=
 =?us-ascii?Q?tQWxNn3qPi53fMIlgWAjhziO/6bSuWkJvN6SBfWvN40kb4gsxAqr6Fqwcn4X?=
 =?us-ascii?Q?QpCXUCXJXFSXlCestZUzpFpB5WOl1KmpSP8Mz7VvbXcwWDdVlXzKCbPrsUCP?=
 =?us-ascii?Q?igbR8G9c1jwQRvdBO4rmfvYZ4Pvsl/C4hyUWHbYlBvX+/1rtyXrtx3lqKr4n?=
 =?us-ascii?Q?0AdlMN7FtDH99Gp+wHdLOZpPL5tQSbg2AaKx5L6nHO2kWdhnOw8Y+XnnRI8F?=
 =?us-ascii?Q?XTl3zg9bHxz0phabpf4=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a120248d-c303-4c37-2baf-08d9b39061dc
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2021 23:31:34.9182
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GzI8/mSkYXICL2FAC4r6xn6fLgB0CyPpVBssGoN/KdITybX902bHKRE/skhT9anU
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5191
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Nov 29, 2021 at 03:27:20PM -0700, Logan Gunthorpe wrote:

> In most cases, the NTB code needs more interrupts than the hardware
> actually provides for in its MSI-X table. That's what PCI_IRQ_VIRTUAL is
> for: it allows the driver to request more interrupts than the hardware
> advertises (ie. pci_msix_vec_count()). These extra interrupts are
> created, but get flagged with msi_attrib.is_virtual which ensures
> functions that program the MSI-X table don't try to write past the end
> of the hardware's table.

AFAICT what you've described is what Intel is calling IMS in other
contexts.

IMS is fundamentally a way to control MSI interrupt descriptors that
are not accessed through PCI SIG compliant means. In this case the NTB
driver has to do its magic to relay the addr/data pairs to the real
MSI storage in the hidden devices.

PCI_IRQ_VIRTUAL should probably be fully replaced by the new dynamic
APIs in the fullness of time..

> Existing NTB hardware does already have what's called a doorbell which
> provides the same functionally as the above technique. However, existing
> hardware implementations of doorbells have significant latency and thus
> slow down performance substantially. Implementing the MSI interrupts as
> described above increased the performance of ntb_transport by more than
> three times[1].

Does the doorbell scheme allow as many interrupts?

Jason