Received-SPF: pass (google.com: domain of linux-kernel+bounces-69222-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Fri, 16 Feb 2024 15:04:26 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Christoph Hellwig <hch@infradead.org>,
	Saeed Mahameed <saeed@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Leon Romanovsky <leonro@nvidia.com>, Jiri Pirko <jiri@nvidia.com>,
	Leonid Bloch <lbloch@nvidia.com>, Itay Avraham <itayavr@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>,
	David Ahern <dsahern@kernel.org>,
	Aron Silverton <aron.silverton@oracle.com>,
	andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org,
	netdev@vger.kernel.org
Subject: Re: [PATCH V4 0/5] mlx5 ConnectX control misc driver
Message-ID: <20240216190426.GG13330@nvidia.com>
References: <20240207072435.14182-1-saeed@kernel.org>
 <Zcx53N8lQjkpEu94@infradead.org>
 <20240214074832.713ca16a@kernel.org>
 <20240215132138.GK1088888@nvidia.com>
 <20240215171013.60566d7b@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240215171013.60566d7b@kernel.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?gdjmT7mBZ9TBwW+rx0xPaRqnY+iYoAjmn63sRMQVV069No/IzzBQioLmOCSP?=
 =?us-ascii?Q?yzwDvFkkHUWeDG2jbf90YiFbUdwqy0c3PKWd/9+eKdj0EcKdTyAQdm9uLnLz?=
 =?us-ascii?Q?6pdiKUzaK9KLT2nhhF6L++08KmGHz4ED3TLGGwbSxOIy5qvPjdoMf2JBQLFX?=
 =?us-ascii?Q?m84M897xHcyhkb3GvYYKHO5SujSVjXA8uoR+/aslu3dcTU0SKND67odjAnAR?=
 =?us-ascii?Q?XWX+uY6ZLce/BfMnE8SyhWLgUaCQRiHtIqFlBNJiP3KpRVBwHYggrVBLZnlM?=
 =?us-ascii?Q?/kOuV4Yn5mWWO1F/lUWhjpHqAVoVb/jrBMtV4En7CWw96az6Shg7UYaGPFh1?=
 =?us-ascii?Q?c3svh+IpId4hNqPxoJR92GqVm9ysgWW9OC6ssz1shYSl6TOZNB4TdbqcWG9B?=
 =?us-ascii?Q?Vp13Q4GytPIUAVtpnAzP4iKb5PyXzwH2VXToXtn2T0nwcNgLQYFF9PcRGxTm?=
 =?us-ascii?Q?WQmRUH88aIVXUpjmwkirxD/YH4pHVHyMfAxlncVLhd9dAJBkDEd/E2wPQRXq?=
 =?us-ascii?Q?dQOw3/SzvHQ3lG0er3hk7aSguxCO0nEzAPNCCo48UnsqMqyoUoVgCeW5G3Bs?=
 =?us-ascii?Q?qzW6UgCxLGtFYQc/KOOBsp7Lkj9VG5iHpJqbaJfkntPQhy3MnXiZFIOncky4?=
 =?us-ascii?Q?H2I1uq93MAzRe2D7jQh3Sn9/vskgBaXLqfUYIFi3hKS784kOIl/vOcbQQw0Y?=
 =?us-ascii?Q?WtroKCSLGpEcgbk2r9QGOP17qVpYz3+gvARjI6NEHoni2i2BZG72iErsIHs3?=
 =?us-ascii?Q?y8b4wYQYAdCLJrFeTxEEtae0EHt+XNxWEBfrOb+oJ1ZvYNCYTbJt9x9WeGO7?=
 =?us-ascii?Q?bAJPcGpK1U+E3EiKiCc15gf09wqUEbiTwIhtK+fAAjsQwBDdyWoWt9Zw2o6N?=
 =?us-ascii?Q?/gEeV7u5+VSAFvAZwF3UB+hHwdrpIuT++X+Wz+zUuS5qze4AxDMNa3I/SWF5?=
 =?us-ascii?Q?PRSXhAfuzwTtt30TEeCZhsYpaRBzgWn2hYCHs7AzBLJ6iOmiYiiC828a6InF?=
 =?us-ascii?Q?ETPHzvKqvNhPv1GmwaslQv5K4LXQg77MRmEC6tvROp5vW3XbKRCd8bvXloTR?=
 =?us-ascii?Q?n0rlTEKeN7RIWVG5BNTdICaqajfq5SjTD9xIqvk0lVs1/wcKfmuK1KXH8bho?=
 =?us-ascii?Q?LooLMYKF8yfetAiK+3+6PmufAWvaLULBDCJE+1X0u+102Pvod6B/z02smalv?=
 =?us-ascii?Q?BfHXykwcp/ZaYnIF/wR8hs97/5R5ThbFELGeiOKe5awP2BcMbBof2XIbaTHX?=
 =?us-ascii?Q?sw4dXICOXWQ7wNzW1/SWDG32iUDfo3yinMLA/Tu9hD9MXW9ypTk7uhtq0n3b?=
 =?us-ascii?Q?SVY05/b4eBDBaojz2F8YBXeeiNsoTwbkQL//niFBfe6N+tLJPELTnCA+eNyg?=
 =?us-ascii?Q?KvaxummuvyAEAacv8Qfaq+BnaPcQAh/gTyIGahL+5EqUM4CyHDvOezBrYybV?=
 =?us-ascii?Q?LflxPvS0rJwmtxsrwYehZxvy77NG0MxIYzhjryeQla3rNjSbbfsfjl19c+7E?=
 =?us-ascii?Q?8D0E6jqo1uBmdL7MAe2fF7ODpnt0zBbyqe9PGXvEvEwO+be80aQQgNe7R9vd?=
 =?us-ascii?Q?/ELBDU9D+fWIsIxVDMkC7DP1766fdMkAaqBUIeEp?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 263870e2-ec90-42a3-4046-08dc2f221925
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2024 19:04:27.9038
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nMfciWipur4LS2gBd1LJK0huN4NGaqiLIb47IUerKyvFbrC/+9Czyk+4Bx7BpivS
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6344

On Thu, Feb 15, 2024 at 05:10:13PM -0800, Jakub Kicinski wrote:
> On Thu, 15 Feb 2024 09:21:38 -0400 Jason Gunthorpe wrote:
> > On Wed, Feb 14, 2024 at 07:48:32AM -0800, Jakub Kicinski wrote:
> > > On Wed, 14 Feb 2024 00:29:16 -0800 Christoph Hellwig wrote:  
> > > > With my busy kernel contributor head on I have to voice my
> > > > dissatisfaction with the subsystem maintainer overreach that's causing
> > > > the troubles here.   
> > > 
> > > Overreach is unfortunate, I'd love to say "please do merge it as part 
> > > of RDMA". You probably don't trust my opinion but Jason admitted himself
> > > this is primarily for RDMA.  
> > 
> > "admitted"? You make it sound like a crime. I've been very clear on
> > this need from the RDMA community since the first posting.
> 
> Sorry, unintentional :) Maybe it's a misunderstanding but my impression
> was that at least Saeed was trying hard to make this driver a common
> one, not just for RDMA.

The hardware is common, this is a driver to talk to the shared FW. I
don't know how you'd do just one when netdev is effectively an RDMA
application running in the kernel, from the perspective of the FW.

There is no real line between these things beyond the artificial one
we have created in the kernel.

> > > The problem is that some RDMA stuff is built really closely on TCP,  
> > 
> > Huh? Since when? Are you talking about soft-iwarp? That is a reasearch
> > project and Bernard is very responsive, if you have issues ask him and
> > he will help.
> > 
> > Otherwise the actual HW devices are not entangled with netdev TCP, the
> > few iWarp devices have their own TCP implementation, in accordance
> > with what the IETF standardized.
> 
> There are some things I know either from work or otherwise told me 
> in confidence which I can't share. This is very frustrating for
> me, and probably for you, sorry :(

Well, all I can say is I know of no forthcoming RDMA things with any
different relationship to TCP. I think if someone wants to more TCP
they will have a hard time, and I'm not inclined to seriously help
anyone get more TCP into RDMA. iWarp is trouble enough already.

> > I seem to recall you saying RDMA shouldn't call any netdev APIs at
> > all. We were unable to agree on where to build the fence for this
> > reason.
> 
> Last time you were calling into the IPsec stack right? It's not just 
> a basic API. IDK how to draw a line, definitely open to suggestions!

I thought the two halfs of the mlx5 driver were co-ordinating their
usage of the shared HW around the ipsec configuration pushed into the
device by netdev xfrm.

> > > Ah, and I presume they may also want it for their DOCA products. 
> > > So 80% RDMA, 15% DOCA, 5% the rest is my guess.  
> > 
> > I don't know all details about DOCA, but what I know about runs over
> > RDMA.
> 
> Well, since you're an RDMA person that's not really saying much
> about existence of other parts.

<shrug> why does DOCA matter? Should we have not done io_uring because
Oracle might use it? Besides, from what I know about DOCA it is almost
all data plane stuff and RDMA fully covers that already..

Regards,
Jason