Received-SPF: pass (google.com: domain of linux-kernel+bounces-68779-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 16 Feb 2024 11:05:46 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Christoph Hellwig <hch@infradead.org>,
	Saeed Mahameed <saeed@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Leon Romanovsky <leonro@nvidia.com>, Jiri Pirko <jiri@nvidia.com>,
	Leonid Bloch <lbloch@nvidia.com>, Itay Avraham <itayavr@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>,
	David Ahern <dsahern@kernel.org>,
	Aron Silverton <aron.silverton@oracle.com>,
	andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org,
	netdev@vger.kernel.org
Subject: Re: [PATCH V4 0/5] mlx5 ConnectX control misc driver
Message-ID: <20240216150546.GD13330@nvidia.com>
References: <20240207072435.14182-1-saeed@kernel.org>
 <Zcx53N8lQjkpEu94@infradead.org>
 <20240214074832.713ca16a@kernel.org>
 <Zc22mMN2ovCadgRY@infradead.org>
 <20240215170046.58d565ef@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240215170046.58d565ef@kernel.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?opR+aDskhQKr62SSuzlC1QdeGBRmBR/Ru8kASTB0/rgEvI+kvHta/clRSsRo?=
 =?us-ascii?Q?Ov7Ogxes72MARp7Ftvlg2v2QlYnlM9Tx5sSxiGCg3R3KzmwS8gZ+ufPHMqUA?=
 =?us-ascii?Q?1TKoNDHlJhL07hwcNdAxp5dMWsHfBPAdfQtp0P4w/Qo8HAULKJ6t6NsqyKCL?=
 =?us-ascii?Q?m/jgu/zBFH14R+AkL2oA2c9KuFfqcCnRzOYb2oR4BzPHSleRsLlkz5Q8yUrZ?=
 =?us-ascii?Q?8RrpS8Fx5Rf8FgWQFFX+Z4SW8GCBtNdCjuU7qSsarTQkxmS8NUJ3ZlPpBTvI?=
 =?us-ascii?Q?ZINxGvRV9szkuAG/riCSfWgHNMGBv6sDsZt0NPoxC7tYvIcqf/4f7FWy+sNc?=
 =?us-ascii?Q?+Yv07fSXPQ0E024z8Ytx38J1MHgyA2+U0WYsKKIVJPFFy2uzy2m0vy4hTEBn?=
 =?us-ascii?Q?nJQ6QMTXT/73KfhFdnnvMoO7gn1CI1A+FSlc8n763KP/ajNDAoSEl+ZLqkxi?=
 =?us-ascii?Q?7+1Kk1mnPZIIhlJEJ595INtbmwalAtVJ+m1VpYAdcMH97qdJBjRy5aosYrTu?=
 =?us-ascii?Q?4x6LreeTNRS8ohEiLkqGct37x3/SGyev/OMui/WpPIcv8qrMXZawsf0A0zVr?=
 =?us-ascii?Q?p1VEyqrw5Cn34LJWwR94xkJsQGa8mkBqXpzJZEBfT+6yl7/4KU1HdBaZkw1M?=
 =?us-ascii?Q?tfb3c0qpVpm3HCPX7Oo4xNsbIji+z1vPnBi3aA3KmzTwnAhMgJ07CG/THNcq?=
 =?us-ascii?Q?eMgHCD42mvj8GuSUukmrtLxlsiwTI+tKdCT0aYu4lVREeewpvTUQzThZxHo7?=
 =?us-ascii?Q?v+Vh+5apfO3PArqHLUaxo9/XJnHImjxygnGmWK8Eh5sGnYomAplfVvhTq/d/?=
 =?us-ascii?Q?g+IwyuJ/Q7rNkItYIrWsopbiNATt3OsvbnHSxPo8MyIqlNcYdQvswQlcHcVc?=
 =?us-ascii?Q?K+TZ+/PuqORiDsrHNdD7cDCyCuyNhjdzDHCvLWKg1G/q9VVkszBjTOQni+vA?=
 =?us-ascii?Q?DY+f0bTQfl/GooietdVgzqIYHH53dmaKwLH2L6x9SX0dcISAhJTxflx2YcBu?=
 =?us-ascii?Q?iNd2VedGT7IUP+j5LGzwAtMchu5KL8Ngbh/PlxcY2WMZWfJUfaQMzF4pFP0i?=
 =?us-ascii?Q?dn5GUyHPymPMPv5y9pEfCaVZgdWYZZAMsmcw9LRH4G5lQN/gBGIetyfvZ6yk?=
 =?us-ascii?Q?CD+FhCd4bvFV24w/KZqauE0QQOo7a7in+kWg2pFuHD02c16idSjF1dhUNr7a?=
 =?us-ascii?Q?ZFLlUdXBdEl+LOYLViPi4fdKOZM9NN9nD4W/1I+dayPuu3xUXrXO1+HVKJDm?=
 =?us-ascii?Q?+gYPGyuY3btBNGq+qQIgghK192t+N9RhyEETlanWJMzF60jLHZvP5+ftAWIJ?=
 =?us-ascii?Q?2y7g3AT9QOvPiYXvOaxsTAG26rGcEFQsH0V5f7mkzevDanl81a4rvlE0gDf7?=
 =?us-ascii?Q?JOGWEzhiLB6LUIWvV3RUr5uL0ewR1INIMX9UfFKh9+ZhlkOcF0Ak9vQkYfIg?=
 =?us-ascii?Q?9Mr+VDe9Sc8sTfDzalSXJZ2uw25164UHufWaKBLe3w5q5Mx8L2XDTGHYrmWT?=
 =?us-ascii?Q?TMkSNpHArty0S2Z2LYHFQhasr/5LLQExynlOBXVagjdoXtZ0hd+0pbAtkWy6?=
 =?us-ascii?Q?18PACk74r88MVd6nxPAlXFVh6WSniCmu5nzP72Wx?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 65155c20-d6c7-4a50-da81-08dc2f00c1a5
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2024 15:05:47.6738
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jamKDxHvNmP7I2qpCp9Krw7Dn8HZK3eOZxTZTAZjPRkkmzNBdklC5IxgjADkHelW
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6797

On Thu, Feb 15, 2024 at 05:00:46PM -0800, Jakub Kicinski wrote:

> But this is a bit of a vicious cycle, vendors have little incentive 
> to interoperate, and primarily focus on adding secret sauce outside of 
> the standard. In fact you're lucky if the vendor didn't bake some
> extension which requires custom switches into the NICs :(

This may all seem shocking if you come from the netdev world, but this
has been normal for HPC networking for the last 30 years at least.

My counter perspective would be that we are currently in a pretty good
moment for HPC industry because we actually have open source
implementations for most of it. In fact most actual deployments are
running something quite close to the mainline open source stack.

The main hold out right now is Cray/HPE's Slingshot networking family
(based on ethernet apparently), but less open source.

I would say the HPC community has a very different community goal post
that netdev land. Make your thing, whatever it is. Come with an open
kernel driver, a open rdma-core, a open libfabric/ucx and plug into
the open dpdk/nccl/ucx/libfabric layer and demonstrate your thing
works with openmpi/etc applications.

Supporting that open stack is broadly my north star for the kernel
perspective as Mesa is to DRM.

Several points of this chain are open industry standards driven by
technical working group communities.

This is what the standardization and interoperability looks like
here. It is probably totally foreign from a netdev view point, far
less focus on the wire protocol, devices and kernel. Here the focus is
on application and software interoperability. Still, it is open in
a pretty solid way.

Jason