Received-SPF: pass (google.com: domain of linux-kernel+bounces-132109-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 4 Apr 2024 17:44:54 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Edward Cree <ecree.xilinx@gmail.com>, David Ahern <dsahern@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Christoph Hellwig <hch@infradead.org>,
	Saeed Mahameed <saeed@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Leon Romanovsky <leonro@nvidia.com>, Jiri Pirko <jiri@nvidia.com>,
	Leonid Bloch <lbloch@nvidia.com>, Itay Avraham <itayavr@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>,
	Aron Silverton <aron.silverton@oracle.com>,
	linux-kernel@vger.kernel.org,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Andy Gospodarek <andrew.gospodarek@broadcom.com>
Subject: Re: [PATCH V4 0/5] mlx5 ConnectX control misc driver
Message-ID: <20240404204454.GO1723999@nvidia.com>
References: <Zf2n02q0GevGdS-Z@C02YVCJELVCG>
 <20240322135826.1c4655e2@kernel.org>
 <e5c61607-4d66-4cd8-bf45-0aac2b3af126@kernel.org>
 <20240322154027.5555780a@kernel.org>
 <1cd2a70c-17b8-4421-b70b-3c0199a84a6a@kernel.org>
 <0ea32dd4-f408-5870-77eb-f18899f1ad44@gmail.com>
 <20240402184055.GP946323@nvidia.com>
 <83025203-fefb-d828-724d-259e5df7c1b2@gmail.com>
 <20240404183305.GM1723999@nvidia.com>
 <20240404125339.14695f27@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240404125339.14695f27@kernel.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?RV7qd6cJAwKg9WQG4HNiy0vypXz/ZDNtQ1eWoBfOpqRsmfnSVPGm5dHBQOC6?=
 =?us-ascii?Q?XPiPScUnXyBfV/talf2Gs2W66MIuD8xamjCEKiq+sDVlhr5cmHnxK3kdfhYM?=
 =?us-ascii?Q?f5XDdGNSPjNxy6faoPQjz2LsRMBiuFJMVGz+mzCEMhSLaTMUgoKS10EXvM7D?=
 =?us-ascii?Q?3tQaI7Ybk+8tyO1Avyy/UH/BnSjh0C/WudWNMWRKmR/enc+1A1HVqNhHe5z/?=
 =?us-ascii?Q?9nJCffyizZ4Qt0ztHHaBrAg72j91cu9FjCJaFz+uw32tcwjQ8vpfLa7cxAKE?=
 =?us-ascii?Q?RMLuDw+M7HRoJ367tC4r9C0BVtywGdTezHZHtdCkf14oC3pU7d3SRhbwb6oQ?=
 =?us-ascii?Q?O2FMYa5y64nzvsbb/JxuJgSdEmxR/FRIV76IGNOMrylzljMK/WjV5CX/PYTk?=
 =?us-ascii?Q?KGBg9bDdELpNv1AHQJmHuoiaHAmXVkwKBlR8BgoWlldPVxhgtoxKZSx0INJZ?=
 =?us-ascii?Q?Nu4NbXUFIql5r74CT21j6rVOO1ShLbJwmrIbXitLeFqJEKlsYLZqZ8VXpBlb?=
 =?us-ascii?Q?VkX2TVuqWVxSln4KQ85nqe+N7uvfECPn3t0izRD6/K3FNSMA5i2v/obbPbSL?=
 =?us-ascii?Q?+nc/VXoHPLlXmc9UEzjQsNdf6mkyN2JxwL0X922vWpjyjcZTXn+ux1atvX8z?=
 =?us-ascii?Q?PYGP1VVGi7CVrHHw8feN86a6lgMnOGNDrQHiLZXDEdnbJiGOyZqUyxfOcRku?=
 =?us-ascii?Q?7AMPs7ibZhaODFRThZcaDC1RVxa+EnfAQxtrdaactHW6AQPTZgKPCtc1PK4D?=
 =?us-ascii?Q?U5ROkM/rQrFMpo+br5SGJ1ipkbtgSIKQWcMtJkKpXhyyqln9Z+Vb37VDDnPD?=
 =?us-ascii?Q?aUHbZIiZUtZhjR06bBF1LcvuryYF7D4FWjpP1UljTsbKsev4SQzr5+bFgD3V?=
 =?us-ascii?Q?8dtBekM4Fv4raEJPY+XYdtBzNAaHQ975nPGtLz2jBgwADFtInaQQD5NBTgsx?=
 =?us-ascii?Q?y4nDOc9d7+sZxpwEoRsVIWWPINn5xu+FUomSdCO6Sa3rfJu58GITKGdt4aB9?=
 =?us-ascii?Q?DHRDd/ZI60Y0XDr89jWEYPJCCvpOm7unp33v/Cf7PleOCCJCjMIXeqXev6VP?=
 =?us-ascii?Q?BIEEdyIqAYs3KO8323+qsVLArE2i7AUFHs72nRPEWIsW/AzhgSfDC+DcuBSw?=
 =?us-ascii?Q?95FIDeUxBw+hcwyrmlu2ta3wZc25rzBkPJnPvirpXcl4AuV5OApydtAdGOvc?=
 =?us-ascii?Q?TxEn9D9+542RJMBuEFV2v+jkN9RyEbzXrINh1omQTtoqv6kU5ExAcu3G2h5P?=
 =?us-ascii?Q?pliLmN+TPXPivmd3P8k/uR1izfrEuS30IDBXZsnvt2PO+pLNDX0LC1kMBJTM?=
 =?us-ascii?Q?Nm9jZgR/VMfMwkjR2ScNfKECyy5EFlBDCM+Lx+bQSJCXJvEQKHoPbXTnrYSz?=
 =?us-ascii?Q?hTP8L4yN+eJ5n4gpNPM+8gZDrIjU5wSLAogQckNZ8andPkhrVgD0DnbgVXqf?=
 =?us-ascii?Q?FGxXhGOmtbkOHt+gF1zPIWPVAu7LTgueDsn5MozPe6EMwGfValvmSQQ+NidP?=
 =?us-ascii?Q?OPcIP9nad3GGxD5D9d3v6kirlII+Nwz9ScLjIJ+7gkpVmCjjsvGZLBRVmtrv?=
 =?us-ascii?Q?7gh7cEC6F/ow6gBDBrR3/Mrlxcs6BMB5gieED06x?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 71215c48-14a5-4d12-b10e-08dc54e81617
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Apr 2024 20:44:56.1382
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: e2+BJcGbtrXO+usbmQ8WWG1dKbSak7mc5xTRyKKot81RITD4YhEluAvTkw7VQcIF
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6317

On Thu, Apr 04, 2024 at 12:53:39PM -0700, Jakub Kicinski wrote:
> On Thu, 4 Apr 2024 15:33:05 -0300 Jason Gunthorpe wrote:
> > Uh no, mlx5 already has an excellent in-tree driver, thank you very
> > much. The configuration is not changing the driver, it is changing the
> > device.
> > 
> > Consider, I can ship unique devices pre-configured for each site's
> > special needs. They still work with the same inbox driver.
> > 
> > In fact that happens already for certain large customers. It is why
> > Jakub recently pointed out that Meta doesn't need any
> > provisioning/mlx5ctl/misc driver. They just buy different devices than
> > everyone else.
> 
> > Further it is really rude and inappropriate to say that some customers
> > should not be able to enjoy intree drivers because of your aesthetic
> > opinion of hacks in a device's design.
> 
> To my knowledge the "customizations" are mostly around fitting into OCP
> servers.

Nope. I understand it is significant. If Meta had to work with a COTS
environment like a HP/Dell customer then Meta would have a list of
flash configurables to set. I think you greatly underestimate the
privilege of being at a hyperscaler and having vendors create custom
products just for you..

> Those unfamiliar with how hyperscalers operate can mentally
> replace $hyperscaler with HP or Dell in your message. Minus all the
> proprietary OOB management stuff those guys also provide.

A significant Dell customer will get a server pre-populated with a NIC
with some generic Dell configuration. In most cases the customer will
have to then program the flash to match their needs. Set a specific FW
version, set site specific configurables, etc.

Similar to how a Dell customer will have to change the BIOS settings
in the Dell to match their needs.

> Discussion is about mlx5ctrl not "the device" as much as you'd like
> to equate the two.

I view mlx5ctl/fwctl as a window into the device.

Jason