Received-SPF: pass (google.com: domain of linux-kernel+bounces-112159-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Fri, 22 Mar 2024 22:33:35 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Andy Gospodarek <andrew.gospodarek@broadcom.com>,
	David Ahern <dsahern@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Christoph Hellwig <hch@infradead.org>,
	Saeed Mahameed <saeed@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Leon Romanovsky <leonro@nvidia.com>, Jiri Pirko <jiri@nvidia.com>,
	Leonid Bloch <lbloch@nvidia.com>, Itay Avraham <itayavr@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>,
	Aron Silverton <aron.silverton@oracle.com>,
	linux-kernel@vger.kernel.org,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: [PATCH V4 0/5] mlx5 ConnectX control misc driver
Message-ID: <20240323013335.GM159172@nvidia.com>
References: <ZczntnbWpxUFLxjp@C02YVCJELVCG.dhcp.broadcom.net>
 <20240214175735.GG1088888@nvidia.com>
 <20240304160237.GA2909161@nvidia.com>
 <9cc7127f-8674-43bc-b4d7-b1c4c2d96fed@kernel.org>
 <2024032248-ardently-ribcage-a495@gregkh>
 <510c1b6b-1738-4baa-bdba-54d478633598@kernel.org>
 <Zf2n02q0GevGdS-Z@C02YVCJELVCG>
 <20240322135826.1c4655e2@kernel.org>
 <20240322214423.GL159172@nvidia.com>
 <20240322152924.64be7ec4@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240322152924.64be7ec4@kernel.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?Hs/RFWIbaFw26jwk8SfvVWo2zMd3Or/MVUMy7WtvZcR7CmiPw8Bv5FKwX6aa?=
 =?us-ascii?Q?YkoPCER582Ji9NzrTjkqOs58XPhQ/ctM3yLYz74YyZ0pFSlCZHQoQvavB7vp?=
 =?us-ascii?Q?TNgmAQhnnOtDNm9UHN6Ed+yhxWeRi0iC8bFPdKNqhDnjn6u57v6j4N3LepNM?=
 =?us-ascii?Q?PbYWZ+yoeRQwqtRs2mL/z5lBl8Q1Bg1HN/tSKmxcLO6CCtFWkQZucaZ6RqIt?=
 =?us-ascii?Q?CrkvGKqbIdPINZLwvd272Vb/WBf2IcBMy1y0AdfJfeRxC/FWCBiFNX0CcLdy?=
 =?us-ascii?Q?Qy/9uINYqYYzFKx/qZL0N2OSrw5hJfY4wqcxKDFlY7DFC+btltdTrL3eEnpb?=
 =?us-ascii?Q?xN3Iz6qo0je4BpGuT1zijXgmpI/LvbJIAnWMTZgGE+ncIzG6YGoCzfCL9fae?=
 =?us-ascii?Q?jIPX5TL+gyGBuMRnTPxJZBEpMd8X0UrKW3qc3i8iFJ0KpZt4Q+sESalz2ZgA?=
 =?us-ascii?Q?7FD9LddUjoL5zFGfFhEAu7hTGrou2L2+J2J1pd4CysXcLxieUL3fg2MXI+6z?=
 =?us-ascii?Q?Ie/TE5uVlKTUQFaMyp9RYgCqkLq/C7Idb3GK51S3XIgD4XFQy36Vuv66PBDR?=
 =?us-ascii?Q?2tiaup58hxh2Ax14jMYmCQaw/X7dt7KTkiqD0jbQIPu+Bsu0pVrcCWKEFjOR?=
 =?us-ascii?Q?cIsF+sD138YTwhWaIcmoV0EL0DdXf6kLDvFw7gsG5TKzXUace1AGuVjSD0cI?=
 =?us-ascii?Q?zdXQbF+AvGjC5xBs8HrldfzgmO4GfWq9wJgNdtpsm9b/iah1IZmzxsuOns+2?=
 =?us-ascii?Q?waUflmrSpCGE2coy3dIQSNDmyZlyl1Zi8RISjbM6pnFNQ6RY0pP9NVJjdB/S?=
 =?us-ascii?Q?n2G3fPGrI6xBg8zJTLzElutKfojCT3TxGyTbXPeaK/XWnfojU94Sli1tMMcH?=
 =?us-ascii?Q?BDMuxTzDoZejugFAsRxLPnOnMuPG5RDlLFuiBeYZUSy8qnqBJepUs2iYDrh2?=
 =?us-ascii?Q?HO5f2RSzmyeCup/qLH9jIiO4SkxH/+idbaf71wPS5u8NkG3F3QPcMXfzVqoP?=
 =?us-ascii?Q?ZUT9C+Q4uigBtw9gzI7y4O77tnBsDLEmFRrgY37Ott+r7kGrdJUdWY4905Tz?=
 =?us-ascii?Q?//d5EX/BfmPAFWILWKtUUK4B4AFW+VEdJN8TDuLAj7YUbGl0qRThq0F9pQCw?=
 =?us-ascii?Q?FgZgXx7Qid2k1BqeJHpSuwymnEJDf8pwwxuwacuDkRgbZ97Faiar0RVGeN6M?=
 =?us-ascii?Q?f6Ns7wYpO6xLGnrCCy7Az042XyG4H3t7yMnDScjeOdN8SLpz7vEQJAROGzfP?=
 =?us-ascii?Q?l5cRVGubfI7DCtDumIrk7jLfn7sAM1JOkkI9RSvJ0hcdFDb4UQhafL0A0Gg8?=
 =?us-ascii?Q?jnNSi3Pk3D/qHSRQD357XTVSTSyFfH/sTPxrQrMJysB5zABQdJwAN/ev2thk?=
 =?us-ascii?Q?Np/y0OWjr60sR/4wYYe2OenMhQkZ+6fyiIubAiIW0g6tZ060Wn/5pXEhuo+D?=
 =?us-ascii?Q?y/VHc+TAnA8+QDmzFlE/W126UAxlZkJOqS/gcUTWYAyGXT1bzyCaG7ndm0eJ?=
 =?us-ascii?Q?QuVKCYAUuiq2FMHzkrutBohAJWgm2U1+k7LJOqzzbiD8WPCRrGzQAkR2Ruec?=
 =?us-ascii?Q?F3+Z5M9u7YIKm+xdsvsz9kI4khhPDvyW4jDYIlnq?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3990ba83-db3a-4b5f-5a26-08dc4ad942dd
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Mar 2024 01:33:37.1201
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2ey/UvT9GkK68ETRcj+Va0zIzT2VYskvAw7EUSjUlMEPLw2EIKROZlkf+Fer8zmz
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6954

On Fri, Mar 22, 2024 at 03:29:24PM -0700, Jakub Kicinski wrote:
> On Fri, 22 Mar 2024 18:44:23 -0300 Jason Gunthorpe wrote:
> > On Fri, Mar 22, 2024 at 01:58:26PM -0700, Jakub Kicinski wrote:
> > > > Well said, David.
> > > > 
> > > > I would totally support doing something like this in a fairly generic
> > > > way that could be leveraged/instantiated by drivers that will allow
> > > > communication/inspection of hardware blocks in the datapath.  There are
> > > > lots of different ways this could go, so feedback on this would help get
> > > > us all moving in the right direction.  
> > > 
> > > The more I learn, the more I am convinced that the technical
> > > justifications here are just smoke and mirrors.  
> > 
> > Let's see some evidence of this then, point to some sillicon devices
> > in the multibillion gate space that don't have complex FW built into
> > their design?
> 
> Existence of complex FW does not imply that production systems must
> have a backdoor to talk to that FW in kernel-unmitigated fashion.

I think we've been over this endlessly, production systems require a
solid debugging story for all software in the system, including the
device FW.

> As an existence proof I give you NICs we use at Meta.
> Or old Netronome NICs, you can pick.

I wouldn't pick those, they don't meet the multi-billion gate
criteria.  ie expensive chips built on the most cutting edge and
expensive process. Smaller chips on non-cutting edge process have
different economics. Startups have different economics.

Look at something like an Intel/AMD GPU or Habana labs device. Lack of
FW creates a very opaque driver that is twiddling megabytes of
register mnemonics that is incomprehensible without HW
documentation. A win for code availability, but it hasn't actually
crossed into being usefully open.

> > Despite all of those having built devices like this well before the
> > "AI gold rush" and it being a general overall design principle for the
> > industry because, yes, the silicon technology available actually
> > demands it.
> > 
> > It is not to say you couldn't do otherwise, it is just simply too
> > expensive.
> 
> I do agree that it is expensive, not sure if it's "too" expensive.
> But Linux never promised that our way of doing SW development would
> always be the most cost effective option, right? Especially short
> term. Or that we'll be competitive time to market.

By "too expensive" I mean the vendor cannot produce a chip at a price
that is salable. Or a startup goes out of business because it can't
afford to respin.

Linux promises community collaboration where the community members
broadly should be driving the priorities. When TTM matters and is
agreed it is done. See for instance the various massive security
fixes.

> > > RDMA is what it is but I really hate how you're trying to pretend
> > > that it's is somehow an inherent need of advanced technology and
> > > we need to lower the openness standards for all of the kernel.  
> > 
> > Open hardware has never been an "openness standard" for the kernel.
> 
> I was in the meeting with a vendor this morning and when explicitly
> asked by an SRE (not from my org nor in any way "primed" by me)
> whether configuration of some run of the mill PCI thing can be
> exposed

"run of the mill PCI thing"? Does this thing already have devlink
knob? Usually "run of the mill PCI things" are configured through the
PCI subsystem, not devlink.

> via devlink params instead of whatever proprietary thing the vendor was
> pitching, the vendor's answer was silence and then a pitch of another
> proprietary mechanism.

Our team was very excited about devlink when it first came about. But
now we have so many devlink parameters that have been kept out of
mainline I see that the excitement has died.

> So no, the "open hardware" is certainly not a requirement for the
> kernel. But users can't get vendors to implement standard Linux
> configuration interfaces, and your proposal will make it a lot worse.

I don't agree. If you can't get your vendor to implement the thing you
want on devlink right now today, this fwctl isn't going to change that
one bit. You already have the vendor tool and the vendor telling you
to use it. It makes no difference at all how the existing vendor tools
reach the device.

Indeed, counting on lockdown to break all the existing vendor tools
and render them permanently unusable seems to me to be straining one
of the few hard full project rules Linux actually has: don't break
existing userspace.

Think bigger, maybe your SRE will be happier if as part of this we can
get the vendors to agree on some common userspace tooling for device
configuration! Wouldn't that be a great big ecosystem improvement!

Frankly, there are many forms of common interfaces and many paths to
get there. I don't view your very restrictive approach to be helpful
to growing this space. It hasn't delivered a working ecosystem, and I
don't think it ever will. Instead, as this thread is showing, we have
a bunch of unhappy community members and poor upstream support for
devices.

Jason