Received-SPF: pass (google.com: domain of linux-kernel+bounces-65503-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Andy Gospodarek <andrew.gospodarek@broadcom.com>
Date: Wed, 14 Feb 2024 11:17:58 -0500
To: Christoph Hellwig <hch@infradead.org>
Cc: Saeed Mahameed <saeed@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Leon Romanovsky <leonro@nvidia.com>,
	Jason Gunthorpe <jgg@nvidia.com>, Jiri Pirko <jiri@nvidia.com>,
	Leonid Bloch <lbloch@nvidia.com>, Itay Avraham <itayavr@nvidia.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Saeed Mahameed <saeedm@nvidia.com>,
	David Ahern <dsahern@kernel.org>,
	Aron Silverton <aron.silverton@oracle.com>,
	andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org
Subject: Re: [PATCH V4 0/5] mlx5 ConnectX control misc driver
Message-ID: <ZczntnbWpxUFLxjp@C02YVCJELVCG.dhcp.broadcom.net>
References: <20240207072435.14182-1-saeed@kernel.org>
 <Zcx53N8lQjkpEu94@infradead.org>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Zcx53N8lQjkpEu94@infradead.org>

On Wed, Feb 14, 2024 at 12:29:16AM -0800, Christoph Hellwig wrote:
> Chmining in late after a vacation last week, but I really do not
> understand the discussion this cause.
> 
> Complex devices need diagnostics, and mlx5ctl and the (free software)
> userspace tool provide that.  Given how it is a complex multi-subsystem
> driver that exports at least RDMA, ethernet, nvme and virtio_blk
> interfaces this functionality by definition can't fit into a single
> subsystem.
> 
> So with my nvme co-maintainer hat on:
> 
> Acked-by: Christoph Hellwig <hch@lst.de>
> 
> to th concept (which is not a full code review!).

I've been trying to figure out how to respond correctly to this over the
last few days, but I share your sentiment.  It's probably time to have
something like this upstream for more devices.  My initial concerns with
something that allows direct access to hardware to send messages to FW
or read/write registers over BARs were:

1.  How someone working at a distro would be able to help/understand if
a tool like this was run and may have programmed their hardware
differently than a default driver or FW.  There now exists a chance for
identical systems running identical drivers and FW to behave differently
due to out-of-band configuration.  One thought I had was some sort of
journal to note that config happened from outside, but I'm not sure
there is much value there.  With the ability to dump regs with
devlink health it's possible to know that values may have changed, so I'm not
concerned about this since that infra exists.

2.  If one can make configuration changes to hardware without kernel
APIs (devlink et al), will people still develop new kernel APIs?  I
think the answer to this is 'yes' as realistically using default tools
is much better than using vendor tools for regular configuration.  Even
if vendors provide shortcuts to program hardware for eval/testing/debug
my experience is that these are not acceptable long-term.  Requests are always
made to include this type of changes in future releases.  So I'm not too
concerned about the ossification of kernel APIs due to this being included.

So if there is general agreement that this is acceptable (especially
compared to other out-of-tree drivers, I think a few who find this
useful should sync on the best way forward; I'm not sure a separate
driver for each vendor is the right approach.

If upstream (and therefore distros) are going to accept this we probably
owe it to them to not have misc drivers for every different flavor of
hardware out there when it might be possible to add a generic driver
that can connect to a PCI device via new (auxiliary bus?) API.