Received-SPF: pass (google.com: domain of linux-kernel+bounces-60203-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Fri, 9 Feb 2024 21:01:29 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: David Ahern <dsahern@kernel.org>
Cc: Jakub Kicinski <kuba@kernel.org>, Saeed Mahameed <saeed@kernel.org>,
	Arnd Bergmann <arnd@arndb.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Leon Romanovsky <leonro@nvidia.com>, Jiri Pirko <jiri@nvidia.com>,
	Leonid Bloch <lbloch@nvidia.com>, Itay Avraham <itayavr@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>,
	Aron Silverton <aron.silverton@oracle.com>,
	Christoph Hellwig <hch@infradead.org>,
	andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org,
	netdev@vger.kernel.org
Subject: Re: [PATCH V4 0/5] mlx5 ConnectX control misc driver
Message-ID: <20240210010129.GA1010957@nvidia.com>
References: <20240207072435.14182-1-saeed@kernel.org>
 <20240207070342.21ad3e51@kernel.org>
 <ZcRgp76yWcDfEbMy@x130>
 <20240208181555.22d35b61@kernel.org>
 <2bdc5510-801a-4601-87a3-56eb941d661a@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2bdc5510-801a-4601-87a3-56eb941d661a@kernel.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?PmM/wDtKF3Cp67L+Y74lkTyGHqfVpyRY0bx5GyI1ezt6F+c0N0H6ZA163Cga?=
 =?us-ascii?Q?NVpD28Gn3GPpZV44vndM8L/bp9tRxSCoi1+Gdw8LKtLYsoPWfUhXWche1oZK?=
 =?us-ascii?Q?e4CdC9F+mKI/tOmArNUIj1of0ETn7YGXJcm7L67pSSrkjxbdKToOTUN7hWax?=
 =?us-ascii?Q?/QlHEkhQMzYmrV0qvrlsjmfnbPnKn6URTvU54VCt2m4YlS6vqEyunp7H9xDH?=
 =?us-ascii?Q?DIkh2tQN7vdzYpKi1Ok4LPGgPH9sJ6tEmU8aGjdG9pHwTkyVPLYfNXBsbCAK?=
 =?us-ascii?Q?sruN0iCne5y2WluVVzlU9c8I2uTxi7npZ9J/FWjQ12u9dJDG/SpRiGQ5sDbR?=
 =?us-ascii?Q?TXeqDqt7paQBJfMKvf4V48qYW9ojGRkcFezQVtBZYh7D/UGP/pn99y4sFCzm?=
 =?us-ascii?Q?fDIBrJ96WXgP857wt2aUwkytStcdEheqXmnTkt2kjklwinXwsoq59CTEltFx?=
 =?us-ascii?Q?mTl7V/RVBi9Gx/IeM+i38PRwbvcibHJuw8tiyCX7ojjHBxmT0NwRKvdMR5zB?=
 =?us-ascii?Q?44lHG0CNUqHqZs9D8Zk6r3rz+VLfCf6Aviv3t3KuJmZk5egSc+CtzE5olK2M?=
 =?us-ascii?Q?C28p5TSErdhFBOzd3ihIlSzPhDkSUfmAFPU7x8QhJuWKgyC8muEjvDlBeR1m?=
 =?us-ascii?Q?AfS6b2YANAIR67KMyH0EYYEkoqy9MUBxS1aW0dkzJNW8v4uxxuTpkBYmXVKb?=
 =?us-ascii?Q?CqpHkY79lXApAFzlnTRf7F3WN4cGy+SyEblEm4KlQBe+hCumt1tPs/tngXWF?=
 =?us-ascii?Q?uz1L2RIqaF+hDLpp096mHT5FZLN4qGnaM2g+7u1gBlzv35Sf0zPN1JJubABg?=
 =?us-ascii?Q?ZjDePrY2ESNuACGYiIAInMcY9V/95qx5cKSy3fxO5DPLy22VGjG+brqJ9v69?=
 =?us-ascii?Q?M4SWUgutIgTRlFOoD2qqNNGyae6KYh7b9oBc3deyRjIh00xctuD3BoAxNfDH?=
 =?us-ascii?Q?LR+oytyZHZvG8iQewPTHmTtO1ZHCjnqKM38ik2KdXZ1W8xK6mXDecmw5zGXq?=
 =?us-ascii?Q?+fikt6QlLtn1xS2BkZ3UoTwUiAgvFqC6uti7ggXN4iUPef5hFkSvdME5wR8W?=
 =?us-ascii?Q?qpEJycwktUn6WkGz2ZAfA5i5O8irZGvkQzAUa2WG3JNnVSw+UEIOTpHftd2n?=
 =?us-ascii?Q?JZ8aqEPT2zBfcvYU1l+iX+HrGM9iOBCZk9Olcr/cpIb1c9VNbMK9AabHwukr?=
 =?us-ascii?Q?5vbmIpsboFCjemapCQ9bxcsQeFz789/sbVvEvLYtAPwt1aANVKG2rd0eMOrf?=
 =?us-ascii?Q?5gEwtFTZPfO3dR2J/5toPUk7zTpxAfq61oVOt0zF71usn93vkD65IomcwMVd?=
 =?us-ascii?Q?9u6lUdksqGvDJoLGc83YnQd1XOcRDsEqFlpmZKNBSF6kI88DnVnKsu8JXEDl?=
 =?us-ascii?Q?tdWDqzfy4FgvRKEzK3YhqK2ZEo7VojiwULpvv884TxciW+sJpH/bDpYMvr94?=
 =?us-ascii?Q?C9if8BkdBLaHDNHnCGgbPWFE9EtV3S8bhqTRM6FlcZNB9cnBDRXdjEk83Jvw?=
 =?us-ascii?Q?G4LyFpiIg+YWugnBRYqXWclcggKYWFyL2hklNB1oqUiiyO0k7SOnW6A2lMmm?=
 =?us-ascii?Q?d0rke40qtPeaDzGy9Ns=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4bdbb7ec-7c67-4ea6-c1b7-08dc29d3d143
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2024 01:01:30.7113
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DFscXawXMlIKAmRZR5HbbwVPmcso2NTlcby1Ni8iERwDZb2yPg3plzJ6ltkoC/YA
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8742

On Fri, Feb 09, 2024 at 03:42:16PM -0700, David Ahern wrote:
> On 2/8/24 7:15 PM, Jakub Kicinski wrote:
> >>> Ah yes, the high frequency counters. Something that is definitely
> >>> impossible to implement in a generic way. You were literally in the
> >>> room at netconf when David Ahern described his proposal for this.
> 
> The key point of that proposal is host memory mapped to userspace where
> H/W counters land (either via direct DMA by a H/W push or a
> kthread/timer pulling in updates). That is similar to what is proposed here.

The counter experiment that inspired Saeed to write about it here was
done using mlx5ctl interfaces and some other POC stuff on an RDMA
network monitoring RDMA workloads, inspecting RDMA objects.

So if your proposal also considers how to select RDMA object counters,
control the detailed sampling hardware with RDMA stuff, and works
on a netdev-free InfiniBand network, then it might be interesting.

It was actually interesting research, I hope some information will be
made public.

> BTW, there is already a broadcom driver under drivers/misc that seems to
> have a lot of overlap capability wise to this driver. Perhaps a Broadcom
> person could chime in.

Yeah, there are lots of examples of drivers that use this kind FW API
direct to userspace. It is a common design pattern across the kernel
in many subsystems. At the core it is following the general philosophy
of pushing things to userspace that don't need to be in the kernel. It
is more secure, more hackable and easier to deploy.

It becomes a userspace decision what kind of tooling community will
develop and what the ultimate user experience will be.

> > Why don't you repost it to netdev and see how many acks you get?
> > I'm not the only netdev maintainer.
> 
> I'll go out on that limb and say I would have no problem ACK'ing the
> driver. It's been proven time and time again that these kinds of
> debugging facilities are needed for these kinds of complex,
> multifunction devices.

Agree as well. Ack for RDMA community. This is perfectly consistent
with the subsystem's existing design of directly exposing the device
to userspace. It is essential as we can't piggyback on any "generic"
netdev stuff on InfiniBand HW. Further, I anticipate most of the
mlx5ctl users would actually be running primarily RDMA related
workloads anyhow.

There is not that many people that buy these expensive cards and don't
use them to their full capability. 

Recently at usenix Microsoft shared some details of their production
network in the paper "Empowering Azure Storage with RDMA".

Notably they shared that "Traffic statistics of all Azure public
regions between January 18 and February 16, 2023. Traffic was measured
by collecting switch counters of server-facing ports on all Top of
Rack (ToR) switches. Around 70% of traffic was RDMA."

It is a rare public insight into what is going on in the industry at
large, and why RDMA is a significant and important subsytem.

Jason