Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D95EBC433F5 for ; Sun, 28 Nov 2021 17:57:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359120AbhK1SAR (ORCPT ); Sun, 28 Nov 2021 13:00:17 -0500 Received: from new3-smtp.messagingengine.com ([66.111.4.229]:42821 "EHLO new3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239305AbhK1R6P (ORCPT ); Sun, 28 Nov 2021 12:58:15 -0500 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailnew.nyi.internal (Postfix) with ESMTP id 5B9425802EE; Sun, 28 Nov 2021 12:54:58 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Sun, 28 Nov 2021 12:54:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=UUHBKcGQTfc55BryKks38P9q1VVkigTZnH+REkQ6S Co=; b=iYiN+YmWJhZj5r+gfNGtNnX5WxIBt9yrnzuHllXQ5/GUOrZwgE+2B6kqq v1LacLeZi5zTVMsNqmYhaoCVMmR7ErWb/6IMezfXpj4u51pcnQtnt5/PL1NDm5AK d9CZlDHePScIbPZ7dXxmGzpUDRkVmvPH5Em/yqEQ5o71cDbP6BJlajUs4CRXojAc LzDVEKJNK78Vz3a45eafY2sJ76R61tmWYjTYc4AtkcIHMr9qmajW5sqDYL0fpYei h6Pe7CTiBbD0MbI7zqJGjx+gCDQ2nW3O0Uo+4yN6QuRPMxk6JKI8Svoa6WW0kNuo tVzHESLM+PoWuo8LTnmU9cpdyMP3A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrheeigddutdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtugfgjgesthekrodttddtudenucfhrhhomhepkfguohcu ufgthhhimhhmvghluceoihguohhstghhsehiughoshgthhdrohhrgheqnecuggftrfgrth htvghrnhepieevhfevtdejhfethedvkefgudetudegudethfdtfeefleektdekkeefjeel tedtnecuffhomhgrihhnpehkvghrnhgvlhdrohhrghdpghhithhhuhgsrdgtohhmnecuve hluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepihguohhstghh sehiughoshgthhdrohhrgh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 28 Nov 2021 12:54:55 -0500 (EST) Date: Sun, 28 Nov 2021 19:54:53 +0200 From: Ido Schimmel To: Jakub Kicinski Cc: Toke =?iso-8859-1?Q?H=F8iland-J=F8rgensen?= , Alexander Lobakin , Daniel Borkmann , "David S. Miller" , Jesse Brandeburg , Michal Swiatkowski , Maciej Fijalkowski , Jonathan Corbet , Shay Agroskin , Arthur Kiyanovski , David Arinzon , Noam Dagan , Saeed Bishara , Ioana Ciornei , Claudiu Manoil , Tony Nguyen , Thomas Petazzoni , Marcin Wojtas , Russell King , Saeed Mahameed , Leon Romanovsky , Alexei Starovoitov , Jesper Dangaard Brouer , John Fastabend , Edward Cree , Martin Habets , "Michael S. Tsirkin" , Jason Wang , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Lorenzo Bianconi , Yajun Deng , Sergey Ryazanov , David Ahern , Andrei Vagin , Johannes Berg , Vladimir Oltean , Cong Wang , netdev@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, bpf@vger.kernel.org, virtualization@lists.linux-foundation.org, petrm@nvidia.com, nikolay@nvidia.com Subject: Re: [PATCH v2 net-next 21/26] ice: add XDP and XSK generic per-channel statistics Message-ID: References: <20211123163955.154512-22-alexandr.lobakin@intel.com> <77407c26-4e32-232c-58e0-2d601d781f84@iogearbox.net> <87bl28bga6.fsf@toke.dk> <20211125170708.127323-1-alexandr.lobakin@intel.com> <20211125094440.6c402d63@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> <20211125204007.133064-1-alexandr.lobakin@intel.com> <87sfvj9k13.fsf@toke.dk> <20211126100611.514df099@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> <87ee72ah56.fsf@toke.dk> <20211126111431.4a2ed007@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20211126111431.4a2ed007@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org +Petr, Nik On Fri, Nov 26, 2021 at 11:14:31AM -0800, Jakub Kicinski wrote: > On Fri, 26 Nov 2021 19:47:17 +0100 Toke H?iland-J?rgensen wrote: > > > Fair. In all honesty I said that hoping to push for a more flexible > > > approach hidden entirely in BPF, and not involving driver changes. > > > Assuming the XDP program has more fine grained stats we should be able > > > to extract those instead of double-counting. Hence my vague "let's work > > > with apps" comment. > > > > > > For example to a person familiar with the workload it'd be useful to > > > know if program returned XDP_DROP because of configured policy or > > > failure to parse a packet. I don't think that sort distinction is > > > achievable at the level of standard stats. > > > > > > The information required by the admin is higher level. As you say the > > > primary concern there is "how many packets did XDP eat". > > > > Right, sure, I am also totally fine with having only a somewhat > > restricted subset of stats available at the interface level and make > > everything else be BPF-based. I'm hoping we can converge of a common > > understanding of what this "minimal set" should be :) > > > > > Speaking of which, one thing that badly needs clarification is our > > > expectation around XDP packets getting counted towards the interface > > > stats. > > > > Agreed. My immediate thought is that "XDP packets are interface packets" > > but that is certainly not what we do today, so not sure if changing it > > at this point would break things? > > I'd vote for taking the risk and trying to align all the drivers. I agree. I think IFLA_STATS64 in RTM_NEWLINK should contain statistics of all the packets seen by the netdev. The breakdown into software / hardware / XDP should be reported via RTM_NEWSTATS. Currently, for soft devices such as VLANs, bridges and GRE, user space only sees statistics of packets forwarded by software, which is quite useless when forwarding is offloaded from the kernel to hardware. Petr is working on exposing hardware statistics for such devices via rtnetlink. Unlike XDP (?), we need to be able to let user space enable / disable hardware statistics as we have a limited number of hardware counters and they can also reduce the bandwidth when enabled. We are thinking of adding a new RTM_SETSTATS for that: # ip stats set dev swp1 hw_stats on For query, something like (under discussion): # ip stats show dev swp1 // all groups # ip stats show dev swp1 group link # ip stats show dev swp1 group offload // all sub-groups # ip stats show dev swp1 group offload sub-group cpu # ip stats show dev swp1 group offload sub-group hw Like other iproute2 commands, these follow the nesting of the RTM_{NEW,GET}STATS uAPI. Looking at patch #1 [1], I think that whatever you decide to expose for XDP can be queried via: # ip stats show dev swp1 group xdp # ip stats show dev swp1 group xdp sub-group regular # ip stats show dev swp1 group xdp sub-group xsk Regardless, the following command should show statistics of all the packets seen by the netdev: # ip -s link show dev swp1 There is a PR [2] for node_exporter to use rtnetlink to fetch netdev statistics instead of the old proc interface. It should be possible to extend it to use RTM_*STATS for more fine-grained statistics. [1] https://lore.kernel.org/netdev/20211123163955.154512-2-alexandr.lobakin@intel.com/ [2] https://github.com/prometheus/node_exporter/pull/2074