Received-SPF: pass (google.com: domain of linux-kernel+bounces-95851-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
From: Haiyang Zhang <haiyangz@microsoft.com>
To: Shradha Gupta <shradhagupta@linux.microsoft.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>
CC: Eric Dumazet <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>,
	Paolo Abeni <pabeni@redhat.com>, Leon Romanovsky <leon@kernel.org>, Thomas
 Gleixner <tglx@linutronix.de>, Sebastian Andrzej Siewior
	<bigeasy@linutronix.de>, KY Srinivasan <kys@microsoft.com>, Wei Liu
	<wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>, Long Li
	<longli@microsoft.com>, Shradha Gupta <shradhagupta@microsoft.com>
Subject: RE: [PATCH] net :mana : Add per-cpu stats for MANA device
Thread-Topic: [PATCH] net :mana : Add per-cpu stats for MANA device
Thread-Index: AQHacJ8N8LLwvYYKc0y/h+tCzJvDjrEsctXQ
Date: Thu, 7 Mar 2024 16:17:59 +0000
Message-ID:
 <DM6PR21MB1481E60F25B7A379FE8AFC75CA202@DM6PR21MB1481.namprd21.prod.outlook.com>
References:
 <1709823132-22411-1-git-send-email-shradhagupta@linux.microsoft.com>
In-Reply-To:
 <1709823132-22411-1-git-send-email-shradhagupta@linux.microsoft.com>
Accept-Language: en-US
Content-Language: en-US
msip_labels:
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=6909631d-84c7-43bd-bd96-3c3f2658feff;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2024-03-07T16:10:57Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=microsoft.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: DM6PR21MB1481:EE_|PH7PR21MB4043:EE_
x-ms-office365-filtering-correlation-id: 9f4d481d-3f83-4649-9377-08dc3ec227d3
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
 7u4nQuFeANry/UnU/Yz8kAeHtZvQYzdLWViJlOSMyE1oBhQQMG42ejh0rkRCvG0Yzs3aollpApCmz5qDrwkAVqFB8CRQV/+aMn7uNWo3xYlHt4QYnj4MiLSISfz6RlWxqK+F1cKyxhm33jA78spEPngqycOGChgVsAhRaZb/iQUicUzu99WQaClCsGB5KyS7Oesl7Ne9QX52r30IeEEeYh8oY1X7cGutUtHXZpqxcW3b4wcgyvuagVyUOPUMUC/SU5H0khZLbHuBObKxYBaHdQCWhJySST27/WzSFkDFbmgc89ETmZVSHbC1G3/t5apIX8p7D6Q61XG4xrXWE7TCbueCKAsxHngYSH8V6z6BQyenaNavUBxe7JVhK3qN8K9xVg3fDkAQr4SlIaxLuBK5aUyg2SnpKIzaYAN7my/gZgauP91isQoucOVljmsRsAIDEfKXuGorH0TQfOWmgNKxR5k83Pry1vf+Tnc71SmzzOfFajX7t6/MroqctH5erozK2c2HV5Sekx0o4THajFJOEFQWRnWqFM/cZKiS7UCoDR2WT/5b+jogZ7dX9rVzcT4I56mrVD77ZwuPbtWwBx0BUZB1WQq91tjz7RPWO+mxdHCq/DGdfxgwLUeYpbFklwKAN1dr5VzasK+lL6Pcs7Vnp/FvvAgbzekoSFGHTbgR9aXBl18wKMYayieILgVRLYPmFD08PIA/OWo4L8lit9MhEQ==
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1481.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(38070700009);DIR:OUT;SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?75u+gOYjoAb/pUglTqsg8OLNW5hSS1DlbyfzPYdVuJdeuATnPACVKw8oVhq+?=
 =?us-ascii?Q?CxHNBqXKBz2dNF+ONfmZQpy5rN4vb7P4gkmH8lJzAW4zh/mGFWtba8HqRHop?=
 =?us-ascii?Q?vwcK2mVS9VF47xSgn30uCNe43E4HugGwlCN0DDgViVSTmXRE65qEWycui8pe?=
 =?us-ascii?Q?APE29pAriWVWTL7C5XCwwtw0Z7KXEPMMd2o/6KI8vx3z3nVJpTbTSqGL+8HW?=
 =?us-ascii?Q?Fn7LgxltcFdqogzT6wZg5ctzWCWoD0zHhN5xBlybTAVSBzvHNdoiR5EJfZsh?=
 =?us-ascii?Q?4hhEdU/4NJZoGnhA4NRwc42bJ/2/K1lA8wuJIiTVN4JK/jOJIt9ykbHdvPBY?=
 =?us-ascii?Q?LBqWrxbmds7RUYTQSarlZiM7lcCgWe2z35TuqJOEDJ3dSjJVJ8od87P4fw7s?=
 =?us-ascii?Q?M3mwRCssKoQlYo67kGTWJ9UHnazlCoxP0y0JVW9hK2vizI48GQdKk7erx8dk?=
 =?us-ascii?Q?3Gi61HkvXvXQpXDKs81SQvbC04a5vJxSqB/Vzff6mstOo4K8T6/7zfe88Pcb?=
 =?us-ascii?Q?dX1rDqRecTYysQKZE/ZrDRBkZv3uh9q+4L2BQQroDCW1v6gH15SW9iIdC4dz?=
 =?us-ascii?Q?cEpAa/irYjdY2xoG1YE99AVII88cfEpXmcFWqzal5Xra7+zFqAO++UCCRMTY?=
 =?us-ascii?Q?TAhRaf1C22Bk2P5W5E9mUm98/+dFMc1RKAEVrs0hVBesLQiifE6Fu1/oPB3A?=
 =?us-ascii?Q?mhBYducz08fI89wsPMvPk0aOgIXZ3UR3G3fi08Ly6MImnBuwE67Dy43+rSI+?=
 =?us-ascii?Q?4+L2y/SiEcBdD3cHWNdAI+ZYnsyTgJ6uPsdM4iHOO7wIdyRceo6ck3mxZ4uS?=
 =?us-ascii?Q?UAQex/ZO6GtkgF+WRrDoOo7X7vAq0QD/2SYzABk6xOfvTNdv6Z1ekAlX/kwt?=
 =?us-ascii?Q?GdZJvYUSduS6Ilmgcup0FssAlNrtbGb+wQdRpEgmEST/BtzSBqqpJtymFUSs?=
 =?us-ascii?Q?hyuUIqT8a93bMAwYLQSIqex7SCokju7C5a+eJj3tZq3ImxvBH2YLH1MMN0f/?=
 =?us-ascii?Q?sSgfRn/eoJ3GfqWm1zIWpdUhxvB0dmHVObbYPp+TxLfNrSukKdvA9iTNcHGi?=
 =?us-ascii?Q?nTxXOhSi8gto49Jo92kx3WraLNk9QbByeMrXzXQiKJ2y/dkYAWIssB6ECfg9?=
 =?us-ascii?Q?SjSMfRHRn9YH8cvP2/BOTsb03coB85g+Q7shU8J4ns5VRjMGDAV3j9ZkYPaR?=
 =?us-ascii?Q?dvH3G8Xht6pB6laLFDdMz0pAb2R3jA3CCmGLlEz7PMgi5EGad2X8N6v6pTHh?=
 =?us-ascii?Q?Mihdcb7uV2Hsv5st0QdRJYQBbz/gNQ/M97qlJe21GWu7TaAJKfDZ2qbGgy5A?=
 =?us-ascii?Q?gFfeEL+7zCQF3xh8yOhWCXXfFw6hXlEqzp+oTM6deGpjV6WBS20yRIDPqKJ4?=
 =?us-ascii?Q?xEH0YAuNLq/+JoUmbBQ7ttfSTqcfp0ze1dxSEKHXvOa13a78Mfk7ib+4YNlY?=
 =?us-ascii?Q?GKpzG75PG6sxI9C/ZDvbKUwMYuhUJjlpMXJlBt4QyCe22hpbl609L/Yfhtl/?=
 =?us-ascii?Q?cerrnBL3zR51Kv+2ctFJ9cweq2mXhiHHWqzyD/AuZ3A1Xm6U+lEssZay7tkM?=
 =?us-ascii?Q?p2Bt6hCaQM23I2Nxzv5R0hJf0BqeZL05IXi9nEwg?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1481.namprd21.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9f4d481d-3f83-4649-9377-08dc3ec227d3
X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Mar 2024 16:17:59.2380
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: nKLWLLPkGaMWEu63oIVnnSVABf76t3UYlqr7ib7J1YRE/XbZkgA545+oahn3dqkXNKsR1DV11XLy6Lic3ckvNw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR21MB4043


> -----Original Message-----
> From: Shradha Gupta <shradhagupta@linux.microsoft.com>
> Sent: Thursday, March 7, 2024 9:52 AM
> To: linux-kernel@vger.kernel.org; linux-hyperv@vger.kernel.org; linux-
> rdma@vger.kernel.org; netdev@vger.kernel.org
> Cc: Shradha Gupta <shradhagupta@linux.microsoft.com>; Eric Dumazet
> <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni
> <pabeni@redhat.com>; Ajay Sharma <sharmaajay@microsoft.com>; Leon
> Romanovsky <leon@kernel.org>; Thomas Gleixner <tglx@linutronix.de>;
> Sebastian Andrzej Siewior <bigeasy@linutronix.de>; KY Srinivasan
> <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>; Wei Liu
> <wei.liu@kernel.org>; Dexuan Cui <decui@microsoft.com>; Long Li
> <longli@microsoft.com>; Michael Kelley <mikelley@microsoft.com>; Shradha
> Gupta <shradhagupta@microsoft.com>
> Subject: [PATCH] net :mana : Add per-cpu stats for MANA device
>=20
> Extend 'ethtool -S' output for mana devices to include per-CPU packet
> stats
>=20
> Built-on: Ubuntu22
> Tested-on: Ubuntu22
> Signed-off-by: Shradha Gupta <shradhagupta@linux.microsoft.com>
> ---
>  drivers/net/ethernet/microsoft/mana/mana_en.c | 22 ++++++++++
>  .../ethernet/microsoft/mana/mana_ethtool.c    | 40 ++++++++++++++++++-
>  include/net/mana/mana.h                       | 12 ++++++
>  3 files changed, 72 insertions(+), 2 deletions(-)
>=20
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c
> b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index 59287c6e6cee..b27ee6684936 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> @@ -224,6 +224,7 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb,
> struct net_device *ndev)
>  	int gso_hs =3D 0; /* zero for non-GSO pkts */
>  	u16 txq_idx =3D skb_get_queue_mapping(skb);
>  	struct gdma_dev *gd =3D apc->ac->gdma_dev;
> +	struct mana_pcpu_stats *pcpu_stats;
>  	bool ipv4 =3D false, ipv6 =3D false;
>  	struct mana_tx_package pkg =3D {};
>  	struct netdev_queue *net_txq;
> @@ -234,6 +235,8 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb,
> struct net_device *ndev)
>  	struct mana_cq *cq;
>  	int err, len;
>=20
> +	pcpu_stats =3D this_cpu_ptr(apc->pcpu_stats);
> +
>  	if (unlikely(!apc->port_is_up))
>  		goto tx_drop;
>=20
> @@ -412,6 +415,12 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb,
> struct net_device *ndev)
>  	tx_stats->bytes +=3D len;
>  	u64_stats_update_end(&tx_stats->syncp);
>=20
> +	/* Also update the per-CPU stats */
> +	u64_stats_update_begin(&pcpu_stats->syncp);
> +	pcpu_stats->tx_packets++;
> +	pcpu_stats->tx_bytes +=3D len;
> +	u64_stats_update_end(&pcpu_stats->syncp);
> +
>  tx_busy:
>  	if (netif_tx_queue_stopped(net_txq) && mana_can_tx(gdma_sq)) {
>  		netif_tx_wake_queue(net_txq);
> @@ -425,6 +434,9 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb,
> struct net_device *ndev)
>  	kfree(pkg.sgl_ptr);
>  tx_drop_count:
>  	ndev->stats.tx_dropped++;
> +	u64_stats_update_begin(&pcpu_stats->syncp);
> +	pcpu_stats->tx_dropped++;
> +	u64_stats_update_end(&pcpu_stats->syncp);
>  tx_drop:
>  	dev_kfree_skb_any(skb);
>  	return NETDEV_TX_OK;
> @@ -1505,6 +1517,8 @@ static void mana_rx_skb(void *buf_va, bool
> from_pool,
>  	struct mana_stats_rx *rx_stats =3D &rxq->stats;
>  	struct net_device *ndev =3D rxq->ndev;
>  	uint pkt_len =3D cqe->ppi[0].pkt_len;
> +	struct mana_pcpu_stats *pcpu_stats;
> +	struct mana_port_context *apc;
>  	u16 rxq_idx =3D rxq->rxq_idx;
>  	struct napi_struct *napi;
>  	struct xdp_buff xdp =3D {};
> @@ -1512,6 +1526,9 @@ static void mana_rx_skb(void *buf_va, bool
> from_pool,
>  	u32 hash_value;
>  	u32 act;
>=20
> +	apc =3D netdev_priv(ndev);
> +	pcpu_stats =3D this_cpu_ptr(apc->pcpu_stats);
> +
>  	rxq->rx_cq.work_done++;
>  	napi =3D &rxq->rx_cq.napi;
>=20
> @@ -1570,6 +1587,11 @@ static void mana_rx_skb(void *buf_va, bool
> from_pool,
>  		rx_stats->xdp_tx++;
>  	u64_stats_update_end(&rx_stats->syncp);
>=20
> +	u64_stats_update_begin(&pcpu_stats->syncp);
> +	pcpu_stats->rx_packets++;
> +	pcpu_stats->rx_bytes +=3D pkt_len;
> +	u64_stats_update_end(&pcpu_stats->syncp);
> +
>  	if (act =3D=3D XDP_TX) {
>  		skb_set_queue_mapping(skb, rxq_idx);
>  		mana_xdp_tx(skb, ndev);
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> index ab2413d71f6c..e3aa47ead601 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> @@ -83,8 +83,9 @@ static int mana_get_sset_count(struct net_device *ndev,
> int stringset)
>  	if (stringset !=3D ETH_SS_STATS)
>  		return -EINVAL;
>=20
> -	return ARRAY_SIZE(mana_eth_stats) + num_queues *
> -				(MANA_STATS_RX_COUNT + MANA_STATS_TX_COUNT);
> +	return ARRAY_SIZE(mana_eth_stats) +
> +	       (num_queues * (MANA_STATS_RX_COUNT + MANA_STATS_TX_COUNT)) +
> +	       (num_present_cpus() * (MANA_STATS_RX_PCPU +
> MANA_STATS_TX_PCPU));
>  }
>=20
>  static void mana_get_strings(struct net_device *ndev, u32 stringset, u8
> *data)
> @@ -139,6 +140,19 @@ static void mana_get_strings(struct net_device
> *ndev, u32 stringset, u8 *data)
>  		sprintf(p, "tx_%d_mana_map_err", i);
>  		p +=3D ETH_GSTRING_LEN;
>  	}
> +
> +	for (i =3D 0; i < num_present_cpus(); i++) {
> +		sprintf(p, "cpu%d_rx_packets", i);
> +		p +=3D ETH_GSTRING_LEN;
> +		sprintf(p, "cpu%d_rx_bytes", i);
> +		p +=3D ETH_GSTRING_LEN;
> +		sprintf(p, "cpu%d_tx_packets", i);
> +		p +=3D ETH_GSTRING_LEN;
> +		sprintf(p, "cpu%d_tx_bytes", i);
> +		p +=3D ETH_GSTRING_LEN;
> +		sprintf(p, "cpu%d_tx_dropped", i);
> +		p +=3D ETH_GSTRING_LEN;
> +	}
>  }
>=20
>  static void mana_get_ethtool_stats(struct net_device *ndev,
> @@ -222,6 +236,28 @@ static void mana_get_ethtool_stats(struct net_device
> *ndev,
>  		data[i++] =3D csum_partial;
>  		data[i++] =3D mana_map_err;
>  	}
> +
> +	for_each_possible_cpu(q) {
> +		const struct mana_pcpu_stats *pcpu_stats =3D
> +				per_cpu_ptr(apc->pcpu_stats, q);
> +		u64 rx_packets, rx_bytes, tx_packets, tx_bytes, tx_dropped;
> +		unsigned int start;
> +
> +		do {
> +			start =3D u64_stats_fetch_begin(&pcpu_stats->syncp);
> +			rx_packets =3D pcpu_stats->rx_packets;
> +			tx_packets =3D pcpu_stats->tx_packets;
> +			rx_bytes =3D pcpu_stats->rx_bytes;
> +			tx_bytes =3D pcpu_stats->tx_bytes;
> +			tx_dropped =3D pcpu_stats->tx_dropped;
> +		} while (u64_stats_fetch_retry(&pcpu_stats->syncp, start));
> +
> +		data[i++] =3D rx_packets;
> +		data[i++] =3D rx_bytes;
> +		data[i++] =3D tx_packets;
> +		data[i++] =3D tx_bytes;
> +		data[i++] =3D tx_dropped;
> +	}
>  }
>=20
>  static int mana_get_rxnfc(struct net_device *ndev, struct ethtool_rxnfc
> *cmd,
> diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
> index 76147feb0d10..9a2414ee7f02 100644
> --- a/include/net/mana/mana.h
> +++ b/include/net/mana/mana.h
> @@ -51,6 +51,8 @@ enum TRI_STATE {
>  /* Update this count whenever the respective structures are changed */
>  #define MANA_STATS_RX_COUNT 5
>  #define MANA_STATS_TX_COUNT 11
> +#define MANA_STATS_RX_PCPU 2
> +#define MANA_STATS_TX_PCPU 3
>=20
>  struct mana_stats_rx {
>  	u64 packets;
> @@ -386,6 +388,15 @@ struct mana_ethtool_stats {
>  	u64 rx_cqe_unknown_type;
>  };
>=20
> +struct mana_pcpu_stats {
> +	u64 rx_packets;
> +	u64 rx_bytes;
> +	u64 tx_packets;
> +	u64 tx_bytes;
> +	u64 tx_dropped;
> +	struct u64_stats_sync syncp;
> +};
> +
>  struct mana_context {
>  	struct gdma_dev *gdma_dev;
>=20
> @@ -449,6 +460,7 @@ struct mana_port_context {
>  	bool port_st_save; /* Saved port state */
>=20
>  	struct mana_ethtool_stats eth_stats;
> +	struct mana_pcpu_stats __percpu *pcpu_stats;

Where are pcpu_stats alloc-ed?
Seems I cannot see any alloc in the patch.

Thanks,
- Haiyang