Received-SPF: pass (google.com: domain of linux-kernel+bounces-79268-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
References: <20240223192658.45893-1-rrameshbabu@nvidia.com>
 <20240223192658.45893-2-rrameshbabu@nvidia.com>
 <abbff26c-c626-42ce-82a9-4dc983372de3@intel.com>
User-agent: mu4e 1.10.8; emacs 28.2
From: Rahul Rameshbabu <rrameshbabu@nvidia.com>
To: Jacob Keller <jacob.e.keller@intel.com>
Cc: Saeed Mahameed <saeed@kernel.org>, Leon Romanovsky <leon@kernel.org>,
 "David S. Miller" <davem@davemloft.net>, Eric Dumazet
 <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni
 <pabeni@redhat.com>, Jonathan Corbet <corbet@lwn.net>, Richard Cochran
 <richardcochran@gmail.com>, Tariq Toukan <tariqt@nvidia.com>, Gal Pressman
 <gal@nvidia.com>, Vadim Fedorenko <vadim.fedorenko@linux.dev>, Andrew Lunn
 <andrew@lunn.ch>, Heiner Kallweit <hkallweit1@gmail.com>, Przemek Kitszel
 <przemyslaw.kitszel@intel.com>, Ahmed  Zaki <ahmed.zaki@intel.com>,
 Alexander Lobakin <aleksander.lobakin@intel.com>, Hangbin Liu
 <liuhangbin@gmail.com>, Paul  Greenwalt <paul.greenwalt@intel.com>, Justin
 Stitt <justinstitt@google.com>, Randy Dunlap <rdunlap@infradead.org>,
 Maxime Chevallier <maxime.chevallier@bootlin.com>, Kory Maincent
 <kory.maincent@bootlin.com>, Wojciech Drewek <wojciech.drewek@intel.com>,
 Vladimir Oltean <vladimir.oltean@nxp.com>, Jiri Pirko <jiri@resnulli.us>,
 Alexandre Torgue <alexandre.torgue@foss.st.com>, Jose Abreu
 <joabreu@synopsys.com>, Dragos  Tatulea <dtatulea@nvidia.com>,
 netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
 linux-doc@vger.kernel.org
Subject: Re: [PATCH RFC net-next v1 1/6] ethtool: add interface to read Tx
 hardware timestamping statistics
Date: Fri, 23 Feb 2024 14:21:12 -0800
In-reply-to: <abbff26c-c626-42ce-82a9-4dc983372de3@intel.com>
Message-ID: <875xyex10q.fsf@nvidia.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?2RSUEXLupRPDV+aq0OxwzCc1aZOaecQFNaQmvm2UYGtBAgiC6R1JZ0T1GF3F?=
 =?us-ascii?Q?PeAhzQQ19L86nT3R/Lv3TSlTn6USg1a5iDA6qiKuxUeUSvvbkqrk6CycLZmu?=
 =?us-ascii?Q?LXZYAuh0kj8iJUP1sW7G2o+VQzCyj45UHIApZJPpdklOrCnicaSnPA6vz0Ho?=
 =?us-ascii?Q?p/BuOfE1jhz/5yPfWiDF6nVPTpmCQ/I+RJw/APnDQ+hcpt9ke9iSJiPKfzSq?=
 =?us-ascii?Q?rIPzSqD8q37zxEHWzxJsiWHBoZx/oxWXglHE3gC2P1+ENvmZA/jy5SkMjyEo?=
 =?us-ascii?Q?KpesfzwBMG1oZ9qtPTdbKQfFf7LevmQyD2grH4B1CTJE70UFehoWvXmmb2NM?=
 =?us-ascii?Q?yXvM5V3xcDUQ9LlETBWXpOoQoeFdyQBOkBFfAXyjAqEcMM4NrhrRhjRAoA6E?=
 =?us-ascii?Q?pkOFYzbd0BnN2SW/6o0Qp5Hu+RA1cgamWZ2Ox6JrN5KGJKt1cG2U/Z7yFcP3?=
 =?us-ascii?Q?Pf23MuViODv/ayb/rcD3bGAxy5iOJv+jlx4ey23kPf9VQypIDug2omxHOkEq?=
 =?us-ascii?Q?HQS2xOdONjk265/mg6y8wlX4qS8lOUX50xcz/TDglOzYwSVqKLFvemFmuq+E?=
 =?us-ascii?Q?+Cp/z8wMaovDLz87rXPKLsrGQGI/PhXbeucOYn4mFGM+n7/2rFoapzHMzwl/?=
 =?us-ascii?Q?KzhXDbQa3PomM8tAfE60BMyNoJNzLQCpZk5H6MyJcnbi+iM8Evk9bfldwlxE?=
 =?us-ascii?Q?Nbi7EcdJwPtBFlivgG32jlCaV04lTTaGFJE8SbMactK60uiNP20b03mT0ht4?=
 =?us-ascii?Q?QO4tmyg3MIF/EvHr4yAGSMgnBeFPh+sODrh/5dvDzY1bwxjmnZa92skV9An4?=
 =?us-ascii?Q?LVUjTmvW81Ix0LsQ69l2QJ/4qq174gE6d/gPSTYocg6d9+I7RoETwLLBqgXr?=
 =?us-ascii?Q?vcFnNIjfDBzO9GVAl6nx5DSUGHj+ku9AqScSV1oLVBDlXKEQ4uTOW4/wX/dc?=
 =?us-ascii?Q?rQwbtvnZksfmHHnUiY0tlWjsR6c5J+/aVyPooye8zgmaXyRcc+cDxK7f5Bjw?=
 =?us-ascii?Q?gef+oEfHnpwbzelGdnMjug8vyOGTCgPY5fydSNjTb6meqWvpKzd7zHuzHKBz?=
 =?us-ascii?Q?t/t17cDealAg+IHgp7qlnXWrOkLqizHCNP/bYRf0480FYzwCC4bME+Nxo/c5?=
 =?us-ascii?Q?5k9BL91JYkWFJc5/VQzft1cU97taVHnb5kljludJwZd3JozmvPjjU+okrSGT?=
 =?us-ascii?Q?pEj3v7xMbntBAzSU2l7ITvx5qGim+3ZP+9E3l92ilEdBLe5Srf8N1LkLP3Pa?=
 =?us-ascii?Q?SIfS7wX0fxrlP9vLMULX5t4zA5V3G17tk94HC+bSqNoHE67MGa4N2t15edvt?=
 =?us-ascii?Q?k5Kx/nBtuteuJLOyGgOoMrLGDoS4LNDjCJq7qmP1tLe77UbE3BiCRMk4rWQN?=
 =?us-ascii?Q?qo1RTCOeySULTC7Z2z+Kxk7CLwuZJuGTiqGcnaMhRxq9lEWB3wGq6BS5uf+H?=
 =?us-ascii?Q?6PL1aD8VVAEyAeBIxJ12Duj9Wl5SfwUUsqDY/bSE6nxrSJr7LlrNwBE7Ohgv?=
 =?us-ascii?Q?C8dzaFhB5fDW3shrGMQM2sSCpdavV34gp24riBUVTv88k5tkWT3dzGGwSI/s?=
 =?us-ascii?Q?9xsZApr8I1NuuWGxn35RC0cCl71emX1QH7bgi08S5EWyuUgikE0o/OM1UGa8?=
 =?us-ascii?Q?cw=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dbd88224-7109-44a1-d4f8-08dc34bfac7c
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2743.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2024 22:35:01.8676
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dMWX30FdNvwIVIDmtT2Gv/QyLyUpyGMQdZQXwC8PAh7X4G/jpYzTqRiGHsX1O+Ntb6d6YTWHy74ZwZ8k1rKMMA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7847

On Fri, 23 Feb, 2024 13:07:08 -0800 Jacob Keller <jacob.e.keller@intel.com> wrote:
> On 2/23/2024 11:24 AM, Rahul Rameshbabu wrote:
>> +/**
>> + * struct ethtool_ts_stats - HW timestamping statistics
>> + * @layer: input field denoting whether stats should be queried from the DMA or
>> + *        PHY timestamping layer. Defaults to the active layer for packet
>> + *        timestamping.
>> + * @tx_stats: struct group for TX HW timestamping
>> + *	@pkts: Number of packets successfully timestamped by the queried
>> + *	      layer.
>> + *	@lost: Number of packet timestamps that failed to get applied on a
>> + *	      packet by the queried layer.
>> + *	@late: Number of packet timestamps that were delivered by the
>> + *	      hardware but were lost due to arriving too late.
>> + *	@err: Number of timestamping errors that occurred on the queried
>> + *	     layer.
>> + */
>> +struct ethtool_ts_stats {
>> +	enum ethtool_ts_stats_layer layer;
>> +	struct_group(tx_stats,
>> +		u64 pkts;
>> +		u64 lost;
>> +		u64 late;
>> +		u64 err;
>> +	);
>> +};
>
> The Intel ice drivers has the following Tx timestamp statistics:
>
> tx_hwtstamp_skipped - indicates when we get a Tx timestamp request but
> are unable to fulfill it.
> tx_hwtstamp_timeouts - indicates we had a Tx timestamp skb waiting for a
> timestamp from hardware but it didn't get received within some internal
> time limit.

This is interesting. In mlx5 land, the only case where we are unable to
fulfill a hwtstamp is when the timestamp information is lost or late.

lost for us means that the timestamp never arrived within some internal
time limit that our device will supposedly never be able to deliver
timestamp information after that point.

late for us is that we got hardware timestamp information delivered
after that internal time limit. We are able to track this by using
identifiers in our completion events and we only release references to
these identifiers upon delivery (never delivering leaks the references.
Enough build up leads to a recovery flow). The theory for us is that
late timestamp information arrival after that period of time should not
happen. However the truth is that it does happen and we want our driver
implementation to be resilient to this case rather than trusting the
time interval.

Do you have any example of a case of skipping timestamp information that
is not related to lack of delivery over time? I am wondering if this
case is more like a hardware error or not. Or is it more like something
along the lines of being busy/would impact line rate of timestamp
information must be recorded?

> tx_hwtstamp_flushed - indicates that we flushed an outstanding timestamp
> before it completed, such as if the link resets or similar.
> tx_hwtstamp_discarded - indicates that we obtained a timestamp from
> hardware but were unable to complete it due to invalid cached data used
> for timestamp extension.
>
> I think these could be translated roughly to one of the lost, late, or
> err stats. I am a bit confused as to how drivers could distinguish
> between lost and late, but I guess that depends on the specific hardware
> design.
>
> In theory we could keep some of these more detailed stats but I don't
> think we strictly need to be as detailed as the ice driver is.

We also converged a statistic in the mlx5 driver to the simple error
counter here. I think what makes sense is design specific counters
should be exposed as driver specific counters and more common counters
should be converged into the ethtool_ts_stats struct.

>
> The only major addition I think is the skipped stat, which I would
> prefer to have. Perhaps that could be tracked in the netdev layer by
> checking whether the skb flags to see whether or not the driver actually
> set the appropriate flag?

I guess the problem is how would the core stack know at what layer this
was skipped at (I think Kory's patch series can be used to help with
this since it's adding a common interface in ethtool to select the
timestamping layer). As of today, mlx5 is the only driver I know of that
supports selecting between the DMA and PHY layers for timestamp
information.

>
> I think i can otherwise translate the flushed status to the lost
> category, the timeout to the late category, and everything else to the
> error category. I can easily add a counter to track completed timestamps
> as well.

Sounds good.

--
Thanks,

Rahul Rameshbabu