Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <2bdeb975-6d45-67bb-3017-f19df62fe7af@intel.com>
Date:   Tue, 17 Jan 2023 10:46:20 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [Intel-wired-lan] [PATCH] ice/ptp: fix the PTP worker retrying
 indefinitely if the link went down
Content-Language: en-US
To:     Daniel Vacek <neelx@redhat.com>,
        Jesse Brandeburg <jesse.brandeburg@intel.com>,
        Tony Nguyen <anthony.l.nguyen@intel.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        "Richard Cochran" <richardcochran@gmail.com>,
        "Kolacinski, Karol" <karol.kolacinski@intel.com>,
        Siddaraju <siddaraju.dh@intel.com>,
        "Michalik, Michal" <michal.michalik@intel.com>
CC:     <netdev@vger.kernel.org>, <intel-wired-lan@lists.osuosl.org>,
        <linux-kernel@vger.kernel.org>
References: <20230117181533.2350335-1-neelx@redhat.com>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <20230117181533.2350335-1-neelx@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NzhyZklwdlB2Ym1YTzdoVS9jbCtJa3N5WG81SmVjWkZUcnY1Y1RURWtPT1dU?=
 =?utf-8?B?VWZ0SGZKaU9PQ29XKzI2TDV0dGREakJEZVRZYTJHUEpNbnJUWDBEdStKYzZE?=
 =?utf-8?B?ZGxMcStvSmJjUUF3SHNtamxUMEtPQnpkeWU4dnQySVpwNVhEeEVxQWZoM1Ew?=
 =?utf-8?B?Q0tHc2JlZlNWMWVuTWlhTTdKM2dpcWJLaXZtbDZjc24xZDNJdXhEM3V1M1A3?=
 =?utf-8?B?R1VNZmdYWDVsaXJudFNqVHVBdWFkdFJYQzFhS25HQmdDcnQ3ZmhTV1pmVWx3?=
 =?utf-8?B?K1J3cFRMSFpEMDNDTmR4SjhyU21VM2pCT0pMemZFQ2U0UjJ2NmZiczJnZDla?=
 =?utf-8?B?MG1POUVOcm4raWRScnJvay9jU1ZLUDlLOCtpSjNBUTNGd0dOblFkZ0ZVVEdS?=
 =?utf-8?B?RXgzU0dVSXAwL2xOZDhxQUtnT3gwY041V3hJMGMxL0pOTWhlQS80TFFna3kz?=
 =?utf-8?B?cUdQbnZmTFF1aXljMHNjM2hoM25EMnB1ZDdFNkVWMDc5TTV0UHBtS0NkdmVS?=
 =?utf-8?B?VmQyNWh4THBCQm1pYXlFV05uRVZGNTV4RXp0d0FNZm5VWFg0a3h2cUxiV2h0?=
 =?utf-8?B?RTNIK2J2S1RpVVVnSTdVeWVrR1lob0ZhT0VSM3kvblRJdUkxSldFcUViOGhq?=
 =?utf-8?B?cWl5cW56UVBGK0hiZGxHL2hiWDFtRXVSYnAwVkRvNVZZVGRDNFhIRWRXZUZl?=
 =?utf-8?B?R0NWUU5SZkNqQlM3Q3dBK1BMV05YaHFGMnljYmhUTGJMYkZUZ1Q0OTlQK1Vn?=
 =?utf-8?B?QlZkYUIrQTNyTzNXZlRYUWhHK0Z1WU9ON2ZXSFRyZW5zTmZxSmJMV1pFOGtC?=
 =?utf-8?B?OWdnenF4VnBJdFpCNS9iNUFQNlo4VysrSTBRTlc0Yzg2bjQ2R1RTdHRhU0dJ?=
 =?utf-8?B?RWtyaVo2M25hMEN1OU81QnFZZVhSQmJyOTFDNDlvRk5mdWh6T3ErcU5scitU?=
 =?utf-8?B?cHFsWlYrZU53b2xkbTJTb2M1a3RWdlNJQmQ5cXRqeURqNXpSTlBYNXJHSTlk?=
 =?utf-8?B?MklWQ0hjb3RXOWJIZmpxV0dhem5COExNWmVyN2JDZ25NZ0JzMm5MMjNmcGhu?=
 =?utf-8?B?S0hwdThzdVlnKzRZUFVSMzNjUS9MTmtBaitFOUNRTG5jdGNHMmZwNGlCK09w?=
 =?utf-8?B?b3d2OGFmUHhRYkpSYW4xa0lJOWMvcGo3cE5nYzRlWkQxTmpVdHVxWjFwOG9S?=
 =?utf-8?B?WGF2Z2gza0VKRTJTYlAxbGVjaVVseG9VSzNxdW9acXlSNTBHbEtIZENidXhh?=
 =?utf-8?B?ZmlMdXU2K29ZbU16QlJMdDBCeFRTRE41ZzltT3l6WHhuZ0w4YkNnQ1FzcUFN?=
 =?utf-8?B?SS9QVVNYYXFKRHptbGNnWmM0bFRudVpJVGhUcUZ3TnpLUUM0MmZOb05SbUhx?=
 =?utf-8?B?dHM4bTNXMmR2MkhGUXdNZHdPUGpMRE9QQlYveGRjUEozMHpJVXlRdTJ3SXY2?=
 =?utf-8?B?K0JZbEJYWjZ0bWs4bGpaQmwrVU0vMmZEbnlhSVpGMlQ4SlFBWG1wRHZ1TzUx?=
 =?utf-8?B?S2FSand4RHZad2U3Um9jaVk2VzFOMFpyUmM4SnVZSGtrdkppTm5vcDFEa1FP?=
 =?utf-8?B?b01xaEVzWWo4alZ3L1pwcW1oVEcyUklOMDRzcWlWa1RLY0xiMUt4M21PMHpl?=
 =?utf-8?B?MGRQUnBsS0RKMVpBZmFJMnUxSUkvbFlaK0NiNC9mbG1rWW91SDhuckoyZHYx?=
 =?utf-8?B?clVJVGFEWFJMencxNCtSTWEreFlzYVFzWGNZc2piaHlUeWpyK0JTa253Yjlz?=
 =?utf-8?B?akRZQngzb2V6RGIvc20rYkoxU21lM21hTkxOWkk2ZmFHRGdHT3J1TkV1bHI2?=
 =?utf-8?B?SklnS1Vwb1NrditvZGtaOVdmZ0tRc3UzNTNLVGJFeFpHUm5SalJ0dWhqcDMz?=
 =?utf-8?B?WXNBU0J4cHh5VnZ5ZkdmTTZYOFo0dks0ak14UWR0NnhOaWRiYkdBeFBkLy9v?=
 =?utf-8?B?dVNhQmdqc2szS3hpVlZXN1RkNllFMkdtOGhFbmNwTXVGeCs1MDlqT1hYRDM3?=
 =?utf-8?B?TW02VlZ2MGJFcHdrZlhIZHVUYXFXVVZjeTBYQ1NpT2F1VmNIZ0V6SnQ4dUE0?=
 =?utf-8?B?QW1MV0xwSTlEVlpMK3c0UDNscHBkN0dPRUJ3eE5uMjNZNkFHNTFpMW96SnNS?=
 =?utf-8?B?UTUwV0hXMCt1b3VxS1U5UUliQnV6TTczSkxYalRSNDFHSnlGTkl0VmFDZ2x2?=
 =?utf-8?B?RGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b1798855-4c74-47e2-abf4-08daf8bb2143
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 18:46:22.8564
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tTD0n+8NwSYa6eyOd7SAw3IgkFBXgo0aIKtfpECnLO52xvfuciyVpMPmHHFFMZKp0dhTSzu9h0WoTrAvQOlaAQsHCd9kjdoJmUpoSq7D57E=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6521
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 1/17/2023 10:15 AM, Daniel Vacek wrote:
> When the link goes down the ice_ptp_tx_tstamp_work() may loop forever trying
> to process the packets. In this case it makes sense to just drop them.
> 

Hi,

Do you have some more details on what situation caused this state? Or is
this just based on code review?

It won't loop forever because if link is down for more than 2 seconds
we'll discard the old timestamps which we assume are not going to arrive.

The trouble is that if a timestamp *does* arrive late, we need to ensure
that we never assign the captured time to the wrong packet, and that for
E822 devices we always read the correct number of timestamps (otherwise
we can get the logic for timestamp interrupt generation broken).

Consider for example this flow for e810:

1) a tx packet with a timestamp request is scheduled to hw
2) the packet begins being transmitted
3) link goes down
4) interrupt triggers, ice_ptp_tx_tstamp is run
5) link is down, so we skip reading this timestamp. Since we don't read
the timestamp, we just discard the skb and we don't update the cached tx
timestamp value
6) link goes up
7) 2 tx packets with a timestamp request are sent and one of them is on
the same index as the packet in (1)
8) the other tx packet completes and we get an interrupt
9) the loop reads both timestamps. Since the tx packet in slot (1)
doesn't match its cached value it looks "new" so the function reports
the old timestamp to the wrong packet.

Consider this flow for e822:

1) 2 tx packets with a timestamp request are scheduled to hw
2) the packets begin being transmitted
3) link goes down
4) an interrupt for the Tx timestamp is triggered, but we don't read the
timestamps because we have link down and we skipped the ts_read.
5) the internal e822 hardware counter is not decremented due to no reads
6) no more timestamp interrupts will be triggered by hardware until we
read the appropriate number of timestamps

I am not sure if link going up will properly reset and re-initialize the
Tx timestamp block but I suspect it does not. @Karol, @Siddaraju,
@Michal, do you recall more details on this?

I understand the desire to avoid polling when unnecessary, but I am
worried because the hardware and firmware interactions here are pretty
complex and its easy to get this wrong. (see: all of the previous
patches and bug fixes we've been working on... we got this wrong a LOT
already...)

Without a more concrete explanation of what this fixes I'm worried about
this change :(

At a minimum I think I would only set drop_ts but not not goto skip_ts_read.

That way if we happen to have a ready timestamp (for E822) we'll still
read it and avoid the miscounting from not reading a completed timestamp.

This also ensures that on e810 the cached timestamp value is updated and
that we avoid the other situation.

I'd still prefer if you have a bug report or more details on the failure
case. I believe even if we poll it should be no more than 2 seconds for
an old timestamp that never got sent to be discarded.

> Signed-off-by: Daniel Vacek <neelx@redhat.com>
> ---
>  drivers/net/ethernet/intel/ice/ice_ptp.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c
> index d63161d73eb16..c313177ba6676 100644
> --- a/drivers/net/ethernet/intel/ice/ice_ptp.c
> +++ b/drivers/net/ethernet/intel/ice/ice_ptp.c
> @@ -680,6 +680,7 @@ static bool ice_ptp_tx_tstamp(struct ice_ptp_tx *tx)
>  	struct ice_pf *pf;
>  	struct ice_hw *hw;
>  	u64 tstamp_ready;
> +	bool link_up;
>  	int err;
>  	u8 idx;
>  
> @@ -695,6 +696,8 @@ static bool ice_ptp_tx_tstamp(struct ice_ptp_tx *tx)
>  	if (err)
>  		return false;
>  
> +	link_up = hw->port_info->phy.link_info.link_info & ICE_AQ_LINK_UP;
> +
>  	for_each_set_bit(idx, tx->in_use, tx->len) {
>  		struct skb_shared_hwtstamps shhwtstamps = {};
>  		u8 phy_idx = idx + tx->offset;
> @@ -702,6 +705,12 @@ static bool ice_ptp_tx_tstamp(struct ice_ptp_tx *tx)
>  		bool drop_ts = false;
>  		struct sk_buff *skb;
>  
> +		/* Drop packets if the link went down */
> +		if (!link_up) {
> +			drop_ts = true;
> +			goto skip_ts_read;
> +		}
> +
>  		/* Drop packets which have waited for more than 2 seconds */
>  		if (time_is_before_jiffies(tx->tstamps[idx].start + 2 * HZ)) {
>  			drop_ts = true;