Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <83ef6401-8736-8416-c898-2fbbb786726e@intel.com>
Date:   Mon, 3 Jul 2023 18:47:49 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.12.0
Subject: Re: [PATCH V4 net] net: mana: Fix MANA VF unload when host is
 unresponsive
Content-Language: en-US
To:     souradeep chakrabarti <schakrabarti@linux.microsoft.com>
CC:     <kys@microsoft.com>, <haiyangz@microsoft.com>,
        <wei.liu@kernel.org>, <decui@microsoft.com>, <davem@davemloft.net>,
        <edumazet@google.com>, <kuba@kernel.org>, <pabeni@redhat.com>,
        <longli@microsoft.com>, <sharmaajay@microsoft.com>,
        <leon@kernel.org>, <cai.huoqing@linux.dev>,
        <ssengar@linux.microsoft.com>, <vkuznets@redhat.com>,
        <tglx@linutronix.de>, <linux-hyperv@vger.kernel.org>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <linux-rdma@vger.kernel.org>, <stable@vger.kernel.org>,
        <schakrabarti@microsoft.com>
References: <1688374171-10534-1-git-send-email-schakrabarti@linux.microsoft.com>
From:   Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <1688374171-10534-1-git-send-email-schakrabarti@linux.microsoft.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OWd3djF6SVlpc2YzS05pUm1VYWN4OXBsZjZWTjBsV2czQXc3dTdEc29NVE1n?=
 =?utf-8?B?U0Z6U0huN2x0ODd0SGRBK05EZ2hFcGcremlJT0kydlBYQUtWTTdpcWlvODlT?=
 =?utf-8?B?N2RCVTJIVkR1OVBob1AzN1E2MkxEMjRkOGJYdVI3c3BDMHVteWloVXFvSEc4?=
 =?utf-8?B?NWxVSERyZG91dUJrU29uUDdPVUtXN0V1QlhpS1oxSGtOdk9aaU5OSkVMcHhi?=
 =?utf-8?B?WW9IdS9Ca0NTSUgrQTFuZXVxOE1USUpVM3NRWkNJbkcyblJwUTlQY211Q29G?=
 =?utf-8?B?NXA0eVhHUHJvSkx2dTVPZTdMN2xpTlJhRXBxSXI5RFA3bm4yZCswNFJabVZO?=
 =?utf-8?B?clptdlkzVXpUamJkZXpJZlJ6ajZQUjd0bzl4SzZJTEF6VmV2K0tEY0ZYOFVa?=
 =?utf-8?B?TDFiRVhORHlyNW0rM20vS05VUUtMTU9yc3B5cEUwMW9TTG9iNm9yU2RMN2hr?=
 =?utf-8?B?WXUwZ1N6bUZVbTl2UElkbzhwTU5BN2czRzhlMEZzcEliV0ZITS8veWVxQmVO?=
 =?utf-8?B?cmFKUzM4bnVFSHVmdkJ3dXZUOXhaMHlLNkpnSVoxVng5ejBjaHVVdys0bHhE?=
 =?utf-8?B?a3QvWDV4YjRIcW1jOU9MdlJUTi9JRkhIVTdqZStIbFZwSXlkTlE0eHNpVjhV?=
 =?utf-8?B?bnBLbzRHemNGSURZR256NjM4TjhBNk9LTHRIVURDTmw3QmtKTzU0TngvcGdp?=
 =?utf-8?B?RUZxSWJiblhNLzFrL3JSNkFzWkJ5cXo5WTlhSjEwMGwwNXJndXEwN2tsR0dJ?=
 =?utf-8?B?YTVnNjF5aHdzd1plMHdkUU5jLzlNN1NDL0FRTDVZb20yMHNtYjRxTy9hR2kr?=
 =?utf-8?B?eG1VaXlYS28yaEZBZnd3TFREUFY5SEUwNWFHc0NkUFl5b0tNQ0dVSkhybGk4?=
 =?utf-8?B?YzdFNjRhZm1TWXV4VTB2MjN2ZjNpdGxMRFJsYlRLcWNjci82N3o2c0dra1BC?=
 =?utf-8?B?WWYrN0tycGwrUzZQak83QnRHUUlYM0RVdHJyUllQZURhRGs5RGZRd05RYlFI?=
 =?utf-8?B?SXBzSVRwckFzTjlXcDA2QnBCdW5rb1o5OTFuZ1NtTmlJYVFmVzArdForZEt1?=
 =?utf-8?B?TGpTRGVPeHo1UjhyM2lWQ0I3MlRJeFhtN0ltVlFrdDlFaTFpMnBYNVRCa285?=
 =?utf-8?B?UzJYY3RtM2YyZ1gzbmFIMTFTT1FhOFZvNGNTTVVHWFlOQUU5Z0xPL2lRWlhT?=
 =?utf-8?B?cXV1ZE5LbzZ4WlRJMlVEaXlPeHVEZGt2SFIrZm5COXB0elhiVnZEN3dLOUtU?=
 =?utf-8?B?cUozSzZiMmZ5MkpRNE5KSVpER05XZ3pTcDhQOW5oL2NSMURFM2hNTEpZNGY5?=
 =?utf-8?B?ZGprLzB2OVVHaS96b2d2bHFEK3dLQXN0a2FPOTJaYVIyUTNhV3lOUlkrK0Na?=
 =?utf-8?B?cGpJaHpUYVIveWUvWWRZaDBLdy9vNWVaWGgvM2pBcTFPVUZkTDVnYmdvMTJV?=
 =?utf-8?B?QVBMcU43TUpMaEhPdUZJelpqN2tyY0doMUpQMHNhcTB4d0dhRzdUbDA3Ulkr?=
 =?utf-8?B?aURMenFmaEY5b2dkaGZWVkRXTFd4MjVZMkxZN0Uwd0hWa3VVR3JNMkVmdmdD?=
 =?utf-8?B?N0MrZDNrZ3BBZ2VaU0pFVXpzM2lWRVdtaGpNUUpBamU0aVZWeXlLdXFST3A0?=
 =?utf-8?B?SDRYeEpwc1JqajIycDZvcEhGTTRtcHJvZmpDQkx5aE9nMFM0RjhYeHdpTzFC?=
 =?utf-8?B?elhkcmoyaVM3TnpZUko4enVKdVE3dnFDQWM5bTlFRDFJemkyc3BFL3BSNFlT?=
 =?utf-8?B?enlqbnMxcG00V1lEYVkxcnE3aEd4azJDckdXUHRJaEwyU2tYS2VJYVlaNUxP?=
 =?utf-8?B?N01vT3F4RmF6cFFRVGlheGtQVzN3MXE1bk1RTlIvNjhPZktwdnpoMVFvNXow?=
 =?utf-8?B?c2ZqMzhCZGZINDR2RTZsVkNZa2JXcXVGcTRablRaYWN0U0FMQXVLc2FaaGVh?=
 =?utf-8?B?ZTAwNU96Sm96RDNCMDRSd1VSY0dBbUZXaVMweXgwYnhJZDFjakQxTzJ2Snlp?=
 =?utf-8?B?S0xtSXBwSVFQRjRkRWxxaGFNdDRLTks2ZGg0R2lIQkhONkNzamhXN1lFT3RN?=
 =?utf-8?B?YkNGREFMUUQycnJOWWxOMnFtSE93OXNWcFpHWXpUdnNpQkJ0Ui94SGdBZ081?=
 =?utf-8?B?QTE1UzZYUklxS2ZQUDlJbXdJdXdjNktNV1o4MTBwL0x5ektCMy9LNitBdlYz?=
 =?utf-8?B?SUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 893acd2d-26d5-4fae-4cef-08db7be56274
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3625.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jul 2023 16:48:53.7600
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: evbX9nlrHpMy2AP8ClxrQv7x9EvMEeZoJXAn0w2sp/agJwxfErqYAg7gSYp73f4PmfkAU86RFLex2zBIui7d58bure70JCM1O+aQEnxcE04=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB6731
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com>
Date: Mon,  3 Jul 2023 01:49:31 -0700

> From: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com>

Please sync your Git name and Git mail account settings, so that your
own patches won't have "From:" when sending. From what I see, you need
to correct first letters of name and surname to capital in the Git email
settings block.

> 
> When unloading the MANA driver, mana_dealloc_queues() waits for the MANA
> hardware to complete any inflight packets and set the pending send count
> to zero. But if the hardware has failed, mana_dealloc_queues()
> could wait forever.
> 
> Fix this by adding a timeout to the wait. Set the timeout to 120 seconds,
> which is a somewhat arbitrary value that is more than long enough for
> functional hardware to complete any sends.
> 
> Signed-off-by: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com>

Where's "Fixes:" tagging the blamed commit?

> ---
> V3 -> V4:
> * Fixed the commit message to describe the context.
> * Removed the vf_unload_timeout, as it is not required.
> ---
>  drivers/net/ethernet/microsoft/mana/mana_en.c | 26 ++++++++++++++++---
>  1 file changed, 23 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index a499e460594b..d26f1da70411 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> @@ -2346,7 +2346,10 @@ static int mana_dealloc_queues(struct net_device *ndev)
>  {
>  	struct mana_port_context *apc = netdev_priv(ndev);
>  	struct gdma_dev *gd = apc->ac->gdma_dev;
> +	unsigned long timeout;
>  	struct mana_txq *txq;
> +	struct sk_buff *skb;
> +	struct mana_cq *cq;
>  	int i, err;
>  
>  	if (apc->port_is_up)
> @@ -2363,15 +2366,32 @@ static int mana_dealloc_queues(struct net_device *ndev)
>  	 * to false, but it doesn't matter since mana_start_xmit() drops any
>  	 * new packets due to apc->port_is_up being false.
>  	 *
> -	 * Drain all the in-flight TX packets
> +	 * Drain all the in-flight TX packets.
> +	 * A timeout of 120 seconds for all the queues is used.
> +	 * This will break the while loop when h/w is not responding.
> +	 * This value of 120 has been decided here considering max
> +	 * number of queues.
>  	 */
> +
> +	timeout = jiffies + 120 * HZ;

Why not initialize it right when declaring?

>  	for (i = 0; i < apc->num_queues; i++) {
>  		txq = &apc->tx_qp[i].txq;
> -
> -		while (atomic_read(&txq->pending_sends) > 0)
> +		while (atomic_read(&txq->pending_sends) > 0 &&
> +		       time_before(jiffies, timeout)) {
>  			usleep_range(1000, 2000);> +		}
>  	}

120 seconds by 2 msec step is 60000 iterations, by 1 msec is 120000
iterations. I know usleep_range() often is much less precise, but still.
Do you really need that much time? Has this been measured during the
tests that it can take up to 120 seconds or is it just some random value
that "should be enough"?
If you really need 120 seconds, I'd suggest using a timer / delayed work
instead of wasting resources.

>  
> +	for (i = 0; i < apc->num_queues; i++) {
> +		txq = &apc->tx_qp[i].txq;
> +		cq = &apc->tx_qp[i].tx_cq;

cq can be just &txq->tx_cq.

> +		while (atomic_read(&txq->pending_sends)) {
> +			skb = skb_dequeue(&txq->pending_skbs);
> +			mana_unmap_skb(skb, apc);
> +			napi_consume_skb(skb, cq->budget);

(you already have comment about this one)

> +			atomic_sub(1, &txq->pending_sends);
> +		}
> +	}
>  	/* We're 100% sure the queues can no longer be woken up, because
>  	 * we're sure now mana_poll_tx_cq() can't be running.
>  	 */

Thanks,
Olek