Date:   Mon, 13 Mar 2023 21:07:11 +0100
From:   Simon Horman <simon.horman@corigine.com>
To:     Jason Xing <kerneljasonxing@gmail.com>
Cc:     davem@davemloft.net, edumazet@google.com, kuba@kernel.org,
        pabeni@redhat.com, ast@kernel.org, daniel@iogearbox.net,
        hawk@kernel.org, john.fastabend@gmail.com, kuniyu@amazon.com,
        liuhangbin@gmail.com, xiangxia.m.yue@gmail.com, jiri@nvidia.com,
        andy.ren@getcruise.com, bpf@vger.kernel.org,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        Jason Xing <kernelxing@tencent.com>
Subject: Re: [PATCH net-next] net: introduce budget_squeeze to help us tune
 rx behavior
Message-ID: <ZA+CbyVQxsKQ4BLp@corigine.com>
References: <20230311163614.92296-1-kerneljasonxing@gmail.com>
 <CAL+tcoAwodpnE2NjMLPhBbmHUvmKMgSykqx0EQ4YZaQHjrx0Hw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAL+tcoAwodpnE2NjMLPhBbmHUvmKMgSykqx0EQ4YZaQHjrx0Hw@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Titwam9HOTlIN0hkcDZDM28wVUtuOExUUEFLSlk1cEd0TzBZZVQ0M0dyVHNj?=
 =?utf-8?B?ZjYyenVxOXIwcERrazlLS2tqaGc1Qk5aMmgvNWh2ZEZtb1RCR3o3SkJVc0Ny?=
 =?utf-8?B?K0RPWUFCdjhITC9OdkdiRjVOaVZjQWx5aU9JTWl5WXUyd1BQYm03SE5GemtF?=
 =?utf-8?B?Z1ZWcitRUm9kTWtrdStoSnB1Qy9zckk1NVRjOXp4MDIvc3VvcGt6R3VsWnRw?=
 =?utf-8?B?d08wMEVoTVBacno1UTNvU0dObEZqd0kwSlBITExlbksrQitoWllkMHZqZk1Y?=
 =?utf-8?B?b2ZhRUtsNzBOdkJyV2JLT2kwdEhtaVhuNG9lU1lsaVJrZnlySEtSR1NpSFlO?=
 =?utf-8?B?RU96dGxsNWh1L0syQnN6NlY2OGd2K01IZ3NKWWFqbGN6bW1YVkRueXp0MUla?=
 =?utf-8?B?RkpvUTY3REUrUk9QWWNqa2JEMitZd0xQVFVBSUN3N0E0eE44RFJVU3FpckM0?=
 =?utf-8?B?TUlqOVBsZ2FpWjV6NytEODJwRFUyOVdYODM1eTZubGQ4U1JJSnRBS1lLMkdP?=
 =?utf-8?B?b1BtRGxsRXNZbUpna3N6ZlhoQzZFZk5MdnRyZUVIYm96dnVvcU9xL0U5dy9v?=
 =?utf-8?B?ZXhsQ294TUdZazhnWExWNEI2UDNPNWtuTU55TmthZGxCVFA5ajBDREJSRTc3?=
 =?utf-8?B?K1VZYVhIdzhHUW9SaXNXKzJTeXg1a1AwSkhrVE1GMFFKVkExaGMwNDMySSta?=
 =?utf-8?B?Q292Unhha3hTMGJ2Mmt2Sk1mSjJHQmJjOWFBMTRQVnlHa0JXQjJMMlJUdzRF?=
 =?utf-8?B?VDA3Z0YxV3JwSzZ6WFZkcCs3YmVUUnN4ZHl1cUpkSXpTWE50R1dFbnVQcVBu?=
 =?utf-8?B?clY4cVUvdWY4S0pxcjMzWWJFYlhhUzdXdU0wSFdMaTBDS3lqY2dDTEI4STVL?=
 =?utf-8?B?Y2ZJNVNwSHl2SmlBYm43RnVpTnVFeTJWcTluQ21wQXJqOTlsU2Z6aTFTWWtE?=
 =?utf-8?B?WjY3UXdKUWdpQjFrUUl6VDhRWkJVOEFwMGFEeHJETmlZOE00Q3M0dnYyNy9r?=
 =?utf-8?B?aFQ2d3Z6am4veGVrYlludHc5QndtdXNpRk1yQnZUSmJHSmdWMHNoVEEyMnEy?=
 =?utf-8?B?TkwrN2tXSDdtZGIvdUl4MlgrYUNlY0JpTHdNQ0x1MXJrckd0SkRObXZRZnBx?=
 =?utf-8?B?bWh6ZXAwZVZMNlE0eGhEK21iUmRzdE9WYmgwRS9YdGNGekJNRG9EUG91elFY?=
 =?utf-8?B?ZmRtdmVxTkQ2Z1Z0T3JPd3kxVmd4NHNTOFowQS92K0JFWFBLRW1CQ01BRGRv?=
 =?utf-8?B?bS9IeGFYNjhhZnI2TXd4djhVSlRkWDdZMkwwa3JoZi9uMGZZTzhldzJsaEY2?=
 =?utf-8?B?UDdHZFBHYit0b0IzSU5ySW5odlRjT3JxczFsQlg4ZXBjM2p0YlJNYXVmZXNm?=
 =?utf-8?B?V3VUTzB3RGd1UGhzR3RLbzFUOFM3b2ljQ1FlQmxkdmZzcWU5bXo3RTZnRjds?=
 =?utf-8?B?dHZ2TThOdWtvT2srcVNJa1g4YXlzYlBPRCtEQld0RDJ4M0d6TExmaGdPQmF5?=
 =?utf-8?B?bjFXVkdodUhaVkp3WFlaWHk5UTYwZDUveXMzd002MFpEaDltZ2VuZDBGV0Fk?=
 =?utf-8?B?YUcrdmRPRFVCNmVBRjBUbE9KNmZkQVozdWV3b003ZUJQVTR4cTF6WksxQ0h5?=
 =?utf-8?B?UjY0dmE0emxaYWpuNUNzTHZ4KzJTUkowanFvUnBPZ28rM0hKM2grdmZZRzF2?=
 =?utf-8?B?SE5PVlFEbkRmZXcydmxwalZqOHcvaHpsUlp4SDBhd2Iyd0JYL3FyOTRsL3Zn?=
 =?utf-8?B?K0IzSzlHVXRtNHk0VWlLdGVtR1lub0xiY095SWd2SW1ldi94bmx6NVZpdjhV?=
 =?utf-8?B?WkdiMzBSRk5LaWUvYUhUVWNQa29lSS9TeFlubkhUMW1tek40ZWxkK00yUEtw?=
 =?utf-8?B?ZHFEeVh1OUgrY0U2bVdZQzg5cEEwNnZROVhlWTFsU3dNdlpuSldYN3JoNEhn?=
 =?utf-8?B?RmtTQWtETzBpelhtVm1jalhnQjBuRHQrZjlTbjBoNW9TeWJ1dm9vWWorbTFG?=
 =?utf-8?B?VE1IbTEySXpqYVVQUXIyOHFGZHBhSDVhcFhYaGEvai9lblpMR1hJWG4xVkgv?=
 =?utf-8?B?ZGJXZUh0TDYxeUttZTJ0Y0IrN1RXYzN1dllmVGhjLzBZeEpBVnovRDJHTEwx?=
 =?utf-8?B?T2NWc2JubXFzSkRtaXB2T1I3ekNleWZ1TlBnS1R1dXVJNHg2YzVFN01raDQw?=
 =?utf-8?B?MEVSN0c1UUJRam93eWk5cUJKb2wxMFZtUTZpSGpIL0ROWWtqUzA4UVllU0RY?=
 =?utf-8?B?NHpPZGlZUlVXdHNMSzBXOExJWk5RPT0=?=
X-OriginatorOrg: corigine.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9858bbb3-470c-4ce0-ae43-08db23fe8d9d
X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB4842.namprd13.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2023 20:07:21.1465
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: h02FrQ4AXHF7wYyYzz+CcXbjU6vNAdcO25cpA+/Ko9g46WUIP1dTfM7Qq3gWuWGjZqnn4zh6bk6XzXUQudXShq0GAVImNFBiEJDNnIwMAP0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR13MB5320
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 13, 2023 at 10:05:18AM +0800, Jason Xing wrote:
> On Sun, Mar 12, 2023 at 12:36 AM Jason Xing <kerneljasonxing@gmail.com> wrote:
> >
> > From: Jason Xing <kernelxing@tencent.com>
> >
> > When we encounter some performance issue and then get lost on how
> > to tune the budget limit and time limit in net_rx_action() function,
> > we can separately counting both of them to avoid the confusion.
> >
> > Signed-off-by: Jason Xing <kernelxing@tencent.com>
> > ---
> > note: this commit is based on the link as below:
> > https://lore.kernel.org/lkml/20230311151756.83302-1-kerneljasonxing@gmail.com/
> > ---
> >  include/linux/netdevice.h |  1 +
> >  net/core/dev.c            | 12 ++++++++----
> >  net/core/net-procfs.c     |  9 ++++++---
> >  3 files changed, 15 insertions(+), 7 deletions(-)
> >
> > diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> > index 6a14b7b11766..5736311a2133 100644
> > --- a/include/linux/netdevice.h
> > +++ b/include/linux/netdevice.h
> > @@ -3157,6 +3157,7 @@ struct softnet_data {
> >         /* stats */
> >         unsigned int            processed;
> >         unsigned int            time_squeeze;
> > +       unsigned int            budget_squeeze;
> >  #ifdef CONFIG_RPS
> >         struct softnet_data     *rps_ipi_list;
> >  #endif
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 253584777101..bed7a68fdb5d 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -6637,6 +6637,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
> >         unsigned long time_limit = jiffies +
> >                 usecs_to_jiffies(READ_ONCE(netdev_budget_usecs));
> >         int budget = READ_ONCE(netdev_budget);
> > +       bool is_continue = true;
> 
> I kept thinking during these days, I think it looks not that concise
> and elegant and also the name is not that good though the function can
> work.
> 
> In the next submission, I'm going to choose to use 'while()' instead
> of 'for()' suggested by Stephen.
> 
> Does anyone else have some advice about this?

What about:

	int done = false

	while (!done) {
		...
	}

Or:

	for (;;) {
		int done = false;

		...
		if (done)
			break;
	}

> 
> Thanks,
> Jason
> 
> >         LIST_HEAD(list);
> >         LIST_HEAD(repoll);
> >
> > @@ -6644,7 +6645,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
> >         list_splice_init(&sd->poll_list, &list);
> >         local_irq_enable();
> >
> > -       for (;;) {
> > +       for (; is_continue;) {
> >                 struct napi_struct *n;
> >
> >                 skb_defer_free_flush(sd);
> > @@ -6662,10 +6663,13 @@ static __latent_entropy void net_rx_action(struct softirq_action *h)
> >                  * Allow this to run for 2 jiffies since which will allow
> >                  * an average latency of 1.5/HZ.
> >                  */
> > -               if (unlikely(budget <= 0 ||
> > -                            time_after_eq(jiffies, time_limit))) {
> > +               if (unlikely(budget <= 0)) {
> > +                       sd->budget_squeeze++;
> > +                       is_continue = false;
> > +               }
> > +               if (unlikely(time_after_eq(jiffies, time_limit))) {
> >                         sd->time_squeeze++;
> > -                       break;
> > +                       is_continue = false;
> >                 }
> >         }
> >
> > diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c
> > index 97a304e1957a..4d1a499d7c43 100644
> > --- a/net/core/net-procfs.c
> > +++ b/net/core/net-procfs.c
> > @@ -174,14 +174,17 @@ static int softnet_seq_show(struct seq_file *seq, void *v)
> >          */
> >         seq_printf(seq,
> >                    "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x "
> > -                  "%08x %08x\n",
> > -                  sd->processed, sd->dropped, sd->time_squeeze, 0,
> > +                  "%08x %08x %08x %08x\n",
> > +                  sd->processed, sd->dropped,
> > +                  0, /* was old way to count time squeeze */
> > +                  0,
> >                    0, 0, 0, 0, /* was fastroute */
> >                    0,   /* was cpu_collision */
> >                    sd->received_rps, flow_limit_count,
> >                    0,   /* was len of two backlog queues */
> >                    (int)seq->index,
> > -                  softnet_input_pkt_queue_len(sd), softnet_process_queue_len(sd));
> > +                  softnet_input_pkt_queue_len(sd), softnet_process_queue_len(sd),
> > +                  sd->time_squeeze, sd->budget_squeeze);
> >         return 0;
> >  }
> >
> > --
> > 2.37.3
> >
>