Received-SPF: pass (google.com: domain of linux-kernel+bounces-37993-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Wed, 24 Jan 2024 22:07:33 -0800
From: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com>
To: Yury Norov <yury.norov@gmail.com>
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
	decui@microsoft.com, davem@davemloft.net, edumazet@google.com,
	kuba@kernel.org, pabeni@redhat.com, longli@microsoft.com,
	leon@kernel.org, cai.huoqing@linux.dev, ssengar@linux.microsoft.com,
	vkuznets@redhat.com, tglx@linutronix.de,
	linux-hyperv@vger.kernel.org, netdev@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
	schakrabarti@microsoft.com, paulros@microsoft.com
Subject: Re: [PATCH 4/4 V2 net-next] net: mana: Assigning IRQ affinity on HT
 cores
Message-ID: <20240125060733.GB18142@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
References: <1705939259-2859-1-git-send-email-schakrabarti@linux.microsoft.com>
 <1705939259-2859-5-git-send-email-schakrabarti@linux.microsoft.com>
 <ZbEqu68J3f9W8nIM@yury-ThinkPad>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZbEqu68J3f9W8nIM@yury-ThinkPad>
User-Agent: Mutt/1.5.21 (2010-09-15)

On Wed, Jan 24, 2024 at 07:20:27AM -0800, Yury Norov wrote:
> On Mon, Jan 22, 2024 at 08:00:59AM -0800, Souradeep Chakrabarti wrote:
> > Existing MANA design assigns IRQ to every CPU, including sibling
> > hyper-threads. This may cause multiple IRQs to be active simultaneously
> > in the same core and may reduce the network performance.
> > 
> > Improve the performance by assigning IRQ to non sibling CPUs in local
> > NUMA node. The performance improvement we are getting using ntttcp with
> > following patch is around 15 percent against existing design and
> > approximately 11 percent, when trying to assign one IRQ in each core
> > across NUMA nodes, if enough cores are present.
> > The change will improve the performance for the system
> > with high number of CPU, where number of CPUs in a node is more than
> > 64 CPUs. Nodes with 64 CPUs or less than 64 CPUs will not be affected
> > by this change.
> > 
> > The performance study was done using ntttcp tool in Azure.
> > The node had 2 nodes with 32 cores each, total 128 vCPU and number of channels
> > were 32 for 32 RX rings.
> > 
> > The below table shows a comparison between existing design and new
> > design:
> > 
> > IRQ   node-num    core-num   CPU        performance(%)
> > 1      0 | 0       0 | 0     0 | 0-1     0
> > 2      0 | 0       0 | 1     1 | 2-3     3
> > 3      0 | 0       1 | 2     2 | 4-5     10
> > 4      0 | 0       1 | 3     3 | 6-7     15
> > 5      0 | 0       2 | 4     4 | 8-9     15
> > ---
> > ---
> > 25     0 | 0       12| 24    24| 48-49   12
> > ---
> > 32     0 | 0       15| 31    31| 62-63   12
> > 33     0 | 0       16| 0     32| 0-1     10
> > ---
> > 64     0 | 0       31| 31    63| 62-63   0
> 
> Did that omitted lines mean 5-24 : 15%, 25-31 : 12% and 33-63 : 10%?
They are same like you have mentioned, so I have skipped those ranges.
Whereever the major changes were there, I have put them.
So it is the full coverage only skimmed a little.
> Or that means that you didn't test those?
> 
> Would be nice to have full coverage...
> 
> Thanks,
> Yury