Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Message-ID: <e5ffec56-5512-1acc-b85c-ac0771634c22@intel.com>
Date:   Fri, 29 Sep 2023 10:35:13 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [PATCH net] net: microchip: lan743x : bidirectional throughuput
 improvement
Content-Language: en-US
To:     Vishvambar Panth S <vishvambarpanth.s@microchip.com>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>
CC:     <bryan.whitehead@microchip.com>, <UNGLinuxDriver@microchip.com>,
        <davem@davemloft.net>, <edumazet@google.com>, <kuba@kernel.org>,
        <pabeni@redhat.com>, <andrew@lunn.ch>
References: <20230927111623.9966-1-vishvambarpanth.s@microchip.com>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <20230927111623.9966-1-vishvambarpanth.s@microchip.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Ym14OVVQejEyaTRiS3lkS3dBdEh4aU52bXQ1Y1pTeWFUTEZ2L1FQQlJpUVRW?=
 =?utf-8?B?RHk0SkhMaktPeDZ2QXJOeXpBcndybVlrQXVKNDE0MTR2WFIxNXNWbWk1RVBl?=
 =?utf-8?B?SWw4QllzTVlpNGNwR0R0bmJpTWF0eEo1emI5THk4UHNndG5VbzhvbWNoZisv?=
 =?utf-8?B?MFhudTBXM2FWbE1EZkprM1BHVVRPTmw4U0JaZWZhWEc2M050Q2NJOTFZRG9o?=
 =?utf-8?B?OGFhT1ZYSGpXT2x2UTE2RnU3azFjUHVIMkYxRHBvUkFKdGtYSkdFb1JwZkp1?=
 =?utf-8?B?NkUxZXFVeUNYU0RkN1J3Tlp3Y3FTUjV2ZFFEalhYVFF5SHoxZWN2Ynk3eGx1?=
 =?utf-8?B?RTJrYnp4MmxRbjBCYUJIQkw5bGhhUEZMTGFJK0RxUjk4RjZKekR0ekI0c0Uv?=
 =?utf-8?B?TWNVV2d2T0htSnBENmdlOXJoZ3Ixd0JMejQvTmJBOGhldUJmUkU2UzN2NWd3?=
 =?utf-8?B?YmNoMGxaUGlkSzlYaFJKemI1Rnc4Mmx5OEg5Q2dKWmcwaHREVEsyMXVLNytl?=
 =?utf-8?B?VmFjVnBzSGhNNHlEdGh5a1VYRm01VUFHUVF6VWQvSHh2b1o5emtRTVZNM2Iv?=
 =?utf-8?B?djRjZExmTCttMjJmS0ozTytRZjRvUVd4RVRpMVNCZUlKc1ZMQzhlNzBDVmhU?=
 =?utf-8?B?eXgxRUtGWlJ5ZDJDY01Xczg2WkNuV3k5ZW41emtMWEd6Y3NyZE1hd1EwK2hz?=
 =?utf-8?B?RzZRZW1TZmthalk4R3lqMmZ2bFNndmRlZ2dzZjBGSko0RU9jUmpqNDNqanha?=
 =?utf-8?B?QjJYdjdUME5sQU5ialhQZVRQYUJPSFFFSzU0NHRYYTZxekNIbmdjZmtnT1Nx?=
 =?utf-8?B?cDFWaUU3aCt6dyt6VXY5T2dtbmdXSENYRHFZWlVrTXBvMUU2d1Mwb0RON3k1?=
 =?utf-8?B?Umk5WVo5VXRmdEQ4N3pEYis3SkEyYzJEcW9xZ0ZKZllTcVdPZWdIYjhKbmZl?=
 =?utf-8?B?SDVNUmRXOGtwbnZtVGFpbC9DL0x6TmF3Z0hIdFhpbmZ5NzNHaFVvNG1Kb0FT?=
 =?utf-8?B?ZUtIREtoNk9Tc0FuR0F4dVlMT3NmMWVGbXBkRHJLOUUvVW1jY1hEVnZ0QWtJ?=
 =?utf-8?B?NW9FcTM3YWZZeEVNRzhNekxPcU0wK1M2QitkWUhDYzJzSksxeWpBWjc5dytU?=
 =?utf-8?B?YThNbWovWWxLRjhhZENsNitHWGlabG0yYit1WmFNeXRpdmhxeUY0MU16NXVO?=
 =?utf-8?B?VEowZlh3aVdCRGtuSi9pQ0dMWUQwRVVXN0J1YndLYW9oNGpOWDJYS2d1Q0ZE?=
 =?utf-8?B?aHNCdFlPTHlpTDZqNHh3a1RBZURJbFFGczlQVERwQWNzZnlVV0tyUlQ1Rkgr?=
 =?utf-8?B?enFkMW8yQUdPTXFHRlZacURwRE5QaWZhLzlXNUJ0UW9LY1VFUURjMnRNUDNE?=
 =?utf-8?B?THZ5eFp3b1VEaEp6YzZPblJkRzBrdmd1b0lKNEFpQ0NXSUNpRW9OSGNFQmlM?=
 =?utf-8?B?NElBTmlYbWpmVmN3aU16ZmkySmxoU3FvNjB6alQ3SFBKWjVRTUlGa2kvVWMy?=
 =?utf-8?B?NUt5ZXZsSXBnbEdFZXRhS1FKS1c5N0UvdG9pNkxsWnR1cjZJRnFiRU9QTVRM?=
 =?utf-8?B?eHVHU2lmYXFLYXZCVUh4di9PM2orK2o4YWFISzJ1SEZSbFJiQ0RkTGVjOXVG?=
 =?utf-8?B?dU50UVBVbGQvZ2kvMktDakRaVGpwMjZkV0N4UmRaa1VaNnFZRTZyQ2swWnNX?=
 =?utf-8?B?TG5wSy9NMlVIV0hBb0JWZlhUekxCYnBYUytoeFBJVFlPeGpKcWFnN1VFWXE2?=
 =?utf-8?B?aG1CRGN6cHAvOUphMm5RVGU0c1huVHllOGJSZW1SZlgycGl6N3JLcitJbmd0?=
 =?utf-8?B?MGZhVFZOMG9hdFYrSC83dkdlclFDWndMUGZqUURMZXlFMW9YaVUxZW13VUQy?=
 =?utf-8?B?a1lEa2kxeTNOU1Blems1TU1aTmZjMUdMR2pCTVlvaXY3NGlCOWs5bXZvWWQy?=
 =?utf-8?B?eloxV29rMUZTb29YU1Yvb2tzOTR5TWowTnhDQ25kQkgxZnV1VVUwRUx3Tm82?=
 =?utf-8?B?bXM2eE84OVh6dFYzRUtuSlFrSzI0bmJmQkZ2eG9JMHpmNmJKeHkvR1hCbXpL?=
 =?utf-8?B?UkZRZnhoa1dKYzhjRFhHaWFXQnV3Uzdwc0NENEVoUTlLNWRkZjJDbmJERE5C?=
 =?utf-8?B?WC8raFBxWGhJdkZvR2hCdlZrdlVoK25MQjZRaXpMZXpiV2ZoZHlhR1M1UGRl?=
 =?utf-8?B?aUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d5b26614-630c-4328-8607-08dbc1127253
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2023 17:35:17.5810
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wmY1Gd+7y9NV40YVL8AWVURYG2hHgS1U3tK6zDRSnNKcuOeV/Cx+SL1y37HOINFfxZbikT35Q4BgQGt7QF1NOcCu3YZ03LHW0SGF5o3km1A=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR11MB5694
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.1 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 29 Sep 2023 11:34:19 -0700 (PDT)


On 9/27/2023 4:16 AM, Vishvambar Panth S wrote:
> The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
> device supports placing the descriptors in memory back to back or
> reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
> configurable hardware setting. Currently DSPACE is unnecessarily set to
> match the host's L1 cache line size, resulting in space reserved in
> between descriptors in most platforms and causing a suboptimal behavior
> (single PCIe Mem transaction per descriptor). By changing the setting
> to DSPACE=16 many descriptors can be packed in a single PCIe Mem
> transaction resulting in a massive performance improvement in
> bidirectional tests without any negative effects.
> Tested and verified improvements on x64 PC and several ARM platforms
> (typical data below)
> 

I assume here the choice of 16 is to get the 16 bytes from 4 dwords?

> Test setup 1: x64 PC with LAN7430 ---> x64 PC
> 
> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval           Transfer     Bitrate
> [  5][TX-C]   0.00-10.00  sec   170 MBytes   143 Mbits/sec  sender
> [  5][TX-C]   0.00-10.04  sec   169 MBytes   141 Mbits/sec  receiver
> [  7][RX-C]   0.00-10.00  sec  1.02 GBytes   876 Mbits/sec  sender
> [  7][RX-C]   0.00-10.04  sec  1.02 GBytes   870 Mbits/sec  receiver
> 
> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval           Transfer     Bitrate
> [  5][TX-C]   0.00-10.00  sec  1.11 GBytes   956 Mbits/sec  sender
> [  5][TX-C]   0.00-10.04  sec  1.11 GBytes   951 Mbits/sec  receiver
> [  7][RX-C]   0.00-10.00  sec  1.10 GBytes   948 Mbits/sec  sender
> [  7][RX-C]   0.00-10.04  sec  1.10 GBytes   942 Mbits/sec  receiver
> 

Going from barely transmitting to hitting the line rate *and* improving
both Tx and Rx slightly is fantastic. Huge win just avoiding all that
unnecessary wasted PCIe bandwidth.

> Test setup 2 : RK3399 with LAN7430 ---> x64 PC
> 
> RK3399 Spec:
> The SOM-RK3399 is ARM module designed and developed by FriendlyElec.
> Cores: 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53
> Frequency: Cortex-A72(up to 2.0GHz), Cortex-A53(up to 1.5GHz)
> PCIe: PCIe x4, compatible with PCIe 2.1, Dual operation mode
> 
> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval           Transfer     Bitrate
> [  5][TX-C]   0.00-10.00  sec   534 MBytes   448 Mbits/sec  sender
> [  5][TX-C]   0.00-10.05  sec   534 MBytes   446 Mbits/sec  receiver
> [  7][RX-C]   0.00-10.00  sec  1.12 GBytes   961 Mbits/sec  sender
> [  7][RX-C]   0.00-10.05  sec  1.11 GBytes   946 Mbits/sec  receiver
> 
> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID][Role] Interval           Transfer     Bitrate
> [  5][TX-C]   0.00-10.00  sec   966 MBytes   810 Mbits/sec   sender
> [  5][TX-C]   0.00-10.04  sec   965 MBytes   806 Mbits/sec   receiver
> [  7][RX-C]   0.00-10.00  sec  1.11 GBytes   956 Mbits/sec   sender
> [  7][RX-C]   0.00-10.04  sec  1.07 GBytes   919 Mbits/sec   receiver
> 
> Fixes: 23f0703c125b ("lan743x: Add main source files for new lan743x driver")
> Signed-off-by: Vishvambar Panth S <vishvambarpanth.s@microchip.com>
> ---

Always fun to see a massive win like this! Even just the 2x improvement
on the RK3399 is huge, let alone 6x improvement on the x86 platform.

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>

>  drivers/net/ethernet/microchip/lan743x_main.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/microchip/lan743x_main.h b/drivers/net/ethernet/microchip/lan743x_main.h
> index 52609fc13ad9..6dac6fef7d24 100644
> --- a/drivers/net/ethernet/microchip/lan743x_main.h
> +++ b/drivers/net/ethernet/microchip/lan743x_main.h
> @@ -1067,7 +1067,7 @@ struct lan743x_adapter {
>  #define DMA_DESCRIPTOR_SPACING_32       (32)
>  #define DMA_DESCRIPTOR_SPACING_64       (64)
>  #define DMA_DESCRIPTOR_SPACING_128      (128)
> -#define DEFAULT_DMA_DESCRIPTOR_SPACING  (L1_CACHE_BYTES)
> +#define DEFAULT_DMA_DESCRIPTOR_SPACING  (DMA_DESCRIPTOR_SPACING_16)
>  
>  #define DMAC_CHANNEL_STATE_SET(start_bit, stop_bit) \
>  	(((start_bit) ? 2 : 0) | ((stop_bit) ? 1 : 0))