Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Message-ID: <03b2fde9-945c-4b28-9177-5e195c866c32@intel.com>
Date:   Fri, 17 Nov 2023 15:51:07 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH net-next] net: microchip: lan743x : bidirectional
 throughput improvement
Content-Language: en-US
To:     Simon Horman <horms@kernel.org>,
        Vishvambar Panth S <vishvambarpanth.s@microchip.com>
CC:     <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <kuba@kernel.org>, <bryan.whitehead@microchip.com>,
        <UNGLinuxDriver@microchip.com>, <davem@davemloft.net>,
        <edumazet@google.com>, <pabeni@redhat.com>
References: <20231116054350.620420-1-vishvambarpanth.s@microchip.com>
 <20231117181602.GP164483@vergenet.net>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <20231117181602.GP164483@vergenet.net>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WnJaTmN6Nnk1eDVCKzJhQzlpZkduSzB6dk9GVmlYY3A1SU1kemY2VUhNekpB?=
 =?utf-8?B?Sm1qNnNGdUljQ1A3anBWM2ZmVFhUUURhKzNLaFZRLzFFNDh2cC9ZTHEyWmpS?=
 =?utf-8?B?NHM0a0Fvck8xNFRoc2ZZVEJvT29rTEFrcU9ZMzNibnRBMGh5T2d4enJNZ3ZI?=
 =?utf-8?B?VFZWeGFGTWdydVF4Y2FiaU5Pdm02U1BlV0s4Nnk3MW9YcmtrOFpyc1Z1RjF3?=
 =?utf-8?B?YWNEV3loeXdtUy81TC83Yjg2RzNrNGNjR3FrTFA0emR4SjRjazY5YW9NUkFP?=
 =?utf-8?B?akdOTUZGbDJBYW9IUi9uYldjZzc1NkFPZnRMb094aGpkelNXcTVKdXczdXo1?=
 =?utf-8?B?ZmNJMFpGcWlyOVNreldOc09TUktiaDI3dVdtU2VNak83cmRvbVRuSkp0QlI4?=
 =?utf-8?B?c242S0U1b1I1a1RZV1B2LzY2RWR6cm9XcDIrTExNQTdmUUFaeXFqSmJ0VzdW?=
 =?utf-8?B?ZkVZUFhoanNTWFhMOFNxaTJQZ2h6WnJQUVV5eU9iNGM1UThkdnBnc0t4eGpF?=
 =?utf-8?B?bDRPRTVxUnZ5Y0d0WHFzeXBiUmpiMlMrbFNNY1hVdTU1SFRDQXFpeHpHN2NG?=
 =?utf-8?B?a2JnTXBJS3JJVFpPWG1WNEs2RGtKS0k5LzBzU2lIZ1VCdFp1dUdpQVk3d3lT?=
 =?utf-8?B?blBZbUo2NWVpVFZGd24yYXFnUHdEMFU4SGZMT3ZiZTZSTVg0d1NNS3NRMk55?=
 =?utf-8?B?OUh5N1ErbTJpY1F1M2tyYjVUd1lVREloUlVWZTVCRUlUOWR4RXMvZ0FYQ1FX?=
 =?utf-8?B?cjhGNXQ1d2NhQncvalBzc3pzOERJcytVVExTUVpQUTRVY3lFWkJJYkMzd1N2?=
 =?utf-8?B?ZTBDQnhiakVwNENESFNJYVhtWGM4TnMyd05mNG1pOXNtamtHZytITjBKL01p?=
 =?utf-8?B?czdTV1N4U2F6ZWVMbCtDOThiOXBSQnNmNGtCa0RwclBOSFlJWlN2Mnh4UnRC?=
 =?utf-8?B?alhyQlJOaFQ1WmZTb3R2QmVUYmVjUDQ1Um9qZEM1ZWJaSEI0ZW94cHlRUDZa?=
 =?utf-8?B?SFRCT1hOcllDaWRpZTlFcHl0STNpTUZsT0xyVjdHRytWTEE3WjNNelQvSHhR?=
 =?utf-8?B?NFloZjNtNjQvQjh2YzBrQkRuQkp2cCtCZWFxdGVMMi9SMGtrZjc1K1hxOUtT?=
 =?utf-8?B?ZWhoM2EwSUtLR3Y5cEVPY213UEc5M1BRY1UyOEZVZVY2Q2pxOWhZNlUwYlhK?=
 =?utf-8?B?WU1RSWVTR1VMNUZTWmd1SHhscWlsMytQVlByUXBMcGRrVStqZU1kWi9kU3BP?=
 =?utf-8?B?eUtLVDJpMWhVRDF3T3JKUWVibjFUREIwcEdxUmRBUFU1T2cvbHFBaVBrdG9O?=
 =?utf-8?B?cUd2enZuSlJsZFRKVkJKSDhYYmI2TlRYaDhQSHhRdVN5ZWxlYkVIVjYxMXBF?=
 =?utf-8?B?VTIzbnN4VDUvRlRnUXJqU0pkNmtTTHBIUXFtMXRzOTVQUmlkY01BSEpWVWFW?=
 =?utf-8?B?dU5BRjVsZkFSZFNwK3RLVTBzdGxVa2J1ZWpycFl2WW5zN1hIMC9JdGNLVG9k?=
 =?utf-8?B?YUNlSm9hR2RLWkFlT3psaUQ1ZXNNb3hFWVJTQnI2U0Y3YWw0VlJjWldqb1pw?=
 =?utf-8?B?NU92MG5JdWhCL3c1SWZ4R1ViOE9qWGd6THRxUVF6WjVvNGhCOWFGUkVXcGhq?=
 =?utf-8?B?cmREcWFTZVFCS3I3SW5kQ1FMeTkzdVVCL0N4OWpaRkd3QWtOcm5IdytuR3Ur?=
 =?utf-8?B?akI0MmNCTHVSVXFUcEFwY3hNUnBLMjM0T0Z4MHdnTGZiVnU1RDJDUGFsMlc0?=
 =?utf-8?B?R3FPM05ySVkvZU0zeXpXTXo0Nk12dm8yNUEvbklMV1l1dzF3aXFvczRPaDdj?=
 =?utf-8?B?Q0ZtbVRuODFIaXlZNG56NEZodS9EUXZaQ0RKdW5KNmNqbkVZdVNGbGRmZVlP?=
 =?utf-8?B?YnJ4bEFDL09SaDc2eHBRVGdvMzJEZzhTN2RRYmhPYSs0S0NJMXNla0tkNmFM?=
 =?utf-8?B?THVWVUIrN21kdTNiUUpraWg4bFQ5OXB2NDd0YTBYYWtBWGc5Y2grM2R5T25k?=
 =?utf-8?B?TFE2RFhhRFp3VVg0SjNMeWhxaXVxeWZxcFpXQnJsMzRVbWFMWm5TUTR6cXJz?=
 =?utf-8?B?eU45RndrOEdka0RHUElzNS9aelR2ejIwdlhyTmtEdmcza2VFVlV5aUxZazVn?=
 =?utf-8?B?WVN6RmhtdGxSQ2g3UUNKSEIxazk4WGZDUDRjZXV5Rzl3SWJlNWJieWkrOWNl?=
 =?utf-8?B?U1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2f9ad91f-6dba-4773-6b2f-08dbe7c812f3
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2023 23:51:10.1513
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: c0fRJNzCRsyQzJ7K/n9Hlvlwm+fjmgItCQSdQ02QUzMSQuOnjrplLJ3TPmxFpErwzdG+gec+Unq0wzMJBOLRqxRIPfuLzDAAeL17DPCEXrw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB7906
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 17 Nov 2023 15:51:33 -0800 (PST)


On 11/17/2023 10:16 AM, Simon Horman wrote:
> On Thu, Nov 16, 2023 at 11:13:50AM +0530, Vishvambar Panth S wrote:
>> The LAN743x/PCI11xxx DMA descriptors are always 4 dwords long, but the
>> device supports placing the descriptors in memory back to back or
>> reserving space in between them using its DMA_DESCRIPTOR_SPACE (DSPACE)
>> configurable hardware setting. Currently DSPACE is unnecessarily set to
>> match the host's L1 cache line size, resulting in space reserved in
>> between descriptors in most platforms and causing a suboptimal behavior
>> (single PCIe Mem transaction per descriptor). By changing the setting
>> to DSPACE=16 many descriptors can be packed in a single PCIe Mem
>> transaction resulting in a massive performance improvement in
>> bidirectional tests without any negative effects.
>> Tested and verified improvements on x64 PC and several ARM platforms
>> (typical data below)
>>
>> Test setup 1: x64 PC with LAN7430 ---> x64 PC
>>
>> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval           Transfer     Bitrate
>> [  5][TX-C]   0.00-10.00  sec   170 MBytes   143 Mbits/sec  sender
>> [  5][TX-C]   0.00-10.04  sec   169 MBytes   141 Mbits/sec  receiver
>> [  7][RX-C]   0.00-10.00  sec  1.02 GBytes   876 Mbits/sec  sender
>> [  7][RX-C]   0.00-10.04  sec  1.02 GBytes   870 Mbits/sec  receiver
>>
>> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval           Transfer     Bitrate
>> [  5][TX-C]   0.00-10.00  sec  1.11 GBytes   956 Mbits/sec  sender
>> [  5][TX-C]   0.00-10.04  sec  1.11 GBytes   951 Mbits/sec  receiver
>> [  7][RX-C]   0.00-10.00  sec  1.10 GBytes   948 Mbits/sec  sender
>> [  7][RX-C]   0.00-10.04  sec  1.10 GBytes   942 Mbits/sec  receiver
>>
>> Test setup 2 : RK3399 with LAN7430 ---> x64 PC
>>
>> RK3399 Spec:
>> The SOM-RK3399 is ARM module designed and developed by FriendlyElec.
>> Cores: 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53
>> Frequency: Cortex-A72(up to 2.0GHz), Cortex-A53(up to 1.5GHz)
>> PCIe: PCIe x4, compatible with PCIe 2.1, Dual operation mode
>>
>> iperf3 UDP bidirectional with DSPACE set to L1 CACHE Size:
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval           Transfer     Bitrate
>> [  5][TX-C]   0.00-10.00  sec   534 MBytes   448 Mbits/sec  sender
>> [  5][TX-C]   0.00-10.05  sec   534 MBytes   446 Mbits/sec  receiver
>> [  7][RX-C]   0.00-10.00  sec  1.12 GBytes   961 Mbits/sec  sender
>> [  7][RX-C]   0.00-10.05  sec  1.11 GBytes   946 Mbits/sec  receiver
>>
>> iperf3 UDP bidirectional with DSPACE set to 16 Bytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID][Role] Interval           Transfer     Bitrate
>> [  5][TX-C]   0.00-10.00  sec   966 MBytes   810 Mbits/sec   sender
>> [  5][TX-C]   0.00-10.04  sec   965 MBytes   806 Mbits/sec   receiver
>> [  7][RX-C]   0.00-10.00  sec  1.11 GBytes   956 Mbits/sec   sender
>> [  7][RX-C]   0.00-10.04  sec  1.07 GBytes   919 Mbits/sec   receiver
>>
>> Signed-off-by: Vishvambar Panth S <vishvambarpanth.s@microchip.com>
> 
> Thanks,
> 
> I think you should have included Jacob's Reviewed-by tag from
> the previous posting of this patch [1].
> 
> And echoing his comments there, a very nice performance boost :)
> 
> Reviewed-by: Simon Horman <horms@kernel.org>
> 
> [1] https://lore.kernel.org/netdev/e5ffec56-5512-1acc-b85c-ac0771634c22@intel.com/

For the record, this version also looks good to me!

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>