Message-ID: <113e81f6-b349-97c0-4cec-d90087e7e13b@nvidia.com>
Date:   Wed, 8 Feb 2023 13:08:54 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Content-Language: en-US
From:   Tariq Toukan <tariqt@nvidia.com>
Subject: Bug report: UDP ~20% degradation
To:     Vincent Guittot <vincent.guittot@linaro.org>,
        David Chen <david.chen@nutanix.com>,
        Zhang Qiao <zhangqiao22@huawei.com>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        Willem de Bruijn <willemdebruijn.kernel@gmail.com>
Cc:     Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Valentin Schneider <vschneid@redhat.com>,
        linux-kernel@vger.kernel.org,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Tariq Toukan <tariqt@nvidia.com>,
        Network Development <netdev@vger.kernel.org>,
        Gal Pressman <gal@nvidia.com>, Malek Imam <mimam@nvidia.com>,
        Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
        David Ahern <dsahern@kernel.org>,
        Tariq Toukan <ttoukan.linux@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NmNuZ2cyMGE3NTdOVHBrT1VpdmE2dC9EY01iYkZRMkNUNmdnK0s2Wnd6UGhX?=
 =?utf-8?B?emhnWmRRcXFDYXlMYWhCQ3VjWkVCMjZrMEVNeGl6bWVsQkR5eHZ0THBsRm9a?=
 =?utf-8?B?Y2RmUStVVGQ1R1VTRnVJWVpGbytqeWZUNDYzbi9MS0RXbnlHQTRwV2pkZnN0?=
 =?utf-8?B?cTZLSXVwRWduTkF5S1VvUzFqeTNSRkpaUUdWcjF6bHdNL0NMZUJreGhsb2Vw?=
 =?utf-8?B?NEVCckpaalUvby82Z2I5eDRRM3RyMkJaR2pZbUg1MFFvZjZuTHVsaEI3Titz?=
 =?utf-8?B?THlzd1V5a0F5b2JxWnRlSDRJMW9YckxaREJGcXp0TWxLd3ptSVY2d3V3bEU1?=
 =?utf-8?B?U0lYc0lUUGpSYVN0TG0yY0l2dFdFNk5DUE5iL1NacnVZWXVmbjVMREZaY3U3?=
 =?utf-8?B?YXFzOTdkRU9OTkZZRFJKbFlYcVBkMXdkeGx3R3g1cDRzaTZFT0MzaGsxcTV6?=
 =?utf-8?B?NUdxU3BaakE2TERzNVFDTzQvbjA1MXpkN0pxeVlaN0pEb3JRVGRNKy9lNTBp?=
 =?utf-8?B?bEFKUUc0U0dtcTlueFRFbU5pUDlVYnh2L0xtanJiNWozNXBNSmpuRnVnMUNz?=
 =?utf-8?B?TEhPdVdTWFB1NDlsNjVqaDl3QmMweHprclNwNFI3bWZ3bVYxc1grVWZOV0h2?=
 =?utf-8?B?UWgvbHozc3dTWUUvUjk1VFU3T21rcmlFanlROTUzc1ZPL0xUSjd5SnEvZkJE?=
 =?utf-8?B?amZoQ3VQbU14Mk9TdjdQeFo1VmhONHBOMkVoLzRzSlZSeG90SE5DUENuT0Zj?=
 =?utf-8?B?QzlWem11V2xoU1pKb1p0a2FPVFpvU0ZmQ2hQMDh3MHVnT0R3RHNpbGliU0h0?=
 =?utf-8?B?cEpZOVpzNm9NSnUvZXc4Z3pOaHloT3RNaWNUcVYrM2E4L2J3a01BOVc2cE9z?=
 =?utf-8?B?MC9JcW1wS3V0WGQ0MHZWcFRZZVQ4NVhvWEUzemlrUHBPUUU4c1VNQ0RHRUQr?=
 =?utf-8?B?S1ZLanEvOEpmdzQ5aWR1R1h1U0dzeUlDU2J1UWtsd0hOWEgwQnRpNDY0Zkt2?=
 =?utf-8?B?bDVYdFhOWnpadWlnNHY0Y2RoVHNxRlhDNWhEaVJMWHAxYmluR2xnbWRlQ2Jl?=
 =?utf-8?B?aXd1Qk4wWEVNc3VXUkZ5V3lvOEU0dTRhTm5HTzZ1OXQyZnF3TGJsamdzQWRE?=
 =?utf-8?B?TUh4dmtBNGRwY3duMDV2eHBjSUVxeVlKakc2ay9wTzlKREIzQ2hWVTVURTha?=
 =?utf-8?B?SGltTUFHTmkzdGVaa01kSm4wOXRsN1I0UVgxc0tnRHNuTk9SV1Q3MlhTY3lm?=
 =?utf-8?B?MDBmL1FjS2ZodUpzS3Z3ZFZjMFgyK0JmTksyWDY3ek1OQTZtZm9vMXJOV3Mr?=
 =?utf-8?B?am11aFQxODFxeXRIRzlKRXdmMHIvZkJYVVkySEwwSStmN29IaS95UmNaRW9M?=
 =?utf-8?B?c0hUeHM0a3o3VUZ2WkJ0TlJDWkpzZUI1aGhJc0pFbjNmSWc2TlAvdUpaSmVQ?=
 =?utf-8?B?VUxleEpwRXA1dFVHZTdNM1ZscDV2Y05na1NsYk4yaDFOMWVnMmxNcmJjMklp?=
 =?utf-8?B?VlJxYUxSOWZzVE5SdjRZRk56eHlmYUxSL25OVnZ3c2Q3S3J3K0lOd1dxZ29j?=
 =?utf-8?B?OUd4SmE0WG91MFVuSGlXanl1dWpZTTNlU1dYL243eEppOHJxTGx0K082aWIv?=
 =?utf-8?B?NjlkWEZGT0tVdFAvblhyN3RqeFRrVTQrakU4blViRnFqZE9VV1pveGV2ekpN?=
 =?utf-8?B?ZjhzOU1qNVpWaXg4Zm0xNGpLWjZWYmp1SGJtdVptakJueVZQYnZIa1djeUZi?=
 =?utf-8?B?NmNyOVdjS3krVWVwNHlxNG5YSC9XNXVhSUtEeTRrZC9JQzNDSVNhTnI2UDIw?=
 =?utf-8?B?RWhxa3JHeHpuY0xRRTI4QjUzNlBwc2hhZDkzZlNkTlFpeS9SQUNvU2o0VjdE?=
 =?utf-8?B?RGZkaXFmc3ViendTMGZSeEdVaFNvNXlPWUhTMllMdEYxZDVwYkVhanZjeEEv?=
 =?utf-8?B?UGY2MERyVzI1NExHREZ1aTMvVzZpRmVPU1ROWUdKQXo4ekNzVzJqVExHb00x?=
 =?utf-8?B?N3ErVzBIVkZtNVE0MzU0b1FadG16dkQ0cVU0YWMyN1hpaTc1UHRYN01aNzRr?=
 =?utf-8?B?RGE1STdwNm4wR3hkcW8vWVkrYXFNSmp6VGZhYXMyK0pST1Qyc2g0ajVVWFBO?=
 =?utf-8?Q?K4UtkY6JPgPt6SY5P4WBTiaHr?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 11c87bd7-473e-47d0-fe85-08db09c4e40f
X-MS-Exchange-CrossTenant-AuthSource: DM5PR12MB2485.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2023 11:09:05.0831
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LtbVN6yK5cVk9UgCDaBYueyEQfxrJW0ShMlvG9lDDSRF/UqkjtSG52rAWjvH1JzlFrHH5AdBOQJmwVAClKPGdQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5732
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi all,

Our performance verification team spotted a degradation of up to ~20% in 
UDP performance, for a specific combination of parameters.

Our matrix covers several parameters values, like:
IP version: 4/6
MTU: 1500/9000
Msg size: 64/1452/8952 (only when applicable while avoiding ip 
fragmentation).
Num of streams: 1/8/16/24.
Num of directions: unidir/bidir.

Surprisingly, the issue exists only with this specific combination:
8 streams,
MTU 9000,
Msg size 8952,
both ipv4/6,
bidir.
(in unidir it repros only with ipv4)

The reproduction is consistent on all the different setups we tested with.

Bisect [2] was done between these two points, v5.19 (Good), and v6.0-rc1 
(Bad), with ConnectX-6DX NIC.

c82a69629c53eda5233f13fc11c3c01585ef48a2 is the first bad commit [1].

We couldn't come up with a good explanation how this patch causes this 
issue. We also looked for related changes in the networking/UDP stack, 
but nothing looked suspicious.

Maybe someone here can help with this.
We can provide more details or do further tests/experiments to progress 
with the debug.

Thanks,
Tariq

[1]
commit c82a69629c53eda5233f13fc11c3c01585ef48a2
Author: Vincent Guittot <vincent.guittot@linaro.org>
Date:   Fri Jul 8 17:44:01 2022 +0200

     sched/fair: fix case with reduced capacity CPU

     The capacity of the CPU available for CFS tasks can be reduced 
because of
     other activities running on the latter. In such case, it's worth 
trying to
     move CFS tasks on a CPU with more available capacity.
 
 
     The rework of the load balance has filtered the case when the CPU 
is 
 

     classified to be fully busy but its capacity is reduced. 
 
 
     Check if CPU's capacity is reduced while gathering load balance 
statistic 
 

     and classify it group_misfit_task instead of group_fully_busy so we 
can 
 

     try to move the load on another CPU. 
 
 
     Reported-by: David Chen <david.chen@nutanix.com> 
 
 
     Reported-by: Zhang Qiao <zhangqiao22@huawei.com> 
 
 
     Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org> 
 
 
     Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> 
 
 
     Tested-by: David Chen <david.chen@nutanix.com> 
 
 
     Tested-by: Zhang Qiao <zhangqiao22@huawei.com> 
 
 
     Link: 
https://lkml.kernel.org/r/20220708154401.21411-1-vincent.guittot@linaro.org 
 
 
[2]

Detailed bisec steps:

+--------------+--------+-----------+-----------+
| Commit       | Status | BW (Gbps) | BW (Gbps) |
|              |        | run1      | run2      |
+--------------+--------+-----------+-----------+
| 526942b8134c | Bad    | ---       | ---       |
+--------------+--------+-----------+-----------+
| 2e7a95156d64 | Bad    | ---       | ---       |
+--------------+--------+-----------+-----------+
| 26c350fe7ae0 | Good   | 279.8     | 281.9     |
+--------------+--------+-----------+-----------+
| 9de1f9c8ca51 | Bad    | 257.243   | ---       |
+--------------+--------+-----------+-----------+
| 892f7237b3ff | Good   | 285       | 300.7     |
+--------------+--------+-----------+-----------+
| 0dd1cabe8a4a | Good   | 305.599   | 290.3     |
+--------------+--------+-----------+-----------+
| dfea84827f7e | Bad    | 250.2     | 258.899   |
+--------------+--------+-----------+-----------+
| 22a39c3d8693 | Bad    | 236.8     | 245.399   |
+--------------+--------+-----------+-----------+
| e2f3e35f1f5a | Good   | 277.599   | 287       |
+--------------+--------+-----------+-----------+
| 401e4963bf45 | Bad    | 250.149   | 248.899   |
+--------------+--------+-----------+-----------+
| 3e8c6c9aac42 | Good   | 299.09    | 294.9     |
+--------------+--------+-----------+-----------+
| 1fcf54deb767 | Good   | 292.719   | 301.299   |
+--------------+--------+-----------+-----------+
| c82a69629c53 | Bad    | 254.7     | 246.1     |
+--------------+--------+-----------+-----------+
| c02d5546ea34 | Good   | 276.4     | 294       |
+--------------+--------+-----------+-----------+