Received: by 2002:ab2:68c1:0:b0:1fd:9a81:d0e4 with SMTP id e1csp403330lqp; Sun, 9 Jun 2024 01:11:20 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCW0m1LKDxwsBQgZvDvp08LP821E1A++vc8tB4UXLqPX3Q0K2CerLGuUN52xPidD2XCGHdOAC4ciI2AFRP04aJ10DYRhaFarLnoxLE1psA== X-Google-Smtp-Source: AGHT+IGpyI+RO8zuZphF3VHrghGw7LhCjFfnItAu63Q1V3PnnsDVyA1JEbJRmA7ELslpdeDULiTC X-Received: by 2002:a05:6808:2388:b0:3d2:21a7:8620 with SMTP id 5614622812f47-3d221a78a37mr2245629b6e.12.1717920680347; Sun, 09 Jun 2024 01:11:20 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1717920680; cv=pass; d=google.com; s=arc-20160816; b=MXnPfAHn0F9/DZYmAvFkUaYPB4We50xDhOGJIz5A+ao54fl61UNA3/gD7rdAkqhNyi CyxXiwEEfeOurLAIFdyNwKfMT21nWTXGtMlV51FQZHIqRccF5btn7aorz52JkWAgATFN EosKYjwE+yV78Pe3Qt+U87dvUUd2Y116oKIdUMtNGs1Mfu1UjWBnQVhsUl0AG8Vs1aTJ Jyv0Rq2mG9fgSs6hikrCfo1hZD9AhB9pwRFc98P+V5BxKJyl9+9FO+sp5XTOw5Mj82vP Qm/PXJSAaLeLfSGr1sOvEL1ava7jMbz7dDpJIM7nYN5y8fRyexYrfHLxwZYXxQLGVKcu jKbg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:reply-to:autocrypt :content-language:from:references:cc:to:subject:user-agent :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :date:message-id:dkim-signature; bh=un0v7NRiuJ73SMR3ZhPeyJlOxaz9WHZsIQkF12rLpnY=; fh=0DTN4ODuD2UGFXDVyJQkJBbu3V3aWlrTBkJ01aKKnrA=; b=ddTJFw0VQonoavZtTYShowINoUhB5yJLwUVduUXuMnxzA5fjpKSVfAZDmvhe7YPdPl FhjmwJVSDWMug5uNAPQ+3BZQk99waIvjYaiQLN6KbsKV3vP/0RW9cBkmXBKf8ReVfroj HuaMTtRDAqK1PsbK/AgV+msUw1o4SmeySQds7nI7HDmGPbNKCkHtrJKXmpLSZdUCTQnT nrGXVv53BXpeBU6S8eTe55oXUhDFjy8f23/I1eSKuQwAn+bCd07u3i7DmFcO4fECxhUB WhgVr5kMTato/zic1KzVeFUQqM2qHe1/i64sN56BEBBdXrwRNBhiJ+wslSZ/r6mqRw8Z vemA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@leemhuis.info header.s=he214686 header.b=pnvg5s3O; arc=pass (i=1 spf=pass spfdomain=leemhuis.info dkim=pass dkdomain=leemhuis.info); spf=pass (google.com: domain of linux-kernel+bounces-207276-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-207276-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id d75a77b69052e-4405f476c35si20626021cf.92.2024.06.09.01.11.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Jun 2024 01:11:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-207276-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@leemhuis.info header.s=he214686 header.b=pnvg5s3O; arc=pass (i=1 spf=pass spfdomain=leemhuis.info dkim=pass dkdomain=leemhuis.info); spf=pass (google.com: domain of linux-kernel+bounces-207276-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-207276-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 067DC1C21325 for ; Sun, 9 Jun 2024 08:11:20 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2896D1C6B4; Sun, 9 Jun 2024 08:11:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="pnvg5s3O" Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD2A5A3F; Sun, 9 Jun 2024 08:11:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717920674; cv=none; b=MCBzhfTPDd1sjjnVA6YZdwuC6wCRxXb060RYmZrZZhYRdxO7Tuk8R1oDXTnJFomacd8uW/u7kIkDw+LQ202sC7l1e/Cokjz8JXP+kz1BcrFCpZNU60uRDQWvqbCrp30M6xbWywt6NkNfWlN0qkvhy3LiirMURYC2X+AZTx2475A= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717920674; c=relaxed/simple; bh=5zbXRU1k+pJeZ0BVwq8DCQkKWnsqWbzOX8ocz4wTois=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=eNK5fL0g4dfbhhcD2ljRUBhfKArU+Hxe2pXSZ8X8gASK2KnY09cXbeB7TtYKTFGidqZkQPelh6Pc6Pfv5jMzcoHckGdZ+3MAuP6aWAjLL3tpG/g6VQPTTOLBpKQy9Xs2bnex99H2wcFMfhotVqqgKyfi0xXGsG0viDYmUGZJZT8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=pnvg5s3O; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:From:Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version: Content-Type:Content-Transfer-Encoding:Content-ID:Content-Description: In-Reply-To:References; bh=un0v7NRiuJ73SMR3ZhPeyJlOxaz9WHZsIQkF12rLpnY=; t=1717920671; x=1718352671; b=pnvg5s3OGzyyNMZ2NQi/7qCIqHJGeDv8lLawx3DipUn29zW fwU7gShJGJVFAbjyt+tH9GVJgTQmEY6ovpTbCagbp/LKYe4iyhcI00pBUgnZmEvQk/hrjdYmhm70D VZQrU3c9brOz70SbTTPwZU0afU+fIGeViV7Mz/kzHKwH09A7tuhBWQaewqD4g9SssmbIWAnhuOOgh zkykhGvBxbFa9kXaV6n0+fkm7z2vYgRHfwGoZ5L52z/SADjzNDvMSGNIGeAcTIFqiG+C+JeWiN9Pb O5Cq/m0qU8pYEsBo7aRbSf5286zsrWhn13RJ3EOhrtOhVxOrOANPoXFPRkZOv/YQ==; Received: from [2a02:8108:8980:2478:8cde:aa2c:f324:937e]; authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) id 1sGDdm-0000XV-OS; Sun, 09 Jun 2024 10:11:02 +0200 Message-ID: Date: Sun, 9 Jun 2024 10:11:02 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Linux 6.10-rc2 - massive performance regression To: David Laight Cc: Linux Kernel Mailing List , Linus Torvalds , Linux kernel regressions list References: <46cb50d65e414bfd9bef5549d68ae4ea@AcuMS.aculab.com> From: "Linux regression tracking (Thorsten Leemhuis)" Content-Language: en-US, de-DE Autocrypt: addr=linux@leemhuis.info; keydata= xsFNBFJ4AQ0BEADCz16x4kl/YGBegAsYXJMjFRi3QOr2YMmcNuu1fdsi3XnM+xMRaukWby47 JcsZYLDKRHTQ/Lalw9L1HI3NRwK+9ayjg31wFdekgsuPbu4x5RGDIfyNpd378Upa8SUmvHik apCnzsxPTEE4Z2KUxBIwTvg+snEjgZ03EIQEi5cKmnlaUynNqv3xaGstx5jMCEnR2X54rH8j QPvo2l5/79Po58f6DhxV2RrOrOjQIQcPZ6kUqwLi6EQOi92NS9Uy6jbZcrMqPIRqJZ/tTKIR OLWsEjNrc3PMcve+NmORiEgLFclN8kHbPl1tLo4M5jN9xmsa0OZv3M0katqW8kC1hzR7mhz+ Rv4MgnbkPDDO086HjQBlS6Zzo49fQB2JErs5nZ0mwkqlETu6emhxneAMcc67+ZtTeUj54K2y Iu8kk6ghaUAfgMqkdIzeSfhO8eURMhvwzSpsqhUs7pIj4u0TPN8OFAvxE/3adoUwMaB+/plk sNe9RsHHPV+7LGADZ6OzOWWftk34QLTVTcz02bGyxLNIkhY+vIJpZWX9UrfGdHSiyYThHCIy /dLz95b9EG+1tbCIyNynr9TjIOmtLOk7ssB3kL3XQGgmdQ+rJ3zckJUQapLKP2YfBi+8P1iP rKkYtbWk0u/FmCbxcBA31KqXQZoR4cd1PJ1PDCe7/DxeoYMVuwARAQABzSdUaG9yc3RlbiBM ZWVtaHVpcyA8bGludXhAbGVlbWh1aXMuaW5mbz7CwZQEEwEKAD4CGwMFCwkIBwMFFQoJCAsF FgIDAQACHgECF4AWIQSoq8a+lZZX4oPULXVytubvTFg9LQUCX31PIwUJFmtPkwAKCRBytubv TFg9LWsyD/4t3g4i2YVp8RoKAcOut0AZ7/uLSqlm8Jcbb+LeeuzjY9T3mQ4ZX8cybc1jRlsL JMYL8GD3a53/+bXCDdk2HhQKUwBJ9PUDbfWa2E/pnqeJeX6naLn1LtMJ78G9gPeG81dX5Yq+ g/2bLXyWefpejlaefaM0GviCt00kG4R/mJJpHPKIPxPbOPY2REzWPoHXJpi7vTOA2R8HrFg/ QJbnA25W55DzoxlRb/nGZYG4iQ+2Eplkweq3s3tN88MxzNpsxZp475RmzgcmQpUtKND7Pw+8 zTDPmEzkHcUChMEmrhgWc2OCuAu3/ezsw7RnWV0k9Pl5AGROaDqvARUtopQ3yEDAdV6eil2z TvbrokZQca2808v2rYO3TtvtRMtmW/M/yyR233G/JSNos4lODkCwd16GKjERYj+sJsW4/hoZ RQiJQBxjnYr+p26JEvghLE1BMnTK24i88Oo8v+AngR6JBxwH7wFuEIIuLCB9Aagb+TKsf+0c HbQaHZj+wSY5FwgKi6psJxvMxpRpLqPsgl+awFPHARktdPtMzSa+kWMhXC4rJahBC5eEjNmP i23DaFWm8BE9LNjdG8Yl5hl7Zx0mwtnQas7+z6XymGuhNXCOevXVEqm1E42fptYMNiANmrpA OKRF+BHOreakveezlpOz8OtUhsew9b/BsAHXBCEEOuuUg87BTQRSeAENARAAzu/3satWzly6 +Lqi5dTFS9+hKvFMtdRb/vW4o9CQsMqL2BJGoE4uXvy3cancvcyodzTXCUxbesNP779JqeHy s7WkF2mtLVX2lnyXSUBm/ONwasuK7KLz8qusseUssvjJPDdw8mRLAWvjcsYsZ0qgIU6kBbvY ckUWkbJj/0kuQCmmulRMcaQRrRYrk7ZdUOjaYmjKR+UJHljxLgeregyiXulRJxCphP5migoy ioa1eset8iF9fhb+YWY16X1I3TnucVCiXixzxwn3uwiVGg28n+vdfZ5lackCOj6iK4+lfzld z4NfIXK+8/R1wD9yOj1rr3OsjDqOaugoMxgEFOiwhQDiJlRKVaDbfmC1G5N1YfQIn90znEYc M7+Sp8Rc5RUgN5yfuwyicifIJQCtiWgjF8ttcIEuKg0TmGb6HQHAtGaBXKyXGQulD1CmBHIW zg7bGge5R66hdbq1BiMX5Qdk/o3Sr2OLCrxWhqMdreJFLzboEc0S13BCxVglnPqdv5sd7veb 0az5LGS6zyVTdTbuPUu4C1ZbstPbuCBwSwe3ERpvpmdIzHtIK4G9iGIR3Seo0oWOzQvkFn8m 2k6H2/Delz9IcHEefSe5u0GjIA18bZEt7R2k8CMZ84vpyWOchgwXK2DNXAOzq4zwV8W4TiYi FiIVXfSj185vCpuE7j0ugp0AEQEAAcLBfAQYAQoAJgIbDBYhBKirxr6Vllfig9QtdXK25u9M WD0tBQJffU8wBQkWa0+jAAoJEHK25u9MWD0tv+0P/A47x8r+hekpuF2KvPpGi3M6rFpdPfeO RpIGkjQWk5M+oF0YH3vtb0+92J7LKfJwv7GIy2PZO2svVnIeCOvXzEM/7G1n5zmNMYGZkSyf x9dnNCjNl10CmuTYud7zsd3cXDku0T+Ow5Dhnk6l4bbJSYzFEbz3B8zMZGrs9EhqNzTLTZ8S Mznmtkxcbb3f/o5SW9NhH60mQ23bB3bBbX1wUQAmMjaDQ/Nt5oHWHN0/6wLyF4lStBGCKN9a TLp6E3100BuTCUCrQf9F3kB7BC92VHvobqYmvLTCTcbxFS4JNuT+ZyV+xR5JiV+2g2HwhxWW uC88BtriqL4atyvtuybQT+56IiiU2gszQ+oxR/1Aq+VZHdUeC6lijFiQblqV6EjenJu+pR9A 7EElGPPmYdO1WQbBrmuOrFuO6wQrbo0TbUiaxYWyoM9cA7v7eFyaxgwXBSWKbo/bcAAViqLW ysaCIZqWxrlhHWWmJMvowVMkB92uPVkxs5IMhSxHS4c2PfZ6D5kvrs3URvIc6zyOrgIaHNzR 8AF4PXWPAuZu1oaG/XKwzMqN/Y/AoxWrCFZNHE27E1RrMhDgmyzIzWQTffJsVPDMQqDfLBhV ic3b8Yec+Kn+ExIF5IuLfHkUgIUs83kDGGbV+wM8NtlGmCXmatyavUwNCXMsuI24HPl7gV2h n7RI Reply-To: Linux regressions mailing list In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-bounce-key: webpack.hosteurope.de;regressions@leemhuis.info;1717920671;017a12b9; X-HE-SMSGID: 1sGDdm-0000XV-OS On 09.06.24 00:00, Linus Torvalds wrote: > On Sat, 8 Jun 2024 at 14:36, David Laight wrote: > [...] >> I've done some tests. >> I'm seeing a three-fold slow down on: >> $ i=0; while [ $i -lt 1000000 ]; do i=$((i + 1)); done >> which goes from 1 second to 3. >> >> I can run that with ftrace monitoring scheduler events (and a few >> other things) and can't spot anywhere the process isn't running >> for a significant time. > > Sounds like cpu frequency. Almost certainly hw-specific. I went > through that on my Threadripper in the 6.9 timeframe, but I'm not > seeing any issues in this current release. David, what kind of hardware do you use? Johan Hovold as man-in-the-middle just reported "CPU frequency of the big cores on the Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency with 6.10-rc2" and confirmed "that once the cores are fully throttled (using the stepwise thermal governor) due to the skin temperature reaching the first trip point, scaling_max_freq gets stuck at the next OPP". For more details, see: https://lore.kernel.org/all/ZmVfcEOxmjUHZTSX@hovoldconsulting.com/ > If you bisect it, we have somebody to blame and point fingers at... Sadly that one is not bisected yet either. :-/ Ciao, Thorsten