Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp547168pxb; Fri, 15 Oct 2021 10:44:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzZjxwLoXXiTfOxhre+kCDHBmkqm0sw1hQbFnATXMuKPjOr5sypp9LUTHQBWMmqk+3jTNhU X-Received: by 2002:a17:906:3715:: with SMTP id d21mr8911366ejc.74.1634319872130; Fri, 15 Oct 2021 10:44:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634319872; cv=none; d=google.com; s=arc-20160816; b=lvGz5aL/cO6U3o6C66rqbSshhHIIm9l2ytIskxqFKIGKUyRREfY6X1pAXR2Qj4/TTw YclT9Ip4RYuqwM5IKnq9SSIDAODs5Zqp4dhfViTIadwPRcvlVLSaf+pQMtiR4TgcstlY Wl2LgBXH/KxDdVU6lde+tyu/4bYAXmnVApDZV/imLADTan6UsOuG0kGISdCn9kdJuiI3 F/71aYKYMKgGpdZjU7WkuRkvdoDl9T+3bNu91SQmHK5gfPvNSfaqXC3OZT1WotaDuglX gEwrWClt5AP7elZjLCpw3Al3GQNMmwWhSsrhYf2SOXU1NiBZmF516K8jPW+3NConLDzt meEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:in-reply-to:from :references:cc:to:content-language:subject:user-agent:mime-version :date:message-id:dkim-signature; bh=A+kABcpWl5XxkQdy8GVu6UutPeMvCwc5El4ILSgOxyA=; b=eTAhKbfN4eFSyO3ffQuPfaRvB/krEv9jyagQGE0YxAo/06j0oZxibZzFHnKkYKouLq 98Aah/Sna4rsPEve95LfqrLo3nhLVBCvBnVLw1/3WFHzyWQjTBsia0JajtMO3oFosHRM v2nNip1sQqG9ruwdxy1wSgNvByrt6a6rAtrC7VSqCC63bQhnZxMqNJDCLd0an1IhGSf1 KOKP9HyxIpmRVKHLCUqVEM8smcbFGCSC238E3C99J81wrFY3IAOPkXY1Gs5qZObCVbUk 1hyq7YLpw2WnOsN4Im5O7l5xCZhEySM3zYRXf0b+jVu5UbJnIeVHjz0iSpnxTcNHHMYx KXIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=CTckGThW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id v19si8311440edw.480.2021.10.15.10.44.09; Fri, 15 Oct 2021 10:44:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=CTckGThW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237537AbhJOJiM (ORCPT + 99 others); Fri, 15 Oct 2021 05:38:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236727AbhJOJiL (ORCPT ); Fri, 15 Oct 2021 05:38:11 -0400 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9191C061570 for ; Fri, 15 Oct 2021 02:36:05 -0700 (PDT) Received: by mail-pg1-x52b.google.com with SMTP id r2so8067868pgl.10 for ; Fri, 15 Oct 2021 02:36:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=A+kABcpWl5XxkQdy8GVu6UutPeMvCwc5El4ILSgOxyA=; b=CTckGThWx99BeEmmpCecOoOFRE4LurGhMWexxNlLQEyaC0LOXYXQ76c0Tb9sAxEExj SLyqtARZhFq4DnlaAGSjeOenWfNltBzPd2xO47x/Vo03PPDgkIQV2bIqy3oKnTEjdjx5 NttU0KD1eC+g36CXTORH6FOtmZ2++jfeHzJJME+zapW931RIZBLdym/5HZoqNRvrFIpX nIH5NhZRW11qXZCSzQi4AHabqbYZeKkCkuOVUl2LKcJzASTZqw59JqUxF4cP0Bee9t2W YsQV7rDU/qFggGuEs21Z31PagYH+9n9eCzEosLxyYI9egm2i8x9S/7I5OiOXr6JDXuor VxCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=A+kABcpWl5XxkQdy8GVu6UutPeMvCwc5El4ILSgOxyA=; b=ufBfrWktOPtjG+LvzB194V4sg2WKd1rYT43+dBH4wdQgHlTW+xzgyKKQ0Toxp4TTEK T0AcnJW2XmIzg32KQi3FkFhOUZ1QV0lVytrVOxzXJkTbSpq4L31dSA/yrIJz4HeGt1rG E/Z26jbWzqmUkWbd3xyCmfRqdmETLoGp6RhNmhHig29WgQx4Sb70i73CY4Kzikgkabf7 hvsTeGZqAVVqs3Kzu5ab5jUYvDTNHN+dFfJdXjt1foOIAbZjXcU4JRARHBlLcSFG1EQy 4oHY0g7PUNDOcDyWXNV6nCrhTFEhuA0Fsc9yPkNvbFRbMiYsCQTn7Z03NQ+f6iT1nr9p uGRw== X-Gm-Message-State: AOAM533sDo+Oy5CW+Y4h8e7Q25gxO7ppWn6O54vRZc65TSDOtuSxBLvT oO2Owyd3htsax2XGu/yo8/M= X-Received: by 2002:a63:8f02:: with SMTP id n2mr8413116pgd.270.1634290565423; Fri, 15 Oct 2021 02:36:05 -0700 (PDT) Received: from ?IPV6:2601:645:8400:1:ffb7:9e6f:baa:dfce? ([2601:645:8400:1:ffb7:9e6f:baa:dfce]) by smtp.gmail.com with ESMTPSA id e15sm4383332pfc.134.2021.10.15.02.36.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 15 Oct 2021 02:36:05 -0700 (PDT) Message-ID: <5fe0ffa5-f2db-ca79-5a10-305310066ff9@gmail.com> Date: Fri, 15 Oct 2021 02:36:03 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.0 Subject: Re: Performance regression: thread wakeup time (latency) increased up to 3x Content-Language: en-US To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Ingo Molnar , Steven Rostedt , Thomas Gleixner References: <035c23b4-118e-6a35-36d9-1b11e3d679f8@gmail.com> From: Norbert In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/15/21 01:52, Peter Zijlstra wrote: > On Fri, Oct 15, 2021 at 12:43:45AM -0700, Norbert wrote: >> Performance regression: thread wakeup time (latency) increased up to 3x. >> >> Happened between 5.13.8 and 5.14.0. Still happening at least on 5.14.11. > > Could you git-bisect this? > So far I haven't built a kernel yet, I'm quite new to Linux in that way, so it may take me some time to figure it all out, but yes. (By the way, of course I meant that throughput *de*creases, not increases.)