Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp498116pxb; Fri, 15 Oct 2021 09:44:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyDsKzzsAnMeaSEsgFVmyuX/MYS06OU4fXdwGF0jxtMXSoim5zto/Vv3+BKyYZ9dmmqeUwJ X-Received: by 2002:a17:90a:6b01:: with SMTP id v1mr29232206pjj.6.1634316292154; Fri, 15 Oct 2021 09:44:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634316292; cv=none; d=google.com; s=arc-20160816; b=bNLo05N7yoZmx3BqingJ2la/gOimyK0WCJbPSWYcN62YXCShMr5y4Qo7C08qzmFv2w mU3wNaBm2Dngjo/WfJfTsGJ9bnjvc6cpMQGtqXQSZN/9d2wFOuVpblw7GtC3+PdaUk8Z GtfDyB9JIdpGqGvjuG/t/g6MKT/TRA7vGDSVhzVIVK2A/9+4pUQBuYR3ykfBzHow9h9m /ciu3NyJSM3ZA90w5lHH4pV15mlWcyTxrmoTlhBWcB6Erh30cId8SXn8ovejeHGhRNKF oabFbhEo3bDR/N8PpNYVt3GrZmF5/pnhs+B07LbUuTcsFv5rWdwHkA0y8Ke98cEltBQT QAYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=kAEEo2bMo6/W8jfykUA5d09Ap+RguoOMMLVNz+7/o4o=; b=V5BlEbgoMCoMihsPSF6WRFh79oGiDwACUguwal4XZ0AXV7h293n8N6Zzib5qfD2Zy8 cWzQ/2syxIYJ4Sd2+3gi5bjUjECUyxDoqKePdGnF/ZjhZ5w0RMGvBEYQnRhxZzxZbGp+ jlFCC/CJM1tcug3NYlBbhAkCA7yBihrhJWCegvohGnCfY9Pt4ygNlZG9J2vvpT5K3Xiq 0us8feyOq1dT3f4hHJFxM2XRtVP7oC1k1lzlKTX7LPwaOnVVL35u5wn4X99SDZ1yEkYH qsAwYCdF7XwfFUWyqOcaOJYx6qm3n9tkqo6nnVHhIXrD4HwP+rV5hU3iwjn7fGsY6Zxa Hgtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="qvoIa/Bf"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id e13si10446219plh.209.2021.10.15.09.44.38; Fri, 15 Oct 2021 09:44:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="qvoIa/Bf"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235895AbhJOIzD (ORCPT + 99 others); Fri, 15 Oct 2021 04:55:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235948AbhJOIy6 (ORCPT ); Fri, 15 Oct 2021 04:54:58 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE2E8C061762 for ; Fri, 15 Oct 2021 01:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=kAEEo2bMo6/W8jfykUA5d09Ap+RguoOMMLVNz+7/o4o=; b=qvoIa/BfsX2Ay4vRwY1vzpwBJe RoAsHNxUPycxn1csWxDtG4n0RKTV5kDmiZD3KuHaJu1zxxkzTrnxxTUsuQlEdbzasoTizeC7EbAOd LHm8DGPaTiO0/L8E1yoVFh9V+tg2ea74SoIWBq7HRQN5WZRnykJCGaQfwSdoaM7a1IytsoxDSgtDK PtHCftnH9dYsJtvj8E4fmACtH7UiDxcBQ0dg1HtRo29EYPOupf6GkXbtc/EixRha6oE3JaQEKS1z/ 61qc682ZOpiEofUqZP7xKt5lOTtVyUk1FpVPAMemJyRfTBXwq2hAatmOX8LDDwneIIkMbgnmBnZaq hJgP0LbA==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mbIx9-009yU0-Fg; Fri, 15 Oct 2021 08:52:35 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D285030031A; Fri, 15 Oct 2021 10:52:33 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id B964420125B2E; Fri, 15 Oct 2021 10:52:33 +0200 (CEST) Date: Fri, 15 Oct 2021 10:52:33 +0200 From: Peter Zijlstra To: Norbert Cc: linux-kernel@vger.kernel.org, Ingo Molnar , Steven Rostedt , Thomas Gleixner Subject: Re: Performance regression: thread wakeup time (latency) increased up to 3x Message-ID: References: <035c23b4-118e-6a35-36d9-1b11e3d679f8@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <035c23b4-118e-6a35-36d9-1b11e3d679f8@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 15, 2021 at 12:43:45AM -0700, Norbert wrote: > Performance regression: thread wakeup time (latency) increased up to 3x. > > Happened between 5.13.8 and 5.14.0. Still happening at least on 5.14.11. Could you git-bisect this?