Received: by 2002:a05:7412:1e0b:b0:fc:a2b0:25d7 with SMTP id kr11csp91797rdb; Wed, 14 Feb 2024 13:51:59 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVDxJxmofCoSJXr+KT/Yk+2aSaOBklmo9IJ6rWCDR85l6u69tYp5auoBaeTV5qITkXJzTZTbN6g8oBU0JoRbAQ2NeGkXVnwwUs/r/6Zrg== X-Google-Smtp-Source: AGHT+IG6udLHPI49HtLGCQAUVmDxo9D9IbfSJk/U4KF0A2J7cRA73XtYgASxQQ45dnd1WHq0WkQJ X-Received: by 2002:aa7:c24c:0:b0:55f:e574:4ea6 with SMTP id y12-20020aa7c24c000000b0055fe5744ea6mr3081258edo.2.1707947518874; Wed, 14 Feb 2024 13:51:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707947518; cv=pass; d=google.com; s=arc-20160816; b=JhoyS92YiXeNowiXLtmquI/hBZGMJ8pw6wLofw3A3+/5uKegqwIHW9lSbMcNcghCYx u2t1gVjhfyE9++eZ+oRWP8KuIo8LZBwv0CPcvHfxtiirnxf51Cw3VL5iwMBYjjxhNrpy 2WfDg/9dV4qKCSxUrE80b8DLtUUD3ROZ7ahB+8IROOSFIZu2y+L4tof5W8zKvrvgtsxQ 4bgboxOQU4rPCgkL5k/FtaYMHJ7D5aFFBeXS+J32pndUA4/RZP3bYd3i/dwEdXAIGKEt GAHvS/DJ9a7S0BApp88V5VeOJktTdRTe1aDuGlPMNfSQU/RdxDlf5XgsJ22bJ1Wbbp94 kLzg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:list-unsubscribe:list-subscribe :list-id:precedence:dkim-signature; bh=YSPUubyM84iE4if7k7+oIKp95nXKMAdaOOfobzF3zB0=; fh=2M32omrjWrLfj0IeDm2spXwOVC+0qOgNTeQ6rG8dpEQ=; b=tuY6U+BID9LX95BVLc9cAmpcQ8r+tsJptYH8n/eRui0W+St/hiReNHUOGj430Xn7TH opceAoN8WhTaQ+/nzUKXqglJHtlWSi55EDlFy5FES29bLpG/aeeGYnk7Wfaoy2fSZeSp vblIqCmR60VNWTpqEqUhahynyWb2hfyMZp4nZCngPjE1OcdpIjaAvXggdNu6rJEONvM7 d2YUF7l48n+BA1tjo9KvhOHWA0AjfnImyNRue3DdnSim4S4PujrSubtR0adbWJyIOwCF 8kDJbZXNgaqAkMLGWHXhwmj8mknJ0+KKfRs9mnq5YkiRW2wPBHaIn6i1nMYq8SpVSS8b DLxw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=R6uHT5gz; arc=pass (i=1 spf=pass spfdomain=google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-65998-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65998-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCVNHbF6QPx/qpadyMAmLB6mNuHXBVWC4L+YRS19ExM8UJwb+QCCjN8PfhHPx+eIGr7rv1Mfx/5aMa50dVM2xG4sPJZrsuzWOsK09gE/0w== Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id n17-20020a5099d1000000b00561be2897bbsi3233786edb.157.2024.02.14.13.51.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 13:51:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65998-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=R6uHT5gz; arc=pass (i=1 spf=pass spfdomain=google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-65998-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65998-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 7BACC1F29B7A for ; Wed, 14 Feb 2024 21:42:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 90DB313F016; Wed, 14 Feb 2024 21:42:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="R6uHT5gz" Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FEB113EFED for ; Wed, 14 Feb 2024 21:42:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707946943; cv=none; b=ssvS7iNuQNI13wxVYcwaQ134EDSX86I9ZxKjrj9UOdqpsJ8OLI964stWt4yBKflOLRnCpU07N+0chlP86lTQ7uMk3sMDCOoFq2ZokLmEOOUO1b4hyKsIlTAumowhKbO3+5nK4sIPprGIbAcHBQl6eZyyFyprW3VpgHdLNkGlosM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707946943; c=relaxed/simple; bh=3c2GIuuOcUIlF/1ZcnXuHB65jZ0swkmR0K4YPD0mw0M=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=mr4fR6CcD4GlV3ldHXFHk5YI6G4RRJ7uk5bVLYHri1uvo3l6FA1RthzOtes549phgzjeWyNMcCM7YH/QfCfvsFxPAx4Amg93xlxo7TCOOG+yJ4rjm7w/ByUdXbK1vhs1SsBTBjY+FN4ykE34yXotg/rMcpb3boul9v4fRC+N9rE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=R6uHT5gz; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1db35934648so7205ad.1 for ; Wed, 14 Feb 2024 13:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707946940; x=1708551740; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=YSPUubyM84iE4if7k7+oIKp95nXKMAdaOOfobzF3zB0=; b=R6uHT5gzBOPIvhY3BIQChUvPniZgQLgKUpwTFUCoIl3oB8tNk/I8MRw7xEotb/zX0Q mkCKfJvW6/CkNOr6ro5+yNc/3XkVUWwgLoZz1S34kpvmABHW++0L18t4pnFs/c3J0436 LGw3hg5+bzkpTOU3yxnjtOD/YSOBwG9oujJTsjs36QtHPu1c2nfGLQPLPzDBkwgSIcxy T/sv5YRoHIsci8PFiw972IlTW7CImJiBemDGZAqi5n1zbXgh9y/PBQqHhAXQlxx9GmnL DCdBnD0peYQBYSuVYkIdiS6equEuFxJrSrPY9AFEAcSw8YDrxOUq5vXNzdPO7X8pECXf DEgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707946940; x=1708551740; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YSPUubyM84iE4if7k7+oIKp95nXKMAdaOOfobzF3zB0=; b=lLt2cttQVKAqN7hLpNRPF/Yyl4SiBeSr9E9Bdv2U2IafBC9EzscgeCUBx2mYRZ80p7 9NlK8MJXuZf3JoaqNBFZ8sENIPBhNqcsHU6t19zvAiw7TQ7vd6VdEPMLnUUxJzikonFB d/GqgoxLd8R+2wz2u4f+SV9sQI+gjaCPgsQNU1wDucRHVaftm0zXjLyuWIn3bvRsfM22 YkGDomBymusqOG7gYzSMaVJVRIweBQsq7Fse0P/GOgIU5QksrzwO07YGI5C8LDDYghjz Ca4lWOrge8Kc2cV+ttcFm3sQrrOWwIWecJFcHg+GJ/hcN6BYncxOoZIGImKrvmfg+kMN YRMg== X-Forwarded-Encrypted: i=1; AJvYcCU+/85yuOQIdPVHlyLG0nWNaHIvyym0Gp37t+LVFa/W3GWxTYsStMLGVRkYKvaLCh4UlcHDw9KTdXSU9c6Gcg3Wb0unbaqRbGXwSYX9 X-Gm-Message-State: AOJu0YyobCjPbwoUT2ss1hCXniZYFbvq9xHH/wbN17TQxBCis6J8RyuX 3rRK2h74qKL0PuyaMCYGgdlXcytTgbH2AcUvQtOab+BDccu5re3/5NqRk33v0cpurm5imE0g0Rv vCsR12xnpuT4fNsRe17kGDfRYszoeB/zIHbTF X-Received: by 2002:a17:902:7b94:b0:1d8:d90d:c9ae with SMTP id w20-20020a1709027b9400b001d8d90dc9aemr335917pll.1.1707946940434; Wed, 14 Feb 2024 13:42:20 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20240214063708.972376-1-irogers@google.com> <20240214063708.972376-3-irogers@google.com> In-Reply-To: From: Ian Rogers Date: Wed, 14 Feb 2024 13:42:09 -0800 Message-ID: Subject: Re: [PATCH v1 2/6] perf trace: Ignore thread hashing in summary To: Arnaldo Carvalho de Melo Cc: Peter Zijlstra , Ingo Molnar , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Adrian Hunter , Oliver Upton , Yang Jihong , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 14, 2024 at 1:36=E2=80=AFPM Ian Rogers wro= te: > > On Wed, Feb 14, 2024 at 1:15=E2=80=AFPM Ian Rogers w= rote: > > > > On Wed, Feb 14, 2024 at 10:27=E2=80=AFAM Ian Rogers wrote: > > > > > > On Wed, Feb 14, 2024 at 9:25=E2=80=AFAM Arnaldo Carvalho de Melo > > > wrote: > > > > > > > > On Tue, Feb 13, 2024 at 10:37:04PM -0800, Ian Rogers wrote: > > > > > Commit 91e467bc568f ("perf machine: Use hashtable for machine > > > > > threads") made the iteration of thread tids unordered. The perf t= race > > > > > --summary output sorts and prints each hash bucket, rather than a= ll > > > > > threads globally. Change this behavior by turn all threads into a > > > > > list, sort the list by number of trace events then by tids, final= ly > > > > > print the list. This also allows the rbtree in threads to be not > > > > > accessed outside of machine. > > > > > > > > Can you please provide a refresh of the output that is changed by y= our patch? > > > > > > Hmm.. looks like perf trace record has broken and doesn't produce > > > output in newer perfs. It works on 6.5 and so a bisect is necessary. > > > > Bisect result: > > ``` > > 9925495d96efc14d885ba66c5696f664fe0e663c is the first bad commit > > commit 9925495d96efc14d885ba66c5696f664fe0e663c > > Author: Ian Rogers > > Date: Thu Sep 14 14:19:45 2023 -0700 > > > > perf build: Default BUILD_BPF_SKEL, warn/disable for missing deps > > ... > > https://lore.kernel.org/r/20230914211948.814999-3-irogers@google.com > > ``` > > > > Now to do the bisect with BUILD_BPF_SKEL=3D1 on each make. > > This looks better (how could I be at fault :-) ): > ``` > 1836480429d173c01664a633b61e525b13d41a2a is the first bad commit > commit 1836480429d173c01664a633b61e525b13d41a2a > Author: Arnaldo Carvalho de Melo > Date: Wed Aug 16 13:53:26 2023 -0300 > > perf bpf_skel augmented_raw_syscalls: Cap the socklen parameter > using &=3D sizeof(saddr) > ... > Cc: Adrian Hunter > Cc: Ian Rogers > Cc: Jiri Olsa > Cc: Namhyung Kim > Signed-off-by: Arnaldo Carvalho de Melo > ``` > No LKML link. Hmm.. basically that change fixed the BPF program to verify and so the problem has been long standing with the BPF code. Maybe perf trace record never worked with BPF. Thanks, Ian