Received: by 2002:a05:6500:1b45:b0:1f5:f2ab:c469 with SMTP id cz5csp1143056lqb; Thu, 18 Apr 2024 01:01:10 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCVt8a8HXSIGSoRej8watLz18y6NeIAeEYbsFHapiWAv7EQVuHIZGYx+U8jU7gjtN983q/Av3nGf57uwnl3C+PWp/avuOrgi3aeJuXmhlA== X-Google-Smtp-Source: AGHT+IFr4XcT6TGsyZnHiOcM3zmd6eRi2WOTv0ItIRvUEqodU11b7XusXNLEK6iNI4vOPziZsEMo X-Received: by 2002:a17:902:d2c5:b0:1e8:7906:5be3 with SMTP id n5-20020a170902d2c500b001e879065be3mr2177684plc.18.1713427266775; Thu, 18 Apr 2024 01:01:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1713427266; cv=pass; d=google.com; s=arc-20160816; b=cXt9zBonTcuUfeetd+nbkRVTw+wzw5RbHWMoG3GjZpdZFJO/m2nsTKUJJJKLfFA0/c Z9rX4CB/PAJO1rixELT1eVCwEJhLo+PhwyZNMyDRm67lnrD17sGnEZK6JPAgGJCB1ejh uR5BMopJbuI0CubkDy6Q8+Pe1Oj5zdSGPWhjHvyQHlheEE40BW/QcXCHbLHUZU8JrPGQ pva7CPIURT3CKI/mg3QR0yoEtfKRTr3uS4foV1B5nkzu6ei3GuttVM9Fn25y8YUGJ2nS aHrldnuzEntiZ3/ZjJCo9rAQineE1l6iEV3nTA9jiipnjrDsirSmQoM2g3L7YJ5H2atr RIdw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:date:message-id:dkim-signature; bh=zu7l8wbiLGqVB4JdaBgTKGQLvd53sCWlM9pIPM62Tbw=; fh=Negnb/xdmkKNGbvqslYc97JPnxjgTdhUu88jKbUC4dM=; b=Ulu21v09hACL8cCkYQK9rZWcYGLNOUiYMi5dB00m98yxxjUgYxiR2vQoyiHihdQzym Y1hprA2TTAaKkMYH90AgnYizsN16S5GrU1xfdFAaHLq5cOn2wexMK60V2Y/mJ2aSwg1L mSKWYAepMXGMX7nVLpxWZqUz2yga1F+pBqKCoYGvkBj024jviCZ3fCcTbICDgpRZX+uf ySYjVRWC0bhf0k4iwTye/siJcUdJxVh9W0e5xFl//gT2S3MX5eyScQqC01c5LVNltDwv wnGyNxviTQaNT6fJgwaw8GjdaxA0fbRLdjJdDUcmGQ4sBdvXlfYwo1Gm+6KutpuQTBND ADaQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gbNnVMCs; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-149687-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-149687-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id q5-20020a170902dac500b001e01baf194asi958618plx.472.2024.04.18.01.01.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 01:01:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-149687-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gbNnVMCs; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-149687-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-149687-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 6554D283EC6 for ; Thu, 18 Apr 2024 08:01:06 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AC1DF7D09A; Thu, 18 Apr 2024 08:00:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gbNnVMCs" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5EF5762E5; Thu, 18 Apr 2024 08:00:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713427221; cv=none; b=JNnGOzqHuzxJ1R1tVW0WiXEFqh4B2mxfoz4NLm07HwRkCVnIJUi5nq/u0ny21+HhacMm9EK4S3h+nqIxfbEzTHjfC5BxiXIW9I2Q2Dg04LFHL+f3km4JZ2YNWaiJIkrzlOOQZx1UinxxmGzV+crgW05jNZE0z2vlX/Pb3qhYEEY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713427221; c=relaxed/simple; bh=8xkMYyuEcKoA6xltCA5FHMbWSOH1kFJ6bzMXvN0SpMY=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=uNDUHBDPEAilRWh276aOOC4QADBr9fOqHJJSrL+CKpoUI/LaXaBKGySkVcBLSIMeRg5p1i4tCMraTIgzJu4mT+M+farwVV5uuMQfPBcVKwbZTrc9jPgNrNoHZekCVYo78AJlSw8oVje6fDLE5HqwOVR490JcdJVLtDX/RsREoGg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gbNnVMCs; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF511C113CE; Thu, 18 Apr 2024 08:00:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713427221; bh=8xkMYyuEcKoA6xltCA5FHMbWSOH1kFJ6bzMXvN0SpMY=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=gbNnVMCs3OpZ7tw3N/dPN0SJ6ZdFxArs40SVqZA5Ck30OuzATWr6xJPV4JA3o4Yr1 b1Wcy7Nd/n4ZsLB/XDkxkqHZE2hU6cfwI2oquZcJEnYExFg4JHxetExSmOPZS0gc3S hOxVkGla6AERdW7iJxLvvHcNwIUKVXX3kZJ//Bvugn/wv6J4By7WhNBrZLzp8O1jrZ rW6HXz2lIB9WdNjkGOhNS2Kcb1MGluHl5pThVJ8QIiXJexT6mkhFytklyPkdimV+iG R07jRsGzXA3zgtZLW6yz+cB+5YfJPI1/mAitJZTjpGjQSAdBouZwbYfkrDmrHpzeR2 YzC017NummtaA== Message-ID: <2f08b2c8-a931-4fc9-85c1-e63b49dbfebc@kernel.org> Date: Thu, 18 Apr 2024 10:00:17 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1 1/3] cgroup/rstat: add cgroup_rstat_lock helpers and tracepoints To: Tejun Heo Cc: hannes@cmpxchg.org, lizefan.x@bytedance.com, cgroups@vger.kernel.org, yosryahmed@google.com, longman@redhat.com, netdev@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, shakeel.butt@linux.dev, kernel-team@cloudflare.com, Arnaldo Carvalho de Melo , Sebastian Andrzej Siewior , mhocko@kernel.org References: <171328983017.3930751.9484082608778623495.stgit@firesoul> <171328988660.3930751.17537768209042139758.stgit@firesoul> Content-Language: en-US From: Jesper Dangaard Brouer In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 16/04/2024 23.36, Tejun Heo wrote: > On Tue, Apr 16, 2024 at 07:51:26PM +0200, Jesper Dangaard Brouer wrote: >> This commit enhances the ability to troubleshoot the global >> cgroup_rstat_lock by introducing wrapper helper functions for the lock >> along with associated tracepoints. > > Applied to cgroup/for-6.10. > Thanks for applying the tracepoint patch. I've backported this to our main production kernels v6.6 LTS (with before mentioned upstream cgroup work from Yosry and Longman). I have it running in production on two machines this morning. Doing manual bpftrace script inspection now, but plan is monitor this continuously (ebpf_exporter[1]) and even have alerts on excessive wait time on contention. It makes sense to delay applying the next two patches, until we have some production experiments with those two patches, and I have fleet monitoring in place. I'm be offline next week (on dive trip), so I'll resume work on this 29 April, before I start doing prod experiments. --Jesper [1] https://github.com/cloudflare/ebpf_exporter