Received: by 2002:a05:7208:9594:b0:7e:5202:c8b4 with SMTP id gs20csp1203852rbb; Mon, 26 Feb 2024 01:57:15 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCV6IHddZ1pp3hXcnkxhKlu8RjlcurlimkHg4dFlTyWgDlu/SVwXr2eJMf+Jt+V4lHIPSK9RhEARMCtwZ84P38Gbxm2ZSUIazSD5QDDJNw== X-Google-Smtp-Source: AGHT+IGewFbmH7s/pXCnvSM3pq3L8xIlruIxvJ5gpMIBH3soZ6C7nTQRL1F79KAsj4why0x9dWcl X-Received: by 2002:a0c:cb89:0:b0:68f:43ec:7df7 with SMTP id p9-20020a0ccb89000000b0068f43ec7df7mr5425269qvk.37.1708941434866; Mon, 26 Feb 2024 01:57:14 -0800 (PST) Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id a14-20020a056214062e00b0068f94ee3422si4680876qvx.467.2024.02.26.01.57.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 01:57:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81049-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@google.com header.s=20230601 header.b="Dr9Tm/Za"; arc=fail (body hash mismatch); spf=pass (google.com: domain of linux-kernel+bounces-81049-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81049-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=REJECT sp=REJECT dis=REJECT) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 870CB1C2554B for ; Mon, 26 Feb 2024 09:57:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 18CD554F9F; Mon, 26 Feb 2024 09:26:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Dr9Tm/Za" Received: from mail-ua1-f51.google.com (mail-ua1-f51.google.com [209.85.222.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93D6054F8C for ; Mon, 26 Feb 2024 09:26:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708939598; cv=none; b=Grqx0Ywt0D8/akKjw3rxgpV3FQGBCMQxvLBRtj5vS7eaJzhkk45EXjifNO5ywGwCuwsIWTkN0Ye/Sqi2273Kr8922mKNVV2EnAOazEleaKfiLmShLpq8oaA5mULebG9uBzEgKOp/4bjThAOpKPr6yjB7sRgZmUlI9OTwiezS6hA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708939598; c=relaxed/simple; bh=9deL/d/BcOgJ90IBVOvhsZVsKJ8Cuz+9AZRxFu2eesM=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=jffTDaCC9B44EadAuY1SBlo4QVjBDPFE87yGTuJgHCo2LADkwsCJ/qBwWRzJrLgAljfDS7dfgndA07svqYn+1gD0lF+uzJWFb3s0uXXiw+kNqdTII1ItnPSGCajDDexePDpKyzFoXSfH0y/P34wqY6lByiuFfLcYCZTW3gDCKi4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Dr9Tm/Za; arc=none smtp.client-ip=209.85.222.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Received: by mail-ua1-f51.google.com with SMTP id a1e0cc1a2514c-7d5cbc4a585so877489241.3 for ; Mon, 26 Feb 2024 01:26:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708939595; x=1709544395; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=9deL/d/BcOgJ90IBVOvhsZVsKJ8Cuz+9AZRxFu2eesM=; b=Dr9Tm/ZatKZhrXF9KI8MCnZw7ZhFyK0ELsljHMsMzle3S2NJuT8QQkUsgLoby6CLSI 4ehZktOHqg4bfJUqiucIMaRpZI41RCI4MbWji2w2k4ciK6Z7GLYvh1alj9a15fxB9/Y1 5lXdwn+JQJhH8UEV8ZzXrfP5tf7G72AlqVLzM16fv0wrQT9Pli2K/AtW4fyqo9ghmN2k G9RoYFkNOG73TTyY9QdLkeZyZFYFOqKpg9LwJS5C+1TbjqwVF0uBRDaLHhQipKYuDlX+ OS1LAEDxKO3IDiOe1dXavOwXIC2rRoC4FY4mWznMJ/1YMi4kOeUjVWgBM7QRWR3AZrHr oGrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708939595; x=1709544395; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9deL/d/BcOgJ90IBVOvhsZVsKJ8Cuz+9AZRxFu2eesM=; b=oh7SI1cWmsnXOslrjoDdxCuoQkAajjaSgffcRk8Prx7AJC19IPPa34jfpFkMgCkwOV hNzmew+/bcG/iucg7hw8ZyZFehHyECPBKyzjQ/4gaoEBN/5/G80s7HGW7JKPxYGlmpaW /tCKmHEXHj6+gxZfk1VZlXslzde2mm/4knaegBeSVd/kOga8N6vdtJuPoverCC+WTJNa FCXp77RFFlkhLVr3PGcVE8zEz4dHYDIY8KOHql64G8vqi+VGeETOSiZ/C9sTquN5V34p eXsH4Qjj/HJfo/DOkxDO5ed4LKjg+5eTshCEK6aNRyGDlXblsrRBgK1QcqMGqXcBkAnX xCOw== X-Forwarded-Encrypted: i=1; AJvYcCXWueFEvhTEhwavq7CLZ+pt7hNifLQj1Z5QT20eZGEto1aUL5oDe37JdDm1bBdwV1lWjupKmbwYTInEZ5gaeWX4I7KM0jAKHGrJpaMY X-Gm-Message-State: AOJu0Yym7FfLKJkmKLfIZFYpBQpALacP4YhtavUCTst03AwR6z0gD/UE 01wAXuF8BW/DH1w3NtFO4xM15Xl5H+7DGFY1ehe1QW4d9TI22Q/nhEPv35crkl06DJhtfsp3Rwd oL8NvNuRXt1CGadBL+/niUElJms5M3muc6YSdRAgU8czzoM2oI5yg X-Received: by 2002:a1f:cc84:0:b0:4b6:d63c:ca8f with SMTP id c126-20020a1fcc84000000b004b6d63cca8fmr2397957vkg.16.1708939595308; Mon, 26 Feb 2024 01:26:35 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <91c50335-e7b6-4ae1-9dad-a0c990b52021@suse.cz> <20240219152836.11d36709c594e66fe3037f2d@linux-foundation.org> <7e31accb-db01-486f-afb8-18a3f5402d00@suse.cz> <20240220093011.bf84486d704c3814079c2aa0@linux-foundation.org> <96c51d35-15ce-42d0-b81b-7e76044e1f2b@suse.cz> In-Reply-To: From: Marco Elver Date: Mon, 26 Feb 2024 10:25:56 +0100 Message-ID: Subject: Re: regression/bisected commit 773688a6cb24b0b3c2ba40354d883348a2befa38 make my system completely unusable under high load To: Vlastimil Babka Cc: Andrew Morton , Mikhail Gavrilov , Andrey Konovalov , glider@google.com, dvyukov@google.com, eugenis@google.com, Oscar Salvador , Linux List Kernel Mailing , Linux Memory Management List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 20 Feb 2024 at 19:51, Marco Elver wrote: > > On Tue, 20 Feb 2024 at 19:16, Vlastimil Babka wrote: > > > > On 2/20/24 18:30, Andrew Morton wrote: > > > On Tue, 20 Feb 2024 10:37:03 +0500 Mikhail Gavrilov wrote: > > > > > >> On Tue, Feb 20, 2024 at 4:50=E2=80=AFAM Vlastimil Babka wrote: > > >> > > > > >> > > I'm all confused. > > >> > > > > >> > > 4434a56ec209 ("stackdepot: make fast paths lock-less again") was > > >> > > mainlined for v6.8-rc3. > > >> > > > >> > Uh sorry, I just trusted the info that it's not merged and didn't = verify > > >> > it myself. Yeah, I can see it is there. > > >> > > > >> > > >> Wait, I am talk about these two patches which is not merged yet: > > >> [PATCH v2 1/2] stackdepot: use variable size records for non-evictab= le entries > > >> [PATCH v2 2/2] kasan: revert eviction of stack traces in generic mod= e > > >> https://lore.kernel.org/linux-mm/20240129100708.39460-1-elver@google= com/ > > > > > > A can move those into the 6.8-rc hotfixes queue, and it appears a > > > cc:stable will not be required. > > > > > > However I'm not seeing anything in the changelogs to indicate that > > > we're fixing a dramatic performance regression, nor why that > > > regressions is occurring. > > It's primarily fixing a regression of memory usage overhead for > stackdepot users in general. Performance is mostly fixed, but patch > 2/2 ("kasan: revert eviction of stack traces in generic mode") also > helps with KASAN performance because entries that were being > repeatedly evicted-then-reallocated are just allocated once and with > increasing system uptime the slow path will be taken much less. > > > We also seem have an unhappy bot with the 2/2 patch :/ although it's no= t yet > > clear if it's a genuine issue. > > > > https://lore.kernel.org/all/202402201506.b7e4b9b6-oliver.sang@intel.com= / This was confirmed to be a non-bug by RCU devs. > While it would be nice if 6.8 would not regress over 6.7 (performance > is mostly fixed, memory usage is not), waiting for confirmation what > the rcutorture issue from the bot is about might be good. > > Mikhail: since you are testing mainline, in about 4 weeks the fixes > should then reach 6.9-rc in the next merge window. Until then, if it's > not too difficult for you, you can apply those 2 patches in your own > tree. There are more issues that are fixed by "[PATCH v2 1/2] stackdepot: use variable size records for non-evictable entries". See https://lore.kernel.org/all/ZdxYXQdZDuuhcqiv@elver.google.com/ This will eventually reach stable, but it might be good to reconsider mainlining it earlier. Thanks, -- Marco