Received: by 2002:a05:6a11:4021:0:0:0:0 with SMTP id ky33csp2863813pxb; Tue, 21 Sep 2021 09:15:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwbzITj4hr9Sax9oyh6JZ9688xGgLG8YFeMuGAPFi+X9siyi927YuarjovaxeODSQ1aGR1d X-Received: by 2002:a17:906:1ec9:: with SMTP id m9mr34598562ejj.115.1632240936763; Tue, 21 Sep 2021 09:15:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1632240936; cv=none; d=google.com; s=arc-20160816; b=bdPoUROEUbsrhm97qvymUrxA/mZ37mxAYqXfuB5Iy74YLR6B6+p/+lzG9QWrKu+AQC dw8KwWBkhoCmviN+zkC+kWpPtBUN5+q72jIXLnplo/3DikQ3//6Sz6AvMzI8Pp0iFM0u Lfx5KAtg7H1jvGGPfD4vI7a/KdjBjIEOu+zRzPEO0upADzLcQQZceFgHSmQDul492Sy8 Oo+/Gt23V4dwxwvUBuLD0OStW84epKI8YLLXkerIexAoqxkFRfhwq/soi9Nw1FBmvv0k BFLDyNd0vwFe0J8mib66Q/EOSChihUMfCVa1hjDeL2Plmphqn0NJ07hewfng0bdTvppi iHow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=qpu+HTkDrTU2emAVPYaio1EuujPd/r4QIzzzcyyaBgg=; b=WFlUVwHayknzCvhoqWDrjRA3KloWwLyt83p7aoxropGYreWpfY1e0VNMBKXA8g/J55 GBhQEeKhHEoxLCcX3HpwNHpcFCBe5dx2P2watktUftMp2kXDt+Vgf+octB04FsJSAXNY 3pplMbRkzOfICLR6rMBBbBkX1kMiP0wTpWkJcRDDvFvEeEjEt3bxh02lBZTxSJLi/m4u /uaXNXKGhEw1Q4FG+h9o6k2calOYYnIaRBH6he4ShXvDovcUkvi/+Hu12gGRa/VqBldu TVm7dkgLxXjOC+9roiUxFtI02DXi9LXBy6DTd0JhK2gbJaWPvXq++RJcgykbBs0OjiE0 DLxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TkhoJv3S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id kl18si21063197ejc.160.2021.09.21.09.15.07; Tue, 21 Sep 2021 09:15:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TkhoJv3S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233792AbhIUQPF (ORCPT + 99 others); Tue, 21 Sep 2021 12:15:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:54295 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229804AbhIUQPE (ORCPT ); Tue, 21 Sep 2021 12:15:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240815; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=qpu+HTkDrTU2emAVPYaio1EuujPd/r4QIzzzcyyaBgg=; b=TkhoJv3S+N9FgvKAQgv269srhFF3NGONY3IruZJpi56O+XEg8aDS+m0GGrjGI8gOq4JavC et8jjpI1P7dbyK5YwBdCFKRVMzmIe2FyHsO87IfvavQ2BX/4DuWE2CrZVA7UnP6kuQEc5W kQ5UUhg6RtNVNm1GKEPh1PluI2eloHA= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-298-eEH2KyFIOZGl1OvDjlYSrw-1; Tue, 21 Sep 2021 12:13:34 -0400 X-MC-Unique: eEH2KyFIOZGl1OvDjlYSrw-1 Received: by mail-wr1-f69.google.com with SMTP id k2-20020adfc702000000b0016006b2da9bso4200026wrg.1 for ; Tue, 21 Sep 2021 09:13:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qpu+HTkDrTU2emAVPYaio1EuujPd/r4QIzzzcyyaBgg=; b=IdUg+HN+96ofWm0xNCvqbTYxOjKZNyxGLYK4G3Za9xW5PskvviFs98DFGQV2OWgeWs yXsdjX4FRRRQF5XjtM3xw4v6WDUrlgzmr41OGggVOLkMkP7eL6XFJiFcy0W4uuEse27Q E4qVRwL+93hqFkE3OAb27m+IH//SYIZrOo8ehenqXOMIGqEGPyj7LlaHbcd6vgK8EAZv JclIYQTt9AP9VzZQcs16AwUd66MdBZWXPqRpsJbsPTAJz5AGEp04XAfrRK4Zmiy0u3WH Bl+U9dZvxucLTXPJUp7mmhKcIXwGGUEhY1ioaBqByWDaBmyJwDL3KwPdVlA1fALScQcy Ouog== X-Gm-Message-State: AOAM5312IlRTrXXc/iS1TtGdkoQKZrz2SYt0+7GicsIPHOIs6TQnAvJH fuVsWKS8vL1nxJhOx7rQF3gnulR8uY79z0l9KvP9EMGzLoBqBU8+pEYRohT79b0IKOCPNiWz7h3 mrMQOOp73jUvP4FIiXROMGLgF X-Received: by 2002:a5d:6c6e:: with SMTP id r14mr8786531wrz.319.1632240813485; Tue, 21 Sep 2021 09:13:33 -0700 (PDT) X-Received: by 2002:a5d:6c6e:: with SMTP id r14mr8786498wrz.319.1632240813317; Tue, 21 Sep 2021 09:13:33 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:32 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 0/6] mm: Remote LRU per-cpu pagevec cache/per-cpu page list drain support Date: Tue, 21 Sep 2021 18:13:18 +0200 Message-Id: <20210921161323.607817-1-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This series introduces an alternative locking scheme around mm/swap.c's per-cpu LRU pagevec caches and mm/page_alloc.c's per-cpu page lists which will allow for remote CPUs to drain them. Currently, only a local CPU is permitted to change its per-cpu lists, and it's expected to do so, on-demand, whenever a process demands it (by means of queueing an drain task on the local CPU). Most systems will handle this promptly, but it'll cause problems for NOHZ_FULL CPUs that can't take any sort of interruption without breaking their functional guarantees (latency, bandwidth, etc...). Having a way for these processes to remotely drain the lists themselves will make co-existing with isolated CPUs possible, at the cost of more constraining locks. Fortunately for non-NOHZ_FULL users, the alternative locking scheme and remote drain code are conditional to a static key which is disabled by default. This guarantees minimal functional or performance regressions. The feature will only be enabled if NOHZ_FULL's initialization process was successful. This work is based on a previous series by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1]. [1] https://patchwork.kernel.org/project/linux-mm/patch/20190424111208.24459-3-bigeasy@linutronix.de/ Nicolas Saenz Julienne (6): mm/swap: Introduce lru_cpu_needs_drain() mm/swap: Introduce alternative per-cpu LRU cache locking mm/swap: Allow remote LRU cache draining mm/page_alloc: Introduce alternative per-cpu list locking mm/page_alloc: Allow remote per-cpu page list draining sched/isolation: Enable 'remote_pcpu_cache_access' on NOHZ_FULL systems kernel/sched/isolation.c | 9 +- mm/internal.h | 2 + mm/page_alloc.c | 111 ++++++++++++++++----- mm/swap.c | 202 ++++++++++++++++++++++++++++++--------- 4 files changed, 253 insertions(+), 71 deletions(-) -- 2.31.1