Received: by 2002:a05:6902:102b:0:0:0:0 with SMTP id x11csp2036441ybt; Thu, 2 Jul 2020 22:11:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxdRySREdi37YkuyuCkoirj3OF9/QjHRkZj2G1z6ecN1UrEpwk5pJDIzHvhSMpEC1Y7p3As X-Received: by 2002:aa7:d8cf:: with SMTP id k15mr37360071eds.250.1593753063583; Thu, 02 Jul 2020 22:11:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1593753063; cv=none; d=google.com; s=arc-20160816; b=0oofOWZirgasjHNfC+PH2Xe+J3hRBhtWxhqgosvd3EEln4E6lmR+U98WZDgypNuR70 LOXqmiBlH/THtoicP/i1D2aykzH9dUIxWBdN7QkdyabPgSUd1zpx3yeJdSQApKkHoqYt iwlD+gZTWYVfoVZGUPl5gfIBseOMx05xevoqb5q/2j0bzAovKF6eN3fHQ4ICgR/XYSri kYGI4LTEqF4gJkN7Xr5eiXM5lO2v4cL1p+NJwxzLZJ8N4nX86Ioj63D21vPKnmbn+TR3 J6FQD2GAT3kgfXQ3KsUxV7DqdgmFOaA+8dNbV7ibW5QjxUFT9z98lYk5RzluBKyrxx6C qyxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:to:from; bh=FAnPFNS5XP7LMb/tBFbXH99C+K6RueZ12QXDW8zrvKo=; b=k38oJkeMSRNI52MdgiXg2wwLfL/FIzHYwvpbrHCN/Hm/1FKljbYH2FkURHgSePVHgY gi09H+cMwJ/AU0vNXOkfMGcQdIoL8gQ2bYQ0QnScPc8hsbvLnChYefuI+BzxBEHVKNPC B7i8JEwdHW/G9imrE0aW4TwoBjMSqYBGQCpqmSA5ZNl6eCxuOngl5d6hjPtj+ZBRcPmj Fk4mlcG9ByrMS+Uks/n5yQRvRxKZ2QoddjeUJolEyGZ2HTRG+RXPMFT2cuOwpG2jw0/6 hVC+8xZ3tOgwFnoalgUDjg5nPNWAoCc3+PJpWK/jiAolay/3kp+7ay6pLwXZCoqxd2To ENvg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id h8si8882218edn.92.2020.07.02.22.10.41; Thu, 02 Jul 2020 22:11:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726297AbgGCFIi (ORCPT + 99 others); Fri, 3 Jul 2020 01:08:38 -0400 Received: from out30-131.freemail.mail.aliyun.com ([115.124.30.131]:35872 "EHLO out30-131.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725764AbgGCFIi (ORCPT ); Fri, 3 Jul 2020 01:08:38 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R371e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04426;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0U1Y4LGX_1593752912; Received: from alexshi-test.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U1Y4LGX_1593752912) by smtp.aliyun-inc.com(127.0.0.1); Fri, 03 Jul 2020 13:08:33 +0800 From: Alex Shi To: akpm@linux-foundation.org, mgorman@techsingularity.net, tj@kernel.org, hughd@google.com, khlebnikov@yandex-team.ru, daniel.m.jordan@oracle.com, yang.shi@linux.alibaba.com, willy@infradead.org, hannes@cmpxchg.org, lkp@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, shakeelb@google.com, iamjoonsoo.kim@lge.com, richard.weiyang@gmail.com Subject: [PATCH v14 00/20] per memcg lru lock Date: Fri, 3 Jul 2020 13:07:33 +0800 Message-Id: <1593752873-4493-1-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This is a new version which bases on v5.8-rc3. Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain. Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold the lru_lock. The typic using for this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it. The above solution suggested by Johannes Weiner, and based on his new memcg charge path, we have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!). The patchset includes 3 parts: 1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's condition 3, replace per node lru_lock with per memcg per node lru_lock Following Daniel Jordan's suggestion, I have run 208 'dd' with on 104 containers on a 2s * 26cores * HT box with a modefied case: https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-lru-file-readtwice With this patchset, the readtwice performance increased about 80% in concurrent containers. Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox etc. Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks! Alex Shi (18): mm/vmscan: remove unnecessary lruvec adding mm/page_idle: no unlikely double check for idle page counting mm/compaction: correct the comments of compact_defer_shift mm/compaction: rename compact_deferred as compact_should_defer mm/thp: move lru_add_page_tail func to huge_memory.c mm/thp: clean up lru_add_page_tail mm/thp: narrow lru locking mm/memcg: add debug checking in lock_page_memcg mm/swap: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lru_lock holding in func lru_note_cost_page mm/lru: move lock into lru_note_cost mm/lru: introduce TestClearPageLRU mm/compaction: do page isolation first in compaction mm/mlock: reorder isolation sequence during munlock mm/swap: serialize memcg changes during pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock mm/lru: introduce the relock_page_lruvec function mm/pgdat: remove pgdat lru_lock Hugh Dickins (2): mm/vmscan: use relock for move_pages_to_lru mm/lru: revise the comments of lru_lock Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 ++- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +-- include/linux/compaction.h | 4 +- include/linux/memcontrol.h | 98 ++++++++++++ include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- include/trace/events/compaction.h | 2 +- mm/compaction.c | 113 +++++++++---- mm/filemap.c | 4 +- mm/huge_memory.c | 53 ++++-- mm/memcontrol.c | 71 ++++++++- mm/memory.c | 3 - mm/mlock.c | 93 +++++------ mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 8 - mm/rmap.c | 4 +- mm/swap.c | 177 +++++++-------------- mm/swap_state.c | 2 - mm/vmscan.c | 174 ++++++++++---------- mm/workingset.c | 2 - 25 files changed, 518 insertions(+), 365 deletions(-) -- 1.8.3.1