Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 2 Sep 2021 10:08:06 +0000
From:   Huang Shijie <shijie@os.amperecomputing.com>
To:     Linus Torvalds <torvalds@linux-foundation.org>
Cc:     Al Viro <viro@zeniv.linux.org.uk>,
        Shijie Huang <shijie@amperemail.onmicrosoft.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux-MM <linux-mm@kvack.org>,
        "Song Bao Hua (Barry Song)" <song.bao.hua@hisilicon.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Frank Wang <zwang@amperecomputing.com>
Subject: Re: Is it possible to implement the per-node page cache for
 programs/libraries?
Message-ID: <YTCihsPZL0HtO2lp@hsj>
References: <a2f423cf-9413-6bc8-e4d8-92374fc0449e@amperemail.onmicrosoft.com>
 <YS8HpUz9FUcFWt0V@zeniv-ca.linux.org.uk>
 <CAHk-=wiwAC8L7UsRsd-sHNS8ZETbrTB0EELvYV1qMOUo86R7kg@mail.gmail.com>
 <CAHk-=wjAPEs3HRGswJ-AE1R048j2MBsBtMfg3GOsaFykHoeKsg@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAHk-=wjAPEs3HRGswJ-AE1R048j2MBsBtMfg3GOsaFykHoeKsg@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?J79CUGleBRQHvTKdyw8T/WKONz/nohoCntvDcewvkQpbyNkFi3ivnVl66MRY?=
 =?us-ascii?Q?I4f8OTBfoJuE5BsW+k4IWaeDR/4decCRbNTdGqDfjK/QP5lPDZDvWgnqUEUi?=
 =?us-ascii?Q?jLmt4+DcdX4ygkRZ99GV8+y+HBJ4WlP5affoA8YauFLubdLkO4tIsO9wYH4S?=
 =?us-ascii?Q?3DaPwXStKKJz/XZlPFmB8W8zrIUdROB+WJr+60v7iXiYFI4pJTvpKTnbS7jG?=
 =?us-ascii?Q?2MrVFVHRc1+GvQMYYXpA+ezOlAA7DEmmpx+ov7pma4b/fQnAtNofQO1hkBJY?=
 =?us-ascii?Q?khrtUheJ+6jiSZPMgIzfG7ZsDzml7BYUISciUPlcUcL8sT9KoSSd7IkarEXp?=
 =?us-ascii?Q?27z1xLK0GB/evZhFmlpJoZnSISV/ppN474l3kEvhxUds1ez1eP08tNP4Lc+m?=
 =?us-ascii?Q?6WqVGIgZBPp6CviA6x6N1WyyTA2G69R6DoqaXBAc1d0N7/d2dsWUGAsS45f0?=
 =?us-ascii?Q?xviwWyygVqgrA7XEmcLbXi0hPLOH33YDtlvRID7U+Ytamt217qDpkrjnLLrA?=
 =?us-ascii?Q?4SP6UsVG0IfmoJJuECB9Thx1W6TuoVf/NDelhBpPIPU2Bk8LVe0psSamGa8Z?=
 =?us-ascii?Q?gmane+TkSaRh7eHrEvz/B5WSyPNMR0vUG2eHCtj9W32DLBfEHHcX4dNS2o9Z?=
 =?us-ascii?Q?nihzDJTbzwM8gYufhJgXRM1nYDy5lyMMbXFKH0ewNd1w7LPogyxGHgIBOoKy?=
 =?us-ascii?Q?AjDV3VjQ7zLPfMN+FX9Nq5R6HkA2jt5gBoeUilewSKdKsyQm0dxHhF19tdyL?=
 =?us-ascii?Q?y4lKGUo1NerdJXV7hFRkRsK68n3EgLGF6WsDnpvkIP/+YOtRyVXYiLPWiMDC?=
 =?us-ascii?Q?Hm1hf6f9pnYaMPQ3Si0tzaLqQ/kxINp0YMI7Ab4TfBp+invJJWAumLeT1YMx?=
 =?us-ascii?Q?FpTGC1WKwiKsJaHrJtZLBrUH0GW3PNGJpdDO7+u+m+BEPzcGiIZBzQmYBB9A?=
 =?us-ascii?Q?5SdGqt2M3vr+LiGmVYihZezA/YWUEOSfA5juF6qQ3kVi8iKDrlstBNK0d5uS?=
 =?us-ascii?Q?1yD8BVZ6ZOG6MMuXbJx58oC6f0tXJOXRpsGo4LQ7vsScI69j34KX/qYzPZ4T?=
 =?us-ascii?Q?vwPfs2X7at4Ge7RL4Phg84Z5XRIUBkegsk3dWFg+iXQDKYEwwfwzc9GhafvO?=
 =?us-ascii?Q?Wb6xUtvJb6TgECD76kK5QNXX+JFlnd60zxkj+ICFk375JSp7nT7iWe6MmEK1?=
 =?us-ascii?Q?KjVHTphS0NyMQEBFtf9mBMSAOIbk9ozo+7oWLwbFtHp3xCi4YX2jx5FGZhVz?=
 =?us-ascii?Q?gORmRjcnA6eiU5c0J2iWGHl5Gex+IIOP5VsOLHH1T6F0zlC/Fzo+3opy6jsn?=
 =?us-ascii?Q?11W99rQQ6nThzLf6HicAtuMb?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9c07b6aa-2003-40c4-09f3-08d96db6b870
X-MS-Exchange-CrossTenant-AuthSource: MWHPR0101MB3165.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2021 02:09:39.7210
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2WTNBDnjHCCsvXzCOtzje06kZ0x4Tcsh1fIWZixvA6ktrJguI4yyY5tcV5TjEeP5actrOmXstd/0O43j7sr+xiY/XK5hSi3fmxa9XlnX2Tz8TlbIBLIrk7KYw0/BjLlZ
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR0102MB3529
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Linus,
On Wed, Sep 01, 2021 at 10:29:01AM -0700, Linus Torvalds wrote:
> On Wed, Sep 1, 2021 at 10:24 AM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > But what you could do, if  you wanted to, would be to catch the
> > situation where you have lots of expensive NUMA accesses either using
> > our VM infrastructure or performance counters, and when the mapping is
> > a MAP_PRIVATE you just do a COW fault on them.
> >
> > Sounds entirely doable, and has absolutely nothing to do with the page
> > cache. It would literally just be an "over-eager COW fault triggered
> > by NUMA access counters".
Yes. You are right, we can use COW. :)

Actually we have _TWO_ levels to do the optimization for NUMA remote-access:
   1.) the page cache which is independent to process.
   2.) the process address space(page table).

   For 2.), we can use the over-eager COW:
        2.1) I have finished a user patch for glibc which uses "over-eager COW" to do the text
	   replication in NUMA.
        2.2) Also a kernel patch uses the "over-eager COW" to do the replication for 
           the programs itself in NUMA. (We may refine it to another topic..)
> 
> Note how it would work perfectly fine for anonymous mappings too. Just
> to reinforce the point that this has nothing to do with any page cache
> issues.
> 
> Of course, if you want to actually then *share* pages within a node
> (rather than replicate them for each process), that gets more
> exciting.
Do we really need to change the page cache?
          The 2.1) above may produces one-copy "shared libraries pages" for each process, such glibc.so.
          Even in the same NUMA node 0, we may run two same processes. So it produces "two glibc.so" now.
	  If We run 5 same processes in NUMA Node 0, it will produces "five glibs.so".

	  But if we have per-node page cache for the glibc.so, we can do it like this:
	  (1) disable the "over-eager COW" in the process.
	  (2) use the per-node page cache's pages to different processes in the _SAME_ NUMA node.
	      So all the processes in the same NUMA node, can use only one same page.
          (3) Processes in other NUMA nodes, use the pages belong to this node.

	  By this way, we can save many pages, and provide more access speed in NUMA.

Thanks
Huang Shijie