Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sat, 23 Apr 2022 11:22:21 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Yang Shi <shy828301@gmail.com>
CC:     "ying.huang@intel.com" <ying.huang@intel.com>,
        Michal Hocko <mhocko@suse.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux MM <linux-mm@kvack.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] mm: swap: determine swap device by using page nid
Message-ID: <YmNw7Q/9/xNuQsPK@ziqianlu-desk1>
References: <20220407020953.475626-1-shy828301@gmail.com>
 <Yk6cutNf5sOuYbDl@ziqianlu-nuc9qn>
 <CAHbLzkq+eKcKCsxXDTiOcBxk8FjMdWBqOxwi4N_NG7PZWbAAkA@mail.gmail.com>
 <Yl/FS9enAD4V8jG3@ziqianlu-nuc9qn>
 <f27ec36beb3cf1dbbfc3b8835e586d5d6fe7f561.camel@intel.com>
 <YmETEHnK/FiNbLO7@ziqianlu-desk1>
 <6f7210be7353d1c01dc9f872b2692b83f87f5452.camel@intel.com>
 <YmJKBaq1yj6/iBJ3@ziqianlu-desk1>
 <CAHbLzkriO6xWzyMNpcVFmyxSn=cqbz2qx+2mJ5d0m-beqPRCUg@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <CAHbLzkriO6xWzyMNpcVFmyxSn=cqbz2qx+2mJ5d0m-beqPRCUg@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BoHlSstnVKYOLYLqVODy5vHkYp/wNtjALOs7HmV840YZn4C3JfBC63+0V7ye?=
 =?us-ascii?Q?eLOrJkDgcC8VRMCvLFXEEVnhZD60WX9DB4AVG9lyzbX91oXsWP2iLSi1eJsE?=
 =?us-ascii?Q?mP9pSI3Vgg47L1WHJpaR2jvx4qBfCUWmL/xI0mhd3qysA6J/dYf+v26G9KOP?=
 =?us-ascii?Q?o2oLz8HEn1bLb5L+OmickpDku35mbGn3XhdYq0j4jZL4kObwkqHWvDP/SNIQ?=
 =?us-ascii?Q?h3QbaqEIJBiXRBRcdCqrAwmUqIVzsw6wrGKSmPXnZqueTrbIoj6bfS6EPslC?=
 =?us-ascii?Q?C9wp/5B8gtqJglbkbM/H3BZ5hgCksUPQSoeYvwvky4M5zpYxghDpjVUkY79S?=
 =?us-ascii?Q?o1xBu5jcENWhAuGhg1uWMZBp80176WriJGmI66P+ZAiwaPAJyM9moE7GrBu5?=
 =?us-ascii?Q?rPUmO12VftIUCbyjdbc2WCz8+1ZBiet8TDhOh+zD9hurpwQUJshhHkozjpOC?=
 =?us-ascii?Q?sRnk6Hlrsoo9+ev20Eh9kmCG+mgqgud84y+EegvJ9r2S9aOvpTxw9OTwocje?=
 =?us-ascii?Q?Qh4P2cE4qpBqz1A0K9Pmi/LNcsJRZzn9N7Z+j5eKRXoNeX8ax/9gSFrqaCox?=
 =?us-ascii?Q?rhXI9sYi5ZXO6TGyEwvsTtXB8oO8baT4HIIajY8vFsbbmqLxpj351oyDqlSI?=
 =?us-ascii?Q?ehdUFZ8jZ/LUxWAAncSUPFtqbKR6jQ82FlEKp6wyEbZTmFYwiaV3NULMY+By?=
 =?us-ascii?Q?yZKCy2ihcVLcpzZrjvH08Izbm11GHAUsMTkfR9EEDnBgGngvSF54OqkbVAWP?=
 =?us-ascii?Q?V5/NLFbT07EQgl/R58zQfAcxWIjFuNg/uNXrfFHlbSOMRWwR5ysKuGJqWadd?=
 =?us-ascii?Q?YqSSZf6/CbKhWYABvXlKaPvda4kjs0UWvzIADdsgnkaeefcQuPb6WcmKh+lx?=
 =?us-ascii?Q?IUp9feZYx5fKbcZHaEt4jzqfhmwVvBWvPbir73bujPjPys27nmTzIN+fxHDt?=
 =?us-ascii?Q?AeNAzUNrNmij/1uPRUzhNwNoRVHlqxsl6sRRL3JItgW2E1FKVelukQ9pnaBD?=
 =?us-ascii?Q?3j68Ml6DFTHVykaFK77SHvgjDVzuP0vXlY4qWEemQRcnwpCM6AHW+ummsvlr?=
 =?us-ascii?Q?an3APoeFBkYcEDxS7WX4fB7bpz2/XutkqGkqSEiKIRTgAM49JfBO3qGbFm6J?=
 =?us-ascii?Q?i7xHFZEZtPj65IKpbq+sHJsA+slJAZHERx84M2cAr7xoJSi5BFgxkPOiF00x?=
 =?us-ascii?Q?qaqNpSA4/2BfyQIJ8uf8jRb9Kln3pxOOvPRYCU/YtJx1CKNS9+NWESiELuqy?=
 =?us-ascii?Q?PzeCQ3h79zgRCCT83Vz7knQ9qBD4FNYJ5Tg/4/AYmV06oGLgZ0msJEceLUTQ?=
 =?us-ascii?Q?YKkjE+d77t3tXzRC8GtKxTCL8Bv0z87FWFau2R4Mk4qu9T4QEiWL9VNOvSUm?=
 =?us-ascii?Q?FhiEFh63/moDqEXkNyYFkZdZyIbZy+sUbXr5hK7/0mR2aBJ7w+TKnkJHeieh?=
 =?us-ascii?Q?ZOGJBQNmZ/FydNog0hGoDb0Fi5RTHrfJKzsElCsBzfMgQKJNx2xE+5v2bf/r?=
 =?us-ascii?Q?6gEaMxZDk3xs9gV4brojC3H00hlGndD/GYnHGwVr4CTxKYDlPro7mFtahjMk?=
 =?us-ascii?Q?WAZlV9jTiHOET6wsKxa3qoXNuAv6c0Qce2xdjMq3OrvqRz+zwQR3V36i/CZL?=
 =?us-ascii?Q?duxeYmtb2XhcjM+grtbNzBL/1wCid4f9gfH2eVJhC6styvfDKAVxUXdR8rjd?=
 =?us-ascii?Q?62HbD5OTDQGvhAfK1QaTREDS0/aPhqnGVfpU7erywrA4vuxBOyuWKk5GPHfp?=
 =?us-ascii?Q?kKOOdNOuPA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 57e9844f-d8b1-4e3b-2113-08da24d87efd
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Apr 2022 03:22:29.0744
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4D66tD3Y5Da9BNya6tC3ICsobexoGAAz2ae5hClHOl2odEM4iKKWYjXxyzZAZX5HPFnmGdZbZAIfP5pro8LK/Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2564
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Apr 22, 2022 at 10:00:59AM -0700, Yang Shi wrote:
> On Thu, Apr 21, 2022 at 11:24 PM Aaron Lu <aaron.lu@intel.com> wrote:
> >
> > On Thu, Apr 21, 2022 at 04:34:09PM +0800, ying.huang@intel.com wrote:
> > > On Thu, 2022-04-21 at 16:17 +0800, Aaron Lu wrote:
> > > > On Thu, Apr 21, 2022 at 03:49:21PM +0800, ying.huang@intel.com wrote:
> >
> > ... ...
> >
> > > > > For swap-in latency, we can use pmbench, which can output latency
> > > > > information.
> > > > >
> > > >
> > > > OK, I'll give pmbench a run, thanks for the suggestion.
> > >
> > > Better to construct a senario with more swapin than swapout.  For
> > > example, start a memory eater, then kill it later.
> >
> > What about vm-scalability/case-swapin?
> > https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swapin
> >
> > I think you are pretty familiar with it but still:
> > 1) it starts $nr_task processes and each mmaps $size/$nr_task area and
> >    then consumes the memory, after this, it waits for a signal;
> > 2) start another process to consume $size memory to push the memory in
> >    step 1) to swap device;
> > 3) kick processes in step 1) to start accessing their memory, thus
> >    trigger swapins. The metric of this testcase is the swapin throughput.
> >
> > I plan to restrict the cgroup's limit to $size.
> >
> > Considering there is only one NVMe drive attached to node 0, I will run
> > the test as described before:
> > 1) bind processes to run on node 0, allocate on node 1 to test the
> >    performance when reclaimer's node id is the same as swap device's.
> > 2) bind processes to run on node 1, allocate on node 0 to test the
> >    performance when page's node id is the same as swap device's.
> >
> > Ying and Yang,
> >
> > Let me know what you think about the case used and the way the test is
> > conducted.
> 
> Looks fine to me. To measure the latency, you could also try the below
> bpftrace script:

Yeah, bpftrace can nicely show us the histogram of the latency.
The hard part is to integrate bpftrace into LKP framework though.

> 
> #! /usr/bin/bpftrace
> 
> kprobe:swap_readpage
> {
>         @start[tid] = nsecs;
> }
> 
> kretprobe:swap_readpage
> /@start[tid]/
> {
>         @us[comm] = hist((nsecs - @start[tid]) / 1000);
>         delete(@start[tid]);
> }