Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 29 Apr 2022 18:26:49 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Yang Shi <shy828301@gmail.com>
CC:     "ying.huang@intel.com" <ying.huang@intel.com>,
        Michal Hocko <mhocko@suse.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Linux MM <linux-mm@kvack.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Tim Chen <tim.c.chen@linux.intel.com>
Subject: Re: [PATCH] mm: swap: determine swap device by using page nid
Message-ID: <Ymu9acl18pTA5GU6@ziqianlu-desk1>
References: <20220407020953.475626-1-shy828301@gmail.com>
 <Yk6cutNf5sOuYbDl@ziqianlu-nuc9qn>
 <CAHbLzkq+eKcKCsxXDTiOcBxk8FjMdWBqOxwi4N_NG7PZWbAAkA@mail.gmail.com>
 <Yl/FS9enAD4V8jG3@ziqianlu-nuc9qn>
 <f27ec36beb3cf1dbbfc3b8835e586d5d6fe7f561.camel@intel.com>
 <YmETEHnK/FiNbLO7@ziqianlu-desk1>
 <6f7210be7353d1c01dc9f872b2692b83f87f5452.camel@intel.com>
 <YmJKBaq1yj6/iBJ3@ziqianlu-desk1>
 <CAHbLzkriO6xWzyMNpcVFmyxSn=cqbz2qx+2mJ5d0m-beqPRCUg@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <CAHbLzkriO6xWzyMNpcVFmyxSn=cqbz2qx+2mJ5d0m-beqPRCUg@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6zCgZS77Z7ljIdsC/ZT8jVgkqnbq2/onqbGZ57mnQkImoDIwpujAhdu8EwiG?=
 =?us-ascii?Q?s9cZkJP2EX5o9PD+FZ3pNkkSdlfHA7aM1Je5B2pZwXYb+92lKju3c2Nb6+YZ?=
 =?us-ascii?Q?l/qE5Eyqh7aXGSn7mOFQ+A7mVl7sKXh/1qR6oobyidZzfjnhKeS8EPgp2vVU?=
 =?us-ascii?Q?6QgJczeOPwDxqffsECfKJldCZVUxVcpaQbKTakVSfULH9xPIr8FBOXH9qt3Y?=
 =?us-ascii?Q?I48MiXInRVDrFJJDG5xOkY5tvMn3udAye5CPInJCwkVf8m4EnRmd1SySfmoX?=
 =?us-ascii?Q?Rspt2uifM7+rgydkz7AmoUpy9+GEruMoyL/YmFos1mGkhiAWdy/Qb5uHSy2j?=
 =?us-ascii?Q?nJY3oPxZTlHfTFKUSHbo5V6JpSQOksonfnDpLd1vFpk8T0lmvcPHoa25/maT?=
 =?us-ascii?Q?ZrR88G+PGKFXVWw6/AVi/I/Ow3Z2XFUy8VCbBKR4ONwqgvIxLfrREZmCb6ku?=
 =?us-ascii?Q?EnCD5luxcEEk2P4U9g9WAk+A8q89WuAd+Wap186Jvids1IQ+623sGc+UKDMK?=
 =?us-ascii?Q?62+cAt7nsk0aMKdFK8bz/ikVu3Efz8aeZ96ArlUSbZ3pb9ii4swNoX9HMOcn?=
 =?us-ascii?Q?ShpaHOzhlpw/EG8qhMffd82BpAXe3D1aANwzzfrzb5nx/8F8xFnJwq9wzPvD?=
 =?us-ascii?Q?86GNPCK/j3ZjyWYD7cq0HEvd+7tzmBlOe1wdGXIBS9fJpodTeyOpW31Yr3h8?=
 =?us-ascii?Q?pXKoUksinYGtle2YR2hckTWIxkW4O9mMivhwFxC4myGpD93BavF8WF3SVrJp?=
 =?us-ascii?Q?9Ff/iDbH0HA9sjs0vyBpaF/gFa17kY5TXTUHcKs/ovMzSknos7EiszhxkQvH?=
 =?us-ascii?Q?xbDNpMGWRcGKHl+VmrKkJltHjJQ0jiQy2O+uJVIc0aGOXQ33U3f9ojz6hgZi?=
 =?us-ascii?Q?LQPzA+u0hd7eelzm9mCVBHHFkvUo06N4IyUkTaOnIeS8mAyZTYYowUzBKFxd?=
 =?us-ascii?Q?0Rw8rkIc+HJ1TVsz62L9+wlLXdfisVb8z0VWFoBZ1FGPtPYDRy+Xi4yDWG+V?=
 =?us-ascii?Q?rUrlkteqUHUmaUbAXJKGiOmCx13/NcZDfcVJm0tKat2yT/52kCBAnBIIjOPe?=
 =?us-ascii?Q?Y5IeWuSWsgYhY3ho1v1N448tBu6kFAzyxz0LzlOkJxDoSfw9nT4yuN6AdAmY?=
 =?us-ascii?Q?uHwH1l///TePI+JyoVNHd4nlGqDUMmsx+E59pivQnv1t2fiUzU8bGPcJouTT?=
 =?us-ascii?Q?ZFhDuDI0LOf/7d7Q+FlZ+uEFe+4XwnHjAnKT72r+pcCeD90WFqMvFf8rPpfX?=
 =?us-ascii?Q?onxviyilbDBbSIa8eaEORna9lgJEbEV4Nril5B2IsDXnk6o3+l9pfUWdeeRZ?=
 =?us-ascii?Q?LklTeyUeQ5Gspa9d6GpJpdOce5w7Qnb5vrqP9Hv4ZMFiiy/HOibETkiBrbZl?=
 =?us-ascii?Q?QNhAa6+JjTdB7ABFkRiuddWfW1qt/nOTdtdRfFHKPJ5UYq+4q3phL8pwTfNB?=
 =?us-ascii?Q?Nx5ZBwpB9SCtPkXUVIbhprsbO3HjpayYS8tSst+PJD+f10UxSfqrj1xJ1wND?=
 =?us-ascii?Q?dF+S5tOYEDbozNm1tTHmhAB/7VeKHn+8IpRCWdSIjsv15iDYFfAU+cURT7UG?=
 =?us-ascii?Q?P51ZPtCsEOOOLmTQ1FPK65shlX85PpwYKvQQ6kLisa/O5esGdv5f6Zktg3BP?=
 =?us-ascii?Q?3Ag0Mv3TE6XnWE2q1pl2tOBo+BmrMv9zBBa2ao3BSL+546Z9ImEMIatC+7bb?=
 =?us-ascii?Q?eQR0xXAFk61mZ1vZKLQdJ683F8a6kQnvc/CkXanQz3tN2rEnwvvvCjvuEQfx?=
 =?us-ascii?Q?rR1OmpMNaA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6c8f75bc-5172-49a5-fcd7-08da29cacb00
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2022 10:26:59.5287
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yRcIfQgbi+QhAVf3zkNrZQj9ygXE6XMMlMoDMp0zc+B6YI4GY4BX5cQxCd8UMG6yTaknkiu6t9Qa5zCIcJK8KA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5485
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Apr 22, 2022 at 10:00:59AM -0700, Yang Shi wrote:
> On Thu, Apr 21, 2022 at 11:24 PM Aaron Lu <aaron.lu@intel.com> wrote:
> >
> > On Thu, Apr 21, 2022 at 04:34:09PM +0800, ying.huang@intel.com wrote:
> > > On Thu, 2022-04-21 at 16:17 +0800, Aaron Lu wrote:
> > > > On Thu, Apr 21, 2022 at 03:49:21PM +0800, ying.huang@intel.com wrote:
> >
> > ... ...
> >
> > > > > For swap-in latency, we can use pmbench, which can output latency
> > > > > information.
> > > > >
> > > >
> > > > OK, I'll give pmbench a run, thanks for the suggestion.
> > >
> > > Better to construct a senario with more swapin than swapout.  For
> > > example, start a memory eater, then kill it later.
> >
> > What about vm-scalability/case-swapin?
> > https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swapin
> >
> > I think you are pretty familiar with it but still:
> > 1) it starts $nr_task processes and each mmaps $size/$nr_task area and
> >    then consumes the memory, after this, it waits for a signal;
> > 2) start another process to consume $size memory to push the memory in
> >    step 1) to swap device;
> > 3) kick processes in step 1) to start accessing their memory, thus
> >    trigger swapins. The metric of this testcase is the swapin throughput.
> >
> > I plan to restrict the cgroup's limit to $size.
> >
> > Considering there is only one NVMe drive attached to node 0, I will run
> > the test as described before:
> > 1) bind processes to run on node 0, allocate on node 1 to test the
> >    performance when reclaimer's node id is the same as swap device's.
> > 2) bind processes to run on node 1, allocate on node 0 to test the
> >    performance when page's node id is the same as swap device's.
> >

Thanks to Tim, he has found me a server that has a single Optane disk
attached to node 0.

Let's use task0_mem0 to denote tasks bound to node 0 and memory bound
to node 0 through cgroup cpuset. And for the above swapin case:
when nr_task=1:
task0_mem0 throughput: [571652, 587158, 594316], avg=584375 -> baseline
task0_mem1 throughput: [582944, 583752, 589026], avg=585240    +0.15%
task1_mem0 throughput: [569349, 577459, 581107], avg=575971    -1.4%
task1_mem1 throughput: [564482, 570664, 571466], avg=568870    -2.6%

task0_mem1 is slightly better than task1_mem0.

For nr_task=8 or nr_task=16, I also gave it a run and the result is
almost the same for all 4 cases.

> > Ying and Yang,
> >
> > Let me know what you think about the case used and the way the test is
> > conducted.
> 
> Looks fine to me. To measure the latency, you could also try the below
> bpftrace script:
>

Trying to install bpftrace on an old distro(Ubuntu 16.04) is a real
pain, I gave up... But I managed to get an old bcc installed. Using
the provided funclatency script to profile 30 seconds swap_readpage(),
there is no obvious difference from the histrogram.

So for now, from the existing results, it did't show big difference.
Theoretically, for IO device, when swapping a remote page, using the
remote swap device that is at the same node as the page can reduce the
traffic of the interlink and improve performance. I think this is the
main motivation for this code change?
On swapin time, it's hard to say which node the task will run on anyway
so it's hard to say where to swap is beneficial.