Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Thu, 21 Apr 2022 22:11:59 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     "ying.huang@intel.com" <ying.huang@intel.com>
CC:     Yang Shi <shy828301@gmail.com>, Michal Hocko <mhocko@suse.com>,
        "Andrew Morton" <akpm@linux-foundation.org>,
        Linux MM <linux-mm@kvack.org>,
        "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] mm: swap: determine swap device by using page nid
Message-ID: <YmFmL42W+OrORElV@ziqianlu-nuc9qn>
References: <20220407020953.475626-1-shy828301@gmail.com>
 <Yk6cutNf5sOuYbDl@ziqianlu-nuc9qn>
 <CAHbLzkq+eKcKCsxXDTiOcBxk8FjMdWBqOxwi4N_NG7PZWbAAkA@mail.gmail.com>
 <Yl/FS9enAD4V8jG3@ziqianlu-nuc9qn>
 <f27ec36beb3cf1dbbfc3b8835e586d5d6fe7f561.camel@intel.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <f27ec36beb3cf1dbbfc3b8835e586d5d6fe7f561.camel@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?bsZdlq1SJmWM2g9s6m0u6lnNHJtJILpmxE8snaNYnnEXWdQF+8gQp3Zmfr?=
 =?iso-8859-1?Q?UsFUruz2Xp0J7u5RmO4MffRuzDjJm/o4TPuZaxXI8KKynDeiGYEHu2m9b8?=
 =?iso-8859-1?Q?TxZxOU0uHJTKn9uL9nYIbvXIXgL09gJAE9IEQQxSqAEiOgABgCl5sweFZF?=
 =?iso-8859-1?Q?o7tAayQlkV/+t4Ydphp3M+sYYc6W3RNgpuWX4fc3INSQWGovdAZ/hWUHIY?=
 =?iso-8859-1?Q?xGnjO1uaSAg3KpoBTiqY8PLMU2mLJALRXih5VUebdZHK2dCTRb6ZkgmUL2?=
 =?iso-8859-1?Q?3pXi9wTKm1RZxxm7APt9Im6D1HwQlo7R6oocsHF6XULHJuB5WF8UgxeQli?=
 =?iso-8859-1?Q?c2COjwXD7jMwHWp79qWBXMvF+a1qhQKWoufGR+am7PvKlYtDJ6EuW9aEFA?=
 =?iso-8859-1?Q?fw9HfWi7ooOOy7aEsjuyRARf+rBxlu1skA91ST2lIPW9sH6tkTUNXNdqVw?=
 =?iso-8859-1?Q?WR/FlGYB7ah/fxCA2IzXKclq1dnUj2nLp/SUeOFuxsC0T6EBU+J3fkpzpF?=
 =?iso-8859-1?Q?uq+pg8oG+Y228MUUuZvvEQQPVHqWo+hPtUL3oXPKI5NBC8yBmr52eWZGhL?=
 =?iso-8859-1?Q?gZd1zQPj9TzavvC7A1Um3Lv183J+4sicIdmUdc5DS+1IQTLXpDxFDljz12?=
 =?iso-8859-1?Q?N8MdI4Qp+15y3kfciupBINI26sDqANuOFTTa6z9bPIIoYanYOfgxZb/VHM?=
 =?iso-8859-1?Q?GAXv0GcBQJ+NrDWNBJGm1PCbhric+AIn2r8zNj+SlRMEntP97q98ocRxbb?=
 =?iso-8859-1?Q?9FgFI7qYjd2pzfH8EoXswGt8/UsnUYEHHryafEA9dMQOF/uoUv9SsOIZ77?=
 =?iso-8859-1?Q?tHUNAep+4PG/1CjZGZtKVLk7dSkykbn5VJ48bVFf8850bO1r1819TJpKrh?=
 =?iso-8859-1?Q?wI7rMH0AOovsAxmrIEs8pNbHjes8yAjvqUTsF3qY31Pp0RFsyOTik/wdY9?=
 =?iso-8859-1?Q?x8Cirj6qtMfbbPrznZFtlBMNSf6bjKMAICm1eA5e0u4h5SW0fu8c/LqXog?=
 =?iso-8859-1?Q?Jlwsd7jFWht3ajklvMHZpC4rFTQ4kvzzEa6eaSRt3VKScrzcGDfvAvGtU0?=
 =?iso-8859-1?Q?6Mz8/ZiiSpkq+cINjPJNdi5CO1q7T9KuWjrgv6qODY3ga0wbeQbY2xBf5u?=
 =?iso-8859-1?Q?A2sZN5nqMDeHYvApSWYuDB0w+H3RPhtR8tugSAbXOqxUWqHxjk/P5oxM9f?=
 =?iso-8859-1?Q?SngurioMBwSSV7JTq0S+JpWjpAgeeh+c5bB3DbgUydTSMO0+J6R2d0wfEc?=
 =?iso-8859-1?Q?Mmsfwhx7mM1i9zu7odPnsrp8gdb0gBN8UHIWhyYGndMBNA6kggT5p0fFYI?=
 =?iso-8859-1?Q?6FdJEO3l/9Bz2Zqn6wKaKpslhhkgTzTr699q3Z5kAhFtZoDOXBiHWWNxw5?=
 =?iso-8859-1?Q?P2Z77c3x4lBsSGaoEbCjpPfTg77N/k2txrVKrPMg8XrRcYsl7opOquQ5JG?=
 =?iso-8859-1?Q?+tU2xgTsqys2p/dOChcCdwKpEqYH5NzfXLdWYKezi9EWgOMXrjd8voNzfs?=
 =?iso-8859-1?Q?l1UmVMgPhscYjeIsewQMhYhs3mNHFe0Hhv5A1CfRsdjnGhVoFgoVAtoggS?=
 =?iso-8859-1?Q?P5EjGPXFeLKZnkfQAjPpoG+Q9kZDU90oYZbm8JftPhQWp98sOcq6fXsJtI?=
 =?iso-8859-1?Q?Y3CXZ7cypTskLkooK8EqMc16aQd8BfGW/hzJw0EwVdbObMEI6CoA78ggz9?=
 =?iso-8859-1?Q?zvuVfRLwlnCBC1andpIDGrGnkRRpQa3SiGkOdxz3S4WHOwonRPkJTlmjSi?=
 =?iso-8859-1?Q?1n7oCqwY355ap6OKwP/ud9OHD3arM6j+NGrYWg9Yh9zTsf0tHBG6Q3RC1T?=
 =?iso-8859-1?Q?aiqdnWzeCw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4e7dee00-2338-4eaa-f621-08da23a0e22b
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Apr 2022 14:11:53.5429
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MZy3xn/BLWTxFQuLXqHlsXPmGl5cocgL0in6A90qbTFlRd2nlxyRvp004isl9ERiuE/P89ss5sv2oAENKsnYZA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2744
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Apr 21, 2022 at 03:49:21PM +0800, ying.huang@intel.com wrote:
> On Wed, 2022-04-20 at 16:33 +0800, Aaron Lu wrote:
> > On Thu, Apr 07, 2022 at 10:36:54AM -0700, Yang Shi wrote:
> > > On Thu, Apr 7, 2022 at 1:12 AM Aaron Lu <aaron.lu@intel.com> wrote:
> > > > 
> > > > On Wed, Apr 06, 2022 at 07:09:53PM -0700, Yang Shi wrote:
> > > > > The swap devices are linked to per node priority lists, the swap device
> > > > > closer to the node has higher priority on that node's priority list.
> > > > > This is supposed to improve I/O latency, particularly for some fast
> > > > > devices.  But the current code gets nid by calling numa_node_id() which
> > > > > actually returns the nid that the reclaimer is running on instead of the
> > > > > nid that the page belongs to.
> > > > > 
> > > > 
> > > > Right.
> > > > 
> > > > > Pass the page's nid dow to get_swap_pages() in order to pick up the
> > > > > right swap device.  But it doesn't work for the swap slots cache which
> > > > > is per cpu.  We could skip swap slots cache if the current node is not
> > > > > the page's node, but it may be overkilling. So keep using the current
> > > > > node's swap slots cache.  The issue was found by visual code inspection
> > > > > so it is not sure how much improvement could be achieved due to lack of
> > > > > suitable testing device.  But anyway the current code does violate the
> > > > > design.
> > > > > 
> > > > 
> > > > I intentionally used the reclaimer's nid because I think when swapping
> > > > out to a device, it is faster when the device is on the same node as
> > > > the cpu.
> > > 
> > > OK, the offline discussion with Huang Ying showed the design was to
> > > have page's nid in order to achieve better I/O performance (more
> > > noticeable on faster devices) since the reclaimer may be running on a
> > > different node from the reclaimed page.
> > > 
> > > > 
> > > > Anyway, I think I can make a test case where the workload allocates all
> > > > its memory on the remote node and its workingset memory is larger then
> > > > the available memory so swap is triggered, then we can see which way
> > > > achieves better performance. Sounds reasonable to you?
> > > 
> > > Yeah, definitely, thank you so much. I don't have a fast enough device
> > > by hand to show the difference right now. If you could get some data
> > > it would be perfect.
> > > 
> > 
> > Failed to find a test box that has two NVMe disks attached to different
> > nodes and since Shanghai is locked down right now, we couldn't install
> > another NVMe on the box so I figured it might be OK to test on a box that
> > has a single NVMe attached to node 0 like this:
> > 
> > 1) restrict the test processes to run on node 0 and allocate on node 1;
> > 2) restrict the test processes to run on node 1 and allocate on node 0.
> > 
> > In case 1), the reclaimer's node id is the same as the swap device's so
> > it's the same as current behaviour and in case 2), the page's node id is
> > the same as the swap device's so it's what your patch proposed.
> > 
> > The test I used is vm-scalability/case-swap-w-rand:
> > https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swap-w-seq
> > which spawns $nr_task processes and each will mmap $size and then
> > randomly write to that area. I set nr_task=32 and $size=4G, so a total
> > of 128G memory will be needed and I used memory.limit_in_bytes to
> > restrict the available memory to 64G, to make sure swap is triggered.
> > 
> > The reason why cgroup is used is to avoid waking up the per-node kswapd
> > which can trigger swapping with reclaimer/page/swap device all having the
> > same node id.
> > 
> > And I don't see a measuable difference from the result:
> > case1(using reclaimer's node id) vm-scalability.throughput: 10574 KB/s
> > case2(using page's node id)      vm-scalability.throughput: 10567 KB/s
> > 
> > My interpretation of the result is, when reclaiming a remote page, it
> > doesn't matter much which swap device to use if the swap device is a IO
> > device.
> > 
> > Later Ying reminded me we have test box that has optane installed on
> > different nodes so I also tested there: Icelake 2 sockets server with 2
> > optane installed on each node. I did the test there like this:
> > 1) restrict the test processes to run on node 0 and allocate on node 1
> > ???and only swapon pmem0, which is the optane backed swap device on node 0;
> > 2) restrict the test processes to run on node 0 and allocate on node 1
> > ???and only swapon pmem1, which is the optane backed swap device on node 1.
> > 
> > So case 1) is current behaviour and case 2) is what your patch proposed.
> > 
> > With the same test and the same nr_task/size, the result is:
> > case1(using reclaimer's node id) vm-scalability.throughput: 71033 KB/s
> > case2(using page's node id)      vm-scalability.throughput: 58753 KB/s
> > 
> 
> The per-node swap device support is more about swap-in latency than
> swap-out throughput.  I suspect the test case is more about swap-out
> throughput.  perf profiling can show this.
>

On another thought, swap out can very well affect swap in latency:
since swap is involved, the available memory is in short supply and swap
in may very likely need to reclaim a page and that reclaim can involve a
swap out, so swap out performance can also affect swap in latency.

> For swap-in latency, we can use pmbench, which can output latency
> information.
> 
> Best Regards,
> Huang, Ying
> 
> 
> [snip]
>