From: Zi Yan <zi.yan@cs.rutgers.edu>
To: Balbir Singh <bsingharora@gmail.com>
CC: <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
        <akpm@linux-foundation.org>, <minchan@kernel.org>, <vbabka@suse.cz>,
        <mgorman@techsingularity.net>, <kirill.shutemov@linux.intel.com>,
        <n-horiguchi@ah.jp.nec.com>, <khandual@linux.vnet.ibm.com>
Subject: Re: [PATCH 0/5] Parallel hugepage migration optimization
Date: Mon, 28 Nov 2016 10:22:45 -0500
Message-ID: <B5954713-0912-40E6-898E-BF4A05B7E5CB@cs.rutgers.edu>
In-Reply-To: <9cf7f4c6-6dde-9dbb-cf93-7874437a442d@gmail.com>
References: <20161122162530.2370-1-zi.yan@sent.com>
 <9cf7f4c6-6dde-9dbb-cf93-7874437a442d@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed;
        boundary="=_MailMate_637746C7-C397-4CC8-B358-3C32989FF3D2_=";
        micalg=pgp-sha512; protocol="application/pgp-signature"
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2016 15:22:50.1128 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR14MB1658
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2658
Lines: 73

--=_MailMate_637746C7-C397-4CC8-B358-3C32989FF3D2_=
Content-Type: text/plain

On 24 Nov 2016, at 18:59, Balbir Singh wrote:

> On 23/11/16 03:25, Zi Yan wrote:
>> From: Zi Yan <zi.yan@cs.rutgers.edu>
>>
>> Hi all,
>>
>> This patchset boosts the hugepage migration throughput and helps THP migration
>> which is added by Naoya's patches: https://lwn.net/Articles/705879/.
>>
>> Motivation
>> ===============================
>>
>> In x86, 4KB page migrations are underutilizing the memory bandwidth compared
>> to 2MB THP migrations. I did some page migration benchmarking on a two-socket
>> Intel Xeon E5-2640v3 box, which has 23.4GB/s bandwidth, and discover
>> there are big throughput gap, ~3x, between 4KB and 2MB page migrations.
>>
>> Here are the throughput numbers for different page sizes and page numbers:
>>         | 512 4KB pages | 1 2MB THP  |  1 4KB page
>> x86_64  |  0.98GB/s     |  2.97GB/s  |   0.06GB/s
>>
>> As Linux currently use single-threaded page migration, the throughput is still
>> much lower than the hardware bandwidth, 2.97GB/s vs 23.4GB/s. So I parallelize
>> the copy_page() part of THP migration with workqueue and achieve 2.8x throughput.
>>
>> Here are the throughput numbers of 2MB page migration:
>>            |  single-threaded   | 8-thread
>> x86_64 2MB |    2.97GB/s        | 8.58GB/s
>>
>
> Whats the impact on CPU utilization? Is there a huge impact?
>
> Balbir Singh.

It depends on the throughput we can achieve.

For single-threaded copy, the current routine, it takes one CPU 2MB/(2.97GB/s) = 657.6 us
to copy one 2MB page.

For 8-thread copy, it take 8 CPUs 2MB/(8.58GB/s) = 227.6 us to copy one 2MB page.

If we have 8 idle CPUs, I think it worths using them.

I am going to add code to check idle_cpu() in the system before doing the copy.
If no idle CPUs are present, I can fall back to single-threaded copy.


--
Best Regards
Yan Zi

--=_MailMate_637746C7-C397-4CC8-B358-3C32989FF3D2_=
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJYPEvFAAoJEEGLLxGcTqbMEb8H/2rCChgp11A8SwJhTkt0BFPO
mgusZsNgrSK7Wi2q+j2ZOkoLidSD+1Mh5N8wFrqVs9LCFiKvfQXIlllrVpze9yIU
nTd0c3BQjS7eD2fNSx1oCFSIMaUIgtr0qPq0SaVLVq5ho9k/OxSoTk7F0f6iAhKc
wdb8naUMqUcTFhs0YslzbSRGZNLZQ7cz4QVkR67T6g9uKxYC1NXafyQciP1cpPjc
cBuqhj7j5Ps3mHpthmmyCixn7r298ROZ00HsKrE1bEFHsRNF0VGMtzvl3rE49T61
BN7FMPnZ7YVsCOq6j5L6vUe6UqH4k+TFHm6udIUa4eQNjYaf7S8YVqWrA96gfYM=
=wM2C
-----END PGP SIGNATURE-----

--=_MailMate_637746C7-C397-4CC8-B358-3C32989FF3D2_=--