Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Ankur Arora <ankur.a.arora@oracle.com>
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org
Cc:     torvalds@linux-foundation.org, akpm@linux-foundation.org,
        mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org,
        tglx@linutronix.de, bp@alien8.de, peterz@infradead.org,
        ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com,
        jon.grimm@amd.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, joao.m.martins@oracle.com,
        ankur.a.arora@oracle.com
Subject: [PATCH v3 09/21] x86/asm: add clear_pages_movnt()
Date:   Mon,  6 Jun 2022 20:37:13 +0000
Message-Id: <20220606203725.1313715-5-ankur.a.arora@oracle.com>
In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com>
References: <20220606202109.1306034-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?PYHJgaeRKuns1B3PpeFt/3ZAk9cmcBMG/pVs6jNm7E0xXH+S+4lOmq0izUwz?=
 =?us-ascii?Q?vzzfwGg+OC1lyXJLRgF3e4tLzvzwOcBqmxVyKzrNQga9lpQEimSfarMOPx95?=
 =?us-ascii?Q?zFPn5NDW8IZpy51825jkHeYxR4ZoFnX7gyaFqiFpeFzoubE6c+4PSNnhtjoJ?=
 =?us-ascii?Q?crfTx6V9xD+eBxn/5lEunVJRLyaz03+JbKBf1EwI7tcvkoaMFPlIExNtnwvl?=
 =?us-ascii?Q?U6h4o4sSQI0KvcPzDeAukQOn3vnKnh5OaNSN3DZPOzYloQwYZT0hejDS32TH?=
 =?us-ascii?Q?kvxzf9u++thjEypvyppZ26t0KNUvMyfgXI+HVCV16QiK+kDBtztnRPNdzZ35?=
 =?us-ascii?Q?oRasz6xSKOsQ7FBo9D7VMgKFBpyeXWeAOIJsgU8OOs+liMdpugqqSrxzBH3o?=
 =?us-ascii?Q?55d7L+VfeZPAjZ3QTwBT/3uFrYBJYgq6cV5HX5iDv2frRmSLk2nw+oV1tlda?=
 =?us-ascii?Q?tf7zG+lQEwXO+2LWY6paK4C0UHltOTBmefLnqUJKjpyF2AqqYWuHQa4BDZTA?=
 =?us-ascii?Q?rpk1jBsKd3PcJsWC1u0rhPCv1EHXK4HtqI3gFE5Dz/1VoPCr8IHOMZTdHJej?=
 =?us-ascii?Q?Gwpx0LkH+gN4I2iajXy1v0X/odV4y2gBmVKEfqVlLavOkBQ0qQXM6hXmUevM?=
 =?us-ascii?Q?h1YH6vdDLkNd34L8grGzTfp0ynM5A+CjmeW4H6U9gxAxflz1EBbgqDYU5IeR?=
 =?us-ascii?Q?ewFrpksTW+FyU8ma6BZ9xzrg/lyEi7Gx2n1l8da6haNsy2jOf1qTkuTq6Tlp?=
 =?us-ascii?Q?6ZUMdC+XpwxB37vPXmuRDDp0iJwX4RndCeA2GTvgwqwG6KUxXUMWRdqx4jSU?=
 =?us-ascii?Q?AwyQfS4fTzrXkEZqCuU2a/Jgl3SP63XM8MfQDp3yBH+szsq/FMxlyWl668Y8?=
 =?us-ascii?Q?J2ZWxCJnEbDUByFwoPUQFa9D24BY6293gm2QHFsFk0Vx+7Z3tbMKmC+P3THW?=
 =?us-ascii?Q?7eJWp8+63RWQIUwfFOquNsizf60ZOstBx6+E7e3V6k6OVFGWxHSrkZrRvqDY?=
 =?us-ascii?Q?RNgDI4ODszpWxwFMpV5tSKl/cCgySeYaC6Ost6ehMDHc39qYr1jddpP1BJXk?=
 =?us-ascii?Q?zAfgM7Hkofh+zifmld4bzEi3UZbaU3b0FIJb/vyPNReSzIV2WhimyaftCsTI?=
 =?us-ascii?Q?E3KIqWIq7zSLouWOGiDdj/u3kCxZ3ASsm5YHzCzNs7Mci3T8+MyrPJVuqt+u?=
 =?us-ascii?Q?ULIEKa3fB/cVLhq8dmIe4SxY7YGrKTdbHvbyhEM+1kCH7YVVSNF8j69H9pzQ?=
 =?us-ascii?Q?KyO1mq31mZnnf3CDueNNW4pWLDlXpBjbvtuDT04l0ah1+JNMnbAzsCG0n3F7?=
 =?us-ascii?Q?EJkKn5anbE57mayE5jDQjy8PGRUIpDuChkOiPj9eUPj22ipaySTniKjH4Byh?=
 =?us-ascii?Q?1b57d24FaUwAoyewFTeYlNv5hiUVskUfgwOSDhIkewSj6Y9eU0eLZmLpfkWT?=
 =?us-ascii?Q?xvovhpS2Ifotm2CFluh55oLUjn2Vg/T+sZyUXGvRYLJ9+B82zJxcyhrz4QWi?=
 =?us-ascii?Q?nF9J5xAQIyI6t/+5d0PVf4F9u4oZQprroQDKrZKlkZ7U+HMs1xNcvXwJv8SG?=
 =?us-ascii?Q?gi2s4RNXiPkUZfZzZ5RKg12qYtenJVBZyENiU+qWiryXZS6PSSSHoEIfEEbj?=
 =?us-ascii?Q?oDO/VUkgO0kNUff7YrIi51sxYI6F8ptan9rovH7+jNnZDLm64Czim8HPomQV?=
 =?us-ascii?Q?zgAuTYSTaHWG3MweT6t0J10b89vy94vSrqPciAIs6jhngFWtr/3AjFDK+fQV?=
 =?us-ascii?Q?KqX5a8ed2S9hy7/mMz18o3XcAKSYQBE=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 64c30f3d-a8c2-46cf-5107-08da47fc735f
X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:02.1178
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GJf0u9W4+IQ9gbojQpnXOy6HjyDIo3nvSoONtZEmjfFpncg8VrnZEMbrD+DVgNu9H6Q5uJeedGoQ7SLwYf2BeOzD5zziymbXfTKap6SHISQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 adultscore=0
 spamscore=0 suspectscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2206060081
X-Proofpoint-ORIG-GUID: PzDcp0hrTT5fkCfnLjFQGqmyc9VxQ1SN
X-Proofpoint-GUID: PzDcp0hrTT5fkCfnLjFQGqmyc9VxQ1SN
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Add clear_pages_movnt(), which uses MOVNTI as the underlying primitive.
With this, page-clearing can skip the memory hierarchy, thus providing
a non cache-polluting implementation of clear_pages().

MOVNTI, from the Intel SDM, Volume 2B, 4-101:
 "The non-temporal hint is implemented by using a write combining (WC)
  memory type protocol when writing the data to memory. Using this
  protocol, the processor does not write the data into the cache
  hierarchy, nor does it fetch the corresponding cache line from memory
  into the cache hierarchy."

The AMD Arch Manual has something similar to say as well.

One use-case is to zero large extents without bringing in never-to-be-
accessed cachelines. Also, often clear_pages_movnt() based clearing is
faster once extent sizes are O(LLC-size).

As the excerpt notes, MOVNTI is weakly ordered with respect to other
instructions operating on the memory hierarchy. This needs to be
handled by the caller by executing an SFENCE when done.

The implementation is straight-forward: unroll the inner loop to keep
the code similar to memset_movnti(), so that we can gauge
clear_pages_movnt() performance via perf bench mem memset.

 # Intel Icelakex
 # Performance comparison of 'perf bench mem memset -l 1' for x86-64-stosb
 # (X86_FEATURE_ERMS) and x86-64-movnt:

 System:      Oracle X9-2 (2 nodes * 32 cores * 2 threads)
 Processor:   Intel Xeon(R) Platinum 8358 CPU @ 2.60GHz (Icelakex, 6:106:6)
 Memory:      512 GB evenly split between nodes
 LLC-size:    48MB for each node (32-cores * 2-threads)
 no_turbo: 1, Microcode: 0xd0001e0, scaling-governor: performance

              x86-64-stosb (5 runs)     x86-64-movnt (5 runs)    Delta(%)
              ----------------------    ---------------------    --------
     size            BW   (   stdev)          BW    (   stdev)

      2MB      14.37 GB/s ( +- 1.55)     12.59 GB/s ( +- 1.20)   -12.38%
     16MB      16.93 GB/s ( +- 2.61)     15.91 GB/s ( +- 2.74)    -6.02%
    128MB      12.12 GB/s ( +- 1.06)     22.33 GB/s ( +- 1.84)   +84.24%
   1024MB      12.12 GB/s ( +- 0.02)     23.92 GB/s ( +- 0.14)   +97.35%
   4096MB      12.08 GB/s ( +- 0.02)     23.98 GB/s ( +- 0.18)   +98.50%

Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 arch/x86/include/asm/page_64.h |  1 +
 arch/x86/lib/clear_page_64.S   | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h
index a88a3508888a..3affc4ecb8da 100644
--- a/arch/x86/include/asm/page_64.h
+++ b/arch/x86/include/asm/page_64.h
@@ -55,6 +55,7 @@ extern unsigned long __phys_addr_symbol(unsigned long);
 void clear_pages_orig(void *page, unsigned long npages);
 void clear_pages_rep(void *page, unsigned long npages);
 void clear_pages_erms(void *page, unsigned long npages);
+void clear_pages_movnt(void *page, unsigned long npages);
 
 #define __HAVE_ARCH_CLEAR_USER_PAGES
 static inline void clear_pages(void *page, unsigned int npages)
diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S
index 2cc3b681734a..83d14f1c9f57 100644
--- a/arch/x86/lib/clear_page_64.S
+++ b/arch/x86/lib/clear_page_64.S
@@ -58,3 +58,24 @@ SYM_FUNC_START(clear_pages_erms)
 	RET
 SYM_FUNC_END(clear_pages_erms)
 EXPORT_SYMBOL_GPL(clear_pages_erms)
+
+SYM_FUNC_START(clear_pages_movnt)
+	xorl	%eax,%eax
+	movq	%rsi,%rcx
+	shlq    $PAGE_SHIFT, %rcx
+
+	.p2align 4
+.Lstart:
+	movnti  %rax, 0x00(%rdi)
+	movnti  %rax, 0x08(%rdi)
+	movnti  %rax, 0x10(%rdi)
+	movnti  %rax, 0x18(%rdi)
+	movnti  %rax, 0x20(%rdi)
+	movnti  %rax, 0x28(%rdi)
+	movnti  %rax, 0x30(%rdi)
+	movnti  %rax, 0x38(%rdi)
+	addq    $0x40, %rdi
+	subl    $0x40, %ecx
+	ja      .Lstart
+	RET
+SYM_FUNC_END(clear_pages_movnt)
-- 
2.31.1