Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Ankur Arora <ankur.a.arora@oracle.com>
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org
Cc:     mingo@kernel.org, bp@alien8.de, luto@kernel.org,
        akpm@linux-foundation.org, mike.kravetz@oracle.com,
        jon.grimm@amd.com, kvm@vger.kernel.org, konrad.wilk@oracle.com,
        boris.ostrovsky@oracle.com, Ankur Arora <ankur.a.arora@oracle.com>
Subject: [PATCH v2 04/14] x86/asm: add clzero based page clearing
Date:   Wed, 20 Oct 2021 10:02:55 -0700
Message-Id: <20211020170305.376118-5-ankur.a.arora@oracle.com>
In-Reply-To: <20211020170305.376118-1-ankur.a.arora@oracle.com>
References: <20211020170305.376118-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?XrhWhUpgPHv0dOa4b0GNmvHLxYiCF8qDu432VsE//hQKBVI5V5lylaTd43Qz?=
 =?us-ascii?Q?y+h0qhxfU9OQk02zRXIrLfFQcQDFSrN9ywBp8eBkEXLbwbsK+nLp1jyFb4Xv?=
 =?us-ascii?Q?gP1fznTIqB6b+LJBvPDL8mhLyzaz0qJcUQ44PeLE4nTvGU01E5Q2sm3Vx5v+?=
 =?us-ascii?Q?ddtYnZ3NAOwXbIlo05Ww89woU7CR5B7RP5m8EIKsgCUeeeBprZ3S4IL5qu7w?=
 =?us-ascii?Q?WOjvtPK538qfaA7HFPkG0sSGT+wbSiLaXKQTCtR/HLkWay48cF/pBUr2BEDT?=
 =?us-ascii?Q?iTqv4XBFheiD8Eu9wqZ0wDFrm9ueP5yXktYKtCkAQEqzux6tTiPCD0FakAEg?=
 =?us-ascii?Q?uUxSNvLUlLJCYkYBYidzNdImfY2aUTVut2Y1t1mDR4jnB6O6hfspMY0Exm82?=
 =?us-ascii?Q?3poiVtk5tUzd4KTm7MDj2HuuWuLuy+gm7PUMJjVPgaE+ru2TVFslpZXDiRYl?=
 =?us-ascii?Q?2aaEuX1EKNorm4M0pDar7OV42WIW0AgaHHMpwlKkWBPveyut+bw3O0gu0lBd?=
 =?us-ascii?Q?rpCnzASjQHkwcRRfX+VfWlE6hmlZmLM+B/ETuf/uV6p0kaPdYLNQYmmMnOpE?=
 =?us-ascii?Q?XTIlCT/BBIS3ItBHm8f4UFTxA67oUoN5lKC8CialqwO8cdLMaBOUcJ8F0vmH?=
 =?us-ascii?Q?Vwhu1b/pPcKHCYoWmZtxKFyQI3NuRtIvXvqV6sxkeqBM7XmaSSmHy7KjDDcA?=
 =?us-ascii?Q?TRyGVhi5+Gl40R5dmGHdcX1np5Cj46BOf2k6EJcnpR3Hxqtc8Ob6Wc4SZ1Jb?=
 =?us-ascii?Q?WiYGdvtE9T+jBAj7eR0ihY5BwGAnG4gZC+64YIb162h8GxjcDXmjASiKj1vk?=
 =?us-ascii?Q?IEVvnfYfMdhmdQnj2RDIv99//JkXqp72yx98CvcuFC7p1nPkOICtNvZsJHh6?=
 =?us-ascii?Q?Z/VqVIxSlXKvaoNQiF30FFomS+Wr2wvCGxfUzlzpfBkQDFx5bgXT0cL+WCKl?=
 =?us-ascii?Q?0OZtLrp4+L8da+YMgxNNOCutOZzvQjPTCpcZ5PkhgSgc1bIa4kHXLLF71T6v?=
 =?us-ascii?Q?Jlp/OcAdkWaIMy2FvM1y+C4eq7KX3Yx2bNd6KXQVX6XBShEnIQ7b4qbVjN0f?=
 =?us-ascii?Q?oc7dfPvwvFRkFEj3pn1NUPKB0abmorqqdo7FJT0chmxnCer+7X1FzoohxuJt?=
 =?us-ascii?Q?QFZ7qXMWtYk0UuK+oB7yhdBpZeS4DeO797J2AgJzlUBaMz38IvjeCeSreebq?=
 =?us-ascii?Q?hxhaa7CjJdDmcMAHhTIMjX1uraWWJXnUyDtJPGeRxA+Cjqc53EYl1esLTRYq?=
 =?us-ascii?Q?Dq5Uk4ecN+BA29QwU2YYw/QZvAO6rREH9+0XbnkSHFpXY6IP/hqqzNuyajdT?=
 =?us-ascii?Q?avE3yecoCYsu13F4jEAlnrZ4RVYOwPi64tGRfVcKco4crwhO1+RxVz0ckZzl?=
 =?us-ascii?Q?ozUJX7PWlnXjN1NiYcFsOcZgqsZ5TgWRo5IS3Hf93x3wwxW5qN4WT4cROBP+?=
 =?us-ascii?Q?N3POkILvBu86sVfHPDl+OA0aUJpKMZle?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7914022a-422d-4a94-c2d0-08d993ebce5c
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2021 17:05:24.0470
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ankur.a.arora@oracle.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4577
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10143 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 adultscore=0
 spamscore=0 phishscore=0 bulkscore=0 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001
 definitions=main-2110200095
X-Proofpoint-GUID: 9_jWjnAdBB-xF-a_J6oHPhY7SbdzWyuj
X-Proofpoint-ORIG-GUID: 9_jWjnAdBB-xF-a_J6oHPhY7SbdzWyuj
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Add clear_page_clzero(), which uses CLZERO as the underlying primitive.
CLZERO skips the memory hierarchy, so this provides a non-polluting
implementation of clear_page(). Available if X86_FEATURE_CLZERO is set.

CLZERO, from the AMD architecture guide (Vol 3, Rev 3.30):
 "Clears the cache line specified by the logical address in rAX by
  writing a zero to every byte in the line. The instruction uses an
  implied non temporal memory type, similar to a streaming store, and
  uses the write combining protocol to minimize cache pollution.

  CLZERO is weakly-ordered with respect to other instructions that
  operate on memory. Software should use an SFENCE or stronger to
  enforce memory ordering of CLZERO with respect to other store
  instructions.

  The CLZERO instruction executes at any privilege level. CLZERO
  performs all the segmentation and paging checks that a store of
  the specified cache line would perform."

The use-case is similar to clear_page_movnt(), except that
clear_page_clzero() is expected to be more performant.

Cc: jon.grimm@amd.com
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 arch/x86/include/asm/page_64.h |  1 +
 arch/x86/lib/clear_page_64.S   | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h
index cfb95069cf9e..3c53f8ef8818 100644
--- a/arch/x86/include/asm/page_64.h
+++ b/arch/x86/include/asm/page_64.h
@@ -44,6 +44,7 @@ void clear_page_orig(void *page);
 void clear_page_rep(void *page);
 void clear_page_erms(void *page);
 void clear_page_movnt(void *page);
+void clear_page_clzero(void *page);
 
 static inline void clear_page(void *page)
 {
diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S
index 578f40db0716..1cb29a4454e1 100644
--- a/arch/x86/lib/clear_page_64.S
+++ b/arch/x86/lib/clear_page_64.S
@@ -76,3 +76,22 @@ SYM_FUNC_START(clear_page_movnt)
         ja      .Lstart
 	ret
 SYM_FUNC_END(clear_page_movnt)
+
+/*
+ * Zero a page using clzero (on AMD.)
+ * %rdi - page
+ *
+ * Caller needs to issue a sfence at the end.
+ */
+SYM_FUNC_START(clear_page_clzero)
+	movl	$4096,%ecx
+	movq	%rdi,%rax
+
+	.p2align 4
+.Liter:
+        clzero
+        addq    $0x40, %rax
+        sub     $0x40, %ecx
+        ja      .Liter
+	ret
+SYM_FUNC_END(clear_page_clzero)
-- 
2.29.2