Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Ankur Arora <ankur.a.arora@oracle.com>
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org
Cc:     torvalds@linux-foundation.org, akpm@linux-foundation.org,
        mike.kravetz@oracle.com, mingo@kernel.org, luto@kernel.org,
        tglx@linutronix.de, bp@alien8.de, peterz@infradead.org,
        ak@linux.intel.com, arnd@arndb.de, jgg@nvidia.com,
        jon.grimm@amd.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, joao.m.martins@oracle.com,
        ankur.a.arora@oracle.com
Subject: [PATCH v3 10/21] x86/asm: add clear_pages_clzero()
Date:   Mon,  6 Jun 2022 20:37:14 +0000
Message-Id: <20220606203725.1313715-6-ankur.a.arora@oracle.com>
In-Reply-To: <20220606202109.1306034-1-ankur.a.arora@oracle.com>
References: <20220606202109.1306034-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BVBAT0yY6X9AH+HLUCnHahEdIsaxDmR/AGINUVDVBENUyY4fx+q91HOZCT72?=
 =?us-ascii?Q?J0l4ZkHVIjd8Kh9QUCtc/1WvNXYg7NLYc0pHCNKx+/icSA7Oemku4WGhA5DP?=
 =?us-ascii?Q?0CK11dez8BrVzGUawyZHEE8plyIkuwZpAu2Le9zMo7UWhQzMMtlZRZeUc9dt?=
 =?us-ascii?Q?svIApn1IdLhCNxgdZGqyy1U+CCFDl5pAi7wubohgCzSwNonFGtyaQHichNPo?=
 =?us-ascii?Q?ErNh6lTbE3jnJZgDE8vAKAqz6vteDtV/qrlRoNFo6y2nIGW1aGlTya/ah8WB?=
 =?us-ascii?Q?n1bvmEiWc200L/kkYIjFmvmIt/peLAVlGrLAyQEuvxxXf5mlDWTybemSP+cp?=
 =?us-ascii?Q?p3K8GEnyTFCaDd1rdJ6u5vrQuiT3mKPR2t84XQnqE7UWrUCPGVDXUw5JT3Kw?=
 =?us-ascii?Q?BCvVCyQwgvoHpOVIcH8vRgTzVDlAxDveKN2Y95u7CaTNq+ulepVGoDUu6KZM?=
 =?us-ascii?Q?8nEscrxXjFCHd5KJR4kHQtrUucv9h8jdhREgT1RRMcb1Jkl1VsZb3uGBm7bE?=
 =?us-ascii?Q?DT9FynYCJ1gYl9wlRQyTArrjjpFKK1IS2WycYxBaOQQ0Hs+oM67f0zHgYvWW?=
 =?us-ascii?Q?HTDcEvsb80iAhR7eZimQ8I413bjC2O0Zcw542BwyEpTqMG8F6o1PBL3BuFv7?=
 =?us-ascii?Q?22BwhVtcgzM10x9xnEPeXHzmU1LPIBi3XGhjCO95ph1eDPpzr11A+U2PPpHv?=
 =?us-ascii?Q?u83g5KVT3iB/T6J7TQ4pnVX/YHoHyhjf1zu4ail47AdsEm2oGSLOm1lkk9S1?=
 =?us-ascii?Q?epPXU8riv/nvgKpBK2vvgWW8OYkj0ZCVUPo/V0S9P8xR3ngofiEjKFIzvwuv?=
 =?us-ascii?Q?waAALQc8msZyHaMafbIFWYtcTUstuzkk1FeJ848F0fxXjElEk1zw2TP5uXau?=
 =?us-ascii?Q?kN86aHpOw2S8gfyjCJli7Vytu/05gi2XJe+x7OaCMrqGTmzNgGaMKmqItUIi?=
 =?us-ascii?Q?ljSSOoJfdLrDZxmoAHl+D/St/7TXAshMb70g+LsBY4vFtHiu1DfN1lFBxwJH?=
 =?us-ascii?Q?CG22HSyHPgSRVNYnJc9SneDe2gJoEGNUgFxaEAcyuWsa98d9ZekLPcQ8XM1f?=
 =?us-ascii?Q?GlNx0y9WYwaVyjeosgn4hvuP2f3Lpkw0toRk0cDiPqv3frAMFsub4yoiwOWK?=
 =?us-ascii?Q?CMbiXlDXsa63NRNc5PM8C4Xg999N0nn4QXbY8XP/ztBlTATFZk8Az4xYFkjV?=
 =?us-ascii?Q?0nhfef0GMU47QWX4r9UfStzXxf9ffLA2hGjmH/4CoEqxYSaVrV5OasWcgFCL?=
 =?us-ascii?Q?2fxvOZ5JPOhMVVadoH9cact2TbnVKKKmGuhWgMs5bOtEwowl82KNOhqDJaPh?=
 =?us-ascii?Q?jCDygf04OGNCgMwFZJdK/6hPhVcCh2/BenuVa3ax8939WLKCPSSQkuHWKibX?=
 =?us-ascii?Q?/gMZKMbOjgGnASZIWCaueEGfBBVuKlFX7FXFcue4cYdPAi4o/Io+FAJG1c54?=
 =?us-ascii?Q?g0wRuG/Ns+mIXNwtioloL3gGh/swHzFefQW3XrVtWuMqAMHOANmDgzufo3XS?=
 =?us-ascii?Q?6+ZgXif6JZw+/U823N/L06ud6n7lmD1jOdo3GlpImri7StWiblxdGHTemWnK?=
 =?us-ascii?Q?3rdNzbUG8tEr+NkzlNWtfel5eBz9T4unMWG6k1O/lVw39HUO/VqCdbnGpHNS?=
 =?us-ascii?Q?+xs3PsGMncHIObe9thl14lOpXK9nB8WL5WK5dv/ownx7YtL1ttPy2c3jqO8u?=
 =?us-ascii?Q?bUgrM8ymOWmK3c7lWHp/OkyVYtOsjZSAf/F63qyvmLjrSYYqzP4Z9TCYYsYo?=
 =?us-ascii?Q?qY9dCtsY2Qjo/MMnF8iNx8izhy6Q/zQ=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: be1ccfdc-dd40-45d8-af12-08da47fc752c
X-MS-Exchange-CrossTenant-AuthSource: DM8PR10MB5416.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 20:38:05.1102
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: eTfZpQyXi4kGQkBQ2pSxGoFLvNaTxZPceFK9/L2GDb1WXHBZwIPJxkN8MReB2ffrl/IAf+A3Sgr4FhTAWVIFLZUM96gEnGRiXLs6k11hNbs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2596
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-06-06_06:2022-06-02,2022-06-06 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0
 adultscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2206060081
X-Proofpoint-GUID: -PXLBj47JlhPQ80ioHpcofYKmtgt8nqQ
X-Proofpoint-ORIG-GUID: -PXLBj47JlhPQ80ioHpcofYKmtgt8nqQ
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Add clear_pages_clzero(), which uses CLZERO as the clearing primitive.
CLZERO skips the memory hierarchy, so this provides a non-polluting
implementation of clear_page(). Available if X86_FEATURE_CLZERO is set.

CLZERO, from the AMD architecture guide (Vol 3, Rev 3.30):
 "Clears the cache line specified by the logical address in rAX by
  writing a zero to every byte in the line. The instruction uses an
  implied non temporal memory type, similar to a streaming store, and
  uses the write combining protocol to minimize cache pollution.

  CLZERO is weakly-ordered with respect to other instructions that
  operate on memory. Software should use an SFENCE or stronger to
  enforce memory ordering of CLZERO with respect to other store
  instructions.

  The CLZERO instruction executes at any privilege level. CLZERO
  performs all the segmentation and paging checks that a store of
  the specified cache line would perform."

The use-case is similar to clear_page_movnt(), except that
clear_pages_clzero() is expected to be more performant.

Cc: jon.grimm@amd.com
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 arch/x86/include/asm/page_64.h |  1 +
 arch/x86/lib/clear_page_64.S   | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h
index 3affc4ecb8da..e8d4698fda65 100644
--- a/arch/x86/include/asm/page_64.h
+++ b/arch/x86/include/asm/page_64.h
@@ -56,6 +56,7 @@ void clear_pages_orig(void *page, unsigned long npages);
 void clear_pages_rep(void *page, unsigned long npages);
 void clear_pages_erms(void *page, unsigned long npages);
 void clear_pages_movnt(void *page, unsigned long npages);
+void clear_pages_clzero(void *page, unsigned long npages);
 
 #define __HAVE_ARCH_CLEAR_USER_PAGES
 static inline void clear_pages(void *page, unsigned int npages)
diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S
index 83d14f1c9f57..00203103cf77 100644
--- a/arch/x86/lib/clear_page_64.S
+++ b/arch/x86/lib/clear_page_64.S
@@ -79,3 +79,22 @@ SYM_FUNC_START(clear_pages_movnt)
 	ja      .Lstart
 	RET
 SYM_FUNC_END(clear_pages_movnt)
+
+/*
+ * Zero a page using clzero (On AMD, with CPU_FEATURE_CLZERO.)
+ *
+ * Caller needs to issue a sfence at the end.
+ */
+SYM_FUNC_START(clear_pages_clzero)
+	movq	%rdi,%rax
+	movq	%rsi,%rcx
+	shlq    $PAGE_SHIFT, %rcx
+
+	.p2align 4
+.Liter:
+	clzero
+	addq    $0x40, %rax
+	subl    $0x40, %ecx
+	ja      .Liter
+	RET
+SYM_FUNC_END(clear_pages_clzero)
-- 
2.31.1