Received: by 2002:a25:8b12:0:0:0:0:0 with SMTP id i18csp1457347ybl; Wed, 14 Aug 2019 17:45:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqwp6Zi0YWB19SM39IqFdCNNJQ92CtMISEwitijF2OaatqJajhj6kebxjF15smgSIigAq8d9 X-Received: by 2002:a17:90a:246f:: with SMTP id h102mr560765pje.125.1565829907115; Wed, 14 Aug 2019 17:45:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565829907; cv=none; d=google.com; s=arc-20160816; b=qoZPRU9lXCgwzTBUfNL7b16RU238PEegN68nNdGE2tR9xpisgAxhpoxTskd9fxLFiY 31Ca7xHhZeX2FyxucYaawpZ8va2lPnOs9jRfWFgaxLJPwLwdsj+d1cDtJUSH/1655Hh2 bld1voV5ZTc2fPTSHhWTKcnVkggZOLUREr2w6RWzylH6ZzSY2Y1qFk97+CnbDAKAFbyF QxEmb65tP1PYiM++Tzu3S7FkWyhfdxJmaWG4hiHhg9rvUufvM1LKzDBWFdTw1AMU2P5C ZySVXiacyE4weftjTlZ7RjJgNC5bNgQW8xQWsVz/vDrCqPJznBwZjrxCilRB/NBALB+H Heqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2Wzd0oM19a1Y2XLWndwD600p9ctH3XimEIED6JZrwmc=; b=kFEzn+W/H7Zwz4BjbbKvW2DIEzP/wA6LtK5DHp/0dcDIhJBKsIrq7aSCY98AaME5eV T0OQ7frTktXq+BjGcqa4TJynFOdm0qkgSSriHRAPfju9P9HVrRutq0NCZJFWillJljgk TaH/csuQDEQmzU1P5JWDqAoRgwSTmGY4+fsR7o95GcToICHWIM6YEW5rRh4SAR8CMOTz WDxWRzW6nIyaQrqiI3w6i5XXJWPjutZERRe25UpNrtkCwGc1PM39IiBFf1509iqHe0z4 jjkIau97qHgBYDC9++MQftHGNqJTdKWlAVAZzFERI/2Q/7Xmbge8DbORBsbEKtRZhugO tegA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fossix-org.20150623.gappssmtp.com header.s=20150623 header.b=BUF2xMkN; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k43si248359pje.59.2019.08.14.17.44.51; Wed, 14 Aug 2019 17:45:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@fossix-org.20150623.gappssmtp.com header.s=20150623 header.b=BUF2xMkN; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729597AbfHOAkS (ORCPT + 99 others); Wed, 14 Aug 2019 20:40:18 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:40237 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726490AbfHOAkS (ORCPT ); Wed, 14 Aug 2019 20:40:18 -0400 Received: by mail-pl1-f194.google.com with SMTP id a93so373454pla.7 for ; Wed, 14 Aug 2019 17:40:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2Wzd0oM19a1Y2XLWndwD600p9ctH3XimEIED6JZrwmc=; b=BUF2xMkNrRy+XObr0iloaLannIj4OnjJgiHHZEYhK9ebp6UPfAqFE2HkNcuw485JCZ XUILqfWgZ8d4EQQsUYoXurggNn0HUwY8DgOrdv0IU6alC0SYSnEZ+9KatNzCs3SG/6+o A6cxFZFWAaheBlm1sHW18itGY/zGV4DcmWuZEZlpjsQR7OyANvzMvGXkegOwAOKaeuGP 0ar+yk4KAFQlF+iqpYqQhLjpJZBXUORWqQVjBCSpSB8EZNylW8J3kGohEIa7ZxFmWx3I Xe5dUw6BeFJHT9Z4BgZIWlUuW71rkZ/BIMYrtW8i4pI+IyhXDn+11Djt+0i7Re7ubh4y xMlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2Wzd0oM19a1Y2XLWndwD600p9ctH3XimEIED6JZrwmc=; b=BdJCYGOA4ZByZiP4y4O7/e0UF2gEajwwDr1uvFBUwh+BqtMyeEx6TlsiAixVDIP67I KBEvbiBkrkMYTDXIeEvusZ6s1THbsPrmFZJNlPsAMmkG1HpXwbdojqErp5n0llzqmsoM sFANNFbnTkhKw3YlfgLAxxbZyEFa0gQuO/aos2NscfWRkkoshpqUDjqrphDNG8ViJ/78 DODVtY3PBRYW2FeUNDuLmwU4nuUts85p+NOz4KmS6CjMCt3fxnmf4RhmyTy/9f9/Q01m 8ub6EgYdFn6/YkDFFIv0oV/pbFyG+jSz0/Dvm5V78hvtRvkEojP/RZQUXO0qY+3/XkmL ZsgQ== X-Gm-Message-State: APjAAAUd3XcOQiTOi9O5lj29aQcRWWUpVKZ0hJGL7e1AjSRgoAZ0/7IF YCMEsBF6m2c3fdcj/vvDd5znVKqI0cFaWA== X-Received: by 2002:a17:902:f216:: with SMTP id gn22mr2027447plb.59.1565829617226; Wed, 14 Aug 2019 17:40:17 -0700 (PDT) Received: from santosiv.in.ibm.com ([49.205.218.176]) by smtp.gmail.com with ESMTPSA id g8sm815917pgk.1.2019.08.14.17.40.13 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 14 Aug 2019 17:40:16 -0700 (PDT) From: Santosh Sivaraj To: linuxppc-dev , Linux Kernel Cc: "Aneesh Kumar K.V" , Mahesh Salgaonkar , Reza Arbab , Balbir Singh , Chandan Rajendra , Michael Ellerman , Nicholas Piggin , christophe leroy Subject: [PATCH v10 5/7] powerpc/memcpy: Add memcpy_mcsafe for pmem Date: Thu, 15 Aug 2019 06:09:39 +0530 Message-Id: <20190815003941.18655-6-santosh@fossix.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190815003941.18655-1-santosh@fossix.org> References: <20190815003941.18655-1-santosh@fossix.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Balbir Singh The pmem infrastructure uses memcpy_mcsafe in the pmem layer so as to convert machine check exceptions into a return value on failure in case a machine check exception is encountered during the memcpy. The return value is the number of bytes remaining to be copied. This patch largely borrows from the copyuser_power7 logic and does not add the VMX optimizations, largely to keep the patch simple. If needed those optimizations can be folded in. Signed-off-by: Balbir Singh [arbab@linux.ibm.com: Added symbol export] Co-developed-by: Santosh Sivaraj Signed-off-by: Santosh Sivaraj --- arch/powerpc/include/asm/string.h | 2 + arch/powerpc/lib/Makefile | 2 +- arch/powerpc/lib/memcpy_mcsafe_64.S | 242 ++++++++++++++++++++++++++++ 3 files changed, 245 insertions(+), 1 deletion(-) create mode 100644 arch/powerpc/lib/memcpy_mcsafe_64.S diff --git a/arch/powerpc/include/asm/string.h b/arch/powerpc/include/asm/string.h index 9bf6dffb4090..b72692702f35 100644 --- a/arch/powerpc/include/asm/string.h +++ b/arch/powerpc/include/asm/string.h @@ -53,7 +53,9 @@ void *__memmove(void *to, const void *from, __kernel_size_t n); #ifndef CONFIG_KASAN #define __HAVE_ARCH_MEMSET32 #define __HAVE_ARCH_MEMSET64 +#define __HAVE_ARCH_MEMCPY_MCSAFE +extern int memcpy_mcsafe(void *dst, const void *src, __kernel_size_t sz); extern void *__memset16(uint16_t *, uint16_t v, __kernel_size_t); extern void *__memset32(uint32_t *, uint32_t v, __kernel_size_t); extern void *__memset64(uint64_t *, uint64_t v, __kernel_size_t); diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index eebc782d89a5..fa6b1b657b43 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -39,7 +39,7 @@ obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \ memcpy_power7.o obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \ - memcpy_64.o pmem.o + memcpy_64.o pmem.o memcpy_mcsafe_64.o obj64-$(CONFIG_SMP) += locks.o obj64-$(CONFIG_ALTIVEC) += vmx-helper.o diff --git a/arch/powerpc/lib/memcpy_mcsafe_64.S b/arch/powerpc/lib/memcpy_mcsafe_64.S new file mode 100644 index 000000000000..949976dc115d --- /dev/null +++ b/arch/powerpc/lib/memcpy_mcsafe_64.S @@ -0,0 +1,242 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) IBM Corporation, 2011 + * Derived from copyuser_power7.s by Anton Blanchard + * Author - Balbir Singh + */ +#include +#include +#include + + .macro err1 +100: + EX_TABLE(100b,.Ldo_err1) + .endm + + .macro err2 +200: + EX_TABLE(200b,.Ldo_err2) + .endm + + .macro err3 +300: EX_TABLE(300b,.Ldone) + .endm + +.Ldo_err2: + ld r22,STK_REG(R22)(r1) + ld r21,STK_REG(R21)(r1) + ld r20,STK_REG(R20)(r1) + ld r19,STK_REG(R19)(r1) + ld r18,STK_REG(R18)(r1) + ld r17,STK_REG(R17)(r1) + ld r16,STK_REG(R16)(r1) + ld r15,STK_REG(R15)(r1) + ld r14,STK_REG(R14)(r1) + addi r1,r1,STACKFRAMESIZE +.Ldo_err1: + /* Do a byte by byte copy to get the exact remaining size */ + mtctr r7 +46: +err3; lbz r0,0(r4) + addi r4,r4,1 +err3; stb r0,0(r3) + addi r3,r3,1 + bdnz 46b + li r3,0 + blr + +.Ldone: + mfctr r3 + blr + + +_GLOBAL(memcpy_mcsafe) + mr r7,r5 + cmpldi r5,16 + blt .Lshort_copy + +.Lcopy: + /* Get the source 8B aligned */ + neg r6,r4 + mtocrf 0x01,r6 + clrldi r6,r6,(64-3) + + bf cr7*4+3,1f +err1; lbz r0,0(r4) + addi r4,r4,1 +err1; stb r0,0(r3) + addi r3,r3,1 + subi r7,r7,1 + +1: bf cr7*4+2,2f +err1; lhz r0,0(r4) + addi r4,r4,2 +err1; sth r0,0(r3) + addi r3,r3,2 + subi r7,r7,2 + +2: bf cr7*4+1,3f +err1; lwz r0,0(r4) + addi r4,r4,4 +err1; stw r0,0(r3) + addi r3,r3,4 + subi r7,r7,4 + +3: sub r5,r5,r6 + cmpldi r5,128 + blt 5f + + mflr r0 + stdu r1,-STACKFRAMESIZE(r1) + std r14,STK_REG(R14)(r1) + std r15,STK_REG(R15)(r1) + std r16,STK_REG(R16)(r1) + std r17,STK_REG(R17)(r1) + std r18,STK_REG(R18)(r1) + std r19,STK_REG(R19)(r1) + std r20,STK_REG(R20)(r1) + std r21,STK_REG(R21)(r1) + std r22,STK_REG(R22)(r1) + std r0,STACKFRAMESIZE+16(r1) + + srdi r6,r5,7 + mtctr r6 + + /* Now do cacheline (128B) sized loads and stores. */ + .align 5 +4: +err2; ld r0,0(r4) +err2; ld r6,8(r4) +err2; ld r8,16(r4) +err2; ld r9,24(r4) +err2; ld r10,32(r4) +err2; ld r11,40(r4) +err2; ld r12,48(r4) +err2; ld r14,56(r4) +err2; ld r15,64(r4) +err2; ld r16,72(r4) +err2; ld r17,80(r4) +err2; ld r18,88(r4) +err2; ld r19,96(r4) +err2; ld r20,104(r4) +err2; ld r21,112(r4) +err2; ld r22,120(r4) + addi r4,r4,128 +err2; std r0,0(r3) +err2; std r6,8(r3) +err2; std r8,16(r3) +err2; std r9,24(r3) +err2; std r10,32(r3) +err2; std r11,40(r3) +err2; std r12,48(r3) +err2; std r14,56(r3) +err2; std r15,64(r3) +err2; std r16,72(r3) +err2; std r17,80(r3) +err2; std r18,88(r3) +err2; std r19,96(r3) +err2; std r20,104(r3) +err2; std r21,112(r3) +err2; std r22,120(r3) + addi r3,r3,128 + subi r7,r7,128 + bdnz 4b + + clrldi r5,r5,(64-7) + + /* Up to 127B to go */ +5: srdi r6,r5,4 + mtocrf 0x01,r6 + +6: bf cr7*4+1,7f +err2; ld r0,0(r4) +err2; ld r6,8(r4) +err2; ld r8,16(r4) +err2; ld r9,24(r4) +err2; ld r10,32(r4) +err2; ld r11,40(r4) +err2; ld r12,48(r4) +err2; ld r14,56(r4) + addi r4,r4,64 +err2; std r0,0(r3) +err2; std r6,8(r3) +err2; std r8,16(r3) +err2; std r9,24(r3) +err2; std r10,32(r3) +err2; std r11,40(r3) +err2; std r12,48(r3) +err2; std r14,56(r3) + addi r3,r3,64 + subi r7,r7,64 + +7: ld r14,STK_REG(R14)(r1) + ld r15,STK_REG(R15)(r1) + ld r16,STK_REG(R16)(r1) + ld r17,STK_REG(R17)(r1) + ld r18,STK_REG(R18)(r1) + ld r19,STK_REG(R19)(r1) + ld r20,STK_REG(R20)(r1) + ld r21,STK_REG(R21)(r1) + ld r22,STK_REG(R22)(r1) + addi r1,r1,STACKFRAMESIZE + + /* Up to 63B to go */ + bf cr7*4+2,8f +err1; ld r0,0(r4) +err1; ld r6,8(r4) +err1; ld r8,16(r4) +err1; ld r9,24(r4) + addi r4,r4,32 +err1; std r0,0(r3) +err1; std r6,8(r3) +err1; std r8,16(r3) +err1; std r9,24(r3) + addi r3,r3,32 + subi r7,r7,32 + + /* Up to 31B to go */ +8: bf cr7*4+3,9f +err1; ld r0,0(r4) +err1; ld r6,8(r4) + addi r4,r4,16 +err1; std r0,0(r3) +err1; std r6,8(r3) + addi r3,r3,16 + subi r7,r7,16 + +9: clrldi r5,r5,(64-4) + + /* Up to 15B to go */ +.Lshort_copy: + mtocrf 0x01,r5 + bf cr7*4+0,12f +err1; lwz r0,0(r4) /* Less chance of a reject with word ops */ +err1; lwz r6,4(r4) + addi r4,r4,8 +err1; stw r0,0(r3) +err1; stw r6,4(r3) + addi r3,r3,8 + subi r7,r7,8 + +12: bf cr7*4+1,13f +err1; lwz r0,0(r4) + addi r4,r4,4 +err1; stw r0,0(r3) + addi r3,r3,4 + subi r7,r7,4 + +13: bf cr7*4+2,14f +err1; lhz r0,0(r4) + addi r4,r4,2 +err1; sth r0,0(r3) + addi r3,r3,2 + subi r7,r7,2 + +14: bf cr7*4+3,15f +err1; lbz r0,0(r4) +err1; stb r0,0(r3) + +15: li r3,0 + blr + +EXPORT_SYMBOL_GPL(memcpy_mcsafe); -- 2.21.0