Received: by 2002:ab2:3c46:0:b0:1f5:f2ab:c469 with SMTP id x6csp175175lqf; Fri, 26 Apr 2024 03:26:43 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWkR3MQRswevJmrwlR95W8Eu2gbpn742nz9nkMPsbs8TjYoVINQ5/ZoMYc+WZ0NZ3KmBwbPw9ZRAVcr+omAmsw8/pvEiWNlEWBAv0hwBg== X-Google-Smtp-Source: AGHT+IEbnGwBlv+EA9/hIP4zuRlyGP5ujbvvbIkt+3DBxdx4ISM6HKmHWXxc/8GB+oE8JxdTEem2 X-Received: by 2002:a17:90a:e50d:b0:2af:b977:363a with SMTP id t13-20020a17090ae50d00b002afb977363amr2369928pjy.43.1714127202838; Fri, 26 Apr 2024 03:26:42 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1714127202; cv=pass; d=google.com; s=arc-20160816; b=lJCMHPaf5gIXeaDNMshamIgd8gg1N5bFwoAWH3iv2TGLqXrhAIRGJa9vabhKqRdRxR paAJLPT/oQZs/ANtWENPYYq8ri6R3zyea5XUKmBEZrkww8co6Kso3xQvV1lPFcKLZq2O bjXRJzDTRAIl9iOlEfwPbm2ZNOqIL1xL6XvT33blxvqT4AxA38I6CUQeJS3ruoMjc21A T5dXcIXB8BaIkJ4xjmGdS4M/AKDxtPR6uU2CVx27X+Vt1760w/Eo3AIm6dna/QLSUlAs d4HxaBd/UUA1AN+YC+Boo23uYzyVKKohK/QAwn7MJ/mMY58axyxC8d6FhQmflFwCAfKz Wt/Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:references :in-reply-to:subject:cc:to:from:dkim-signature; bh=zQmf+hskEM38OM0jMzavng/0wfiFH6WYhWSGFx+oMIM=; fh=6gRi3YCjG1e+PQBiSQ6kvZ1evqcR5arGVAojuYC63dk=; b=nxqoh64LMj6B3KrjO7AO0pO5TSEvpA70w98PjnFCe3yyK6q8IntOtoFQneL/D7B37M YLkEVhNeRksEg0mg0G9hoebK6hjCqWD3zyVdvUvH91o3NK38fFf42JbvUqCN6IMArBFj R3uYt+ptvTa83hbL59LZIwGYrB1nY8/JaLosSgaZIvpJ3TaM7W8O40iVtTjumVgg/QR+ hxVMOCAM7I8LryWlGW1LSKHnrOWBBdSNkZzcJihTg071mpo/cGQ1WkAlDO/dpiClL6/4 jf2ZQuA4DpEry6td7peaz3fb4a/XC+OluFwdciU3k56CUEFEc92UTK1Xcc/jYs/8IJRd YXWw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GIQp+fw1; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-159864-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-159864-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id oa11-20020a17090b1bcb00b002a5beeaa6d5si17235252pjb.139.2024.04.26.03.26.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Apr 2024 03:26:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-159864-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GIQp+fw1; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-159864-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-159864-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 3A22E281CBA for ; Fri, 26 Apr 2024 10:26:42 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3ECF0143C5F; Fri, 26 Apr 2024 10:26:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GIQp+fw1" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 165D413FD6F; Fri, 26 Apr 2024 10:26:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714127192; cv=none; b=Y1HYO+iQAjeDCKtfu+sExm0ebYlQLKA4MvYnis8a/Y7uUlGiUXhwG8hFksr2REXk3eaS5xOR1rWljeOexgilTeSTY7T3d18qXinvpHepAS22Z0XeWPPBzNcW9wIZbgAOp3jhQ9JnnUxahKvhVqOgIgF9dDHbW/7fo7L6gbF6oQk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714127192; c=relaxed/simple; bh=Mp1x52ASowv5NbAjsAc/PHMwV+sGt7uEmgf5kXMNu0o=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=mml0/N31l43DSKSB4HpemxAM+hS/af8vVvrJLLjMTLQmItLoJZKA0AO7D54lBfomRCqFKfbzNIjZ3b3uJmbf5BWGC56uOj+BGVceebSJtFN5AJTOyWs5xQzuhNlNKQfm9TrHMa1oH2Y+iYsOBYqJigZzXIFdoTXw5HKc+UBaGOs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GIQp+fw1; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4C7CFC113CD; Fri, 26 Apr 2024 10:26:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714127191; bh=Mp1x52ASowv5NbAjsAc/PHMwV+sGt7uEmgf5kXMNu0o=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=GIQp+fw1b+bCzFwKnZvtyr3gBd7s0dqMzzIfhjgPNDFBppuN0ni2nWziKZb0lXZoE 4Pl3XK0JIHC/wq2VFGZL2vRnKrYiXcfaOKY8FdZ9BMFMryc8MfDVgp5l4thjfYv4Vp hviMf8HKOnJU5xITrRXjCDZQGlLz8BXMdfpeFge4IDXmLHNHedtn+CH6E9pLLHpPLd +InkwL49wttyCk3bNG/X05HdLjagge4ED/TgNE/fP5QNst/bHmKVULAyVzNM3MIIyK 1TfXSaHwLDSkp7SwOQD4+wM9gxS4ftKLBhIdNQjBwgoxWALPOlT855UgnJTyPe1/m4 ultrnqWHzAPkA== From: Puranjay Mohan To: Andrii Nakryiko Cc: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: Re: [PATCH bpf-next v2 2/2] bpf, arm64: inline bpf_get_smp_processor_id() helper In-Reply-To: References: <20240424173550.16359-1-puranjay@kernel.org> <20240424173550.16359-3-puranjay@kernel.org> Date: Fri, 26 Apr 2024 10:26:28 +0000 Message-ID: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Andrii Nakryiko writes: > On Thu, Apr 25, 2024 at 11:56=E2=80=AFAM Puranjay Mohan wrote: >> >> Andrii Nakryiko writes: >> >> > On Thu, Apr 25, 2024 at 3:14=E2=80=AFAM Puranjay Mohan wrote: >> >> >> >> Andrii Nakryiko writes: >> >> >> >> > On Wed, Apr 24, 2024 at 10:36=E2=80=AFAM Puranjay Mohan wrote: >> >> >> >> >> >> As ARM64 JIT now implements BPF_MOV64_PERCPU_REG instruction, inli= ne >> >> >> bpf_get_smp_processor_id(). >> >> >> >> >> >> ARM64 uses the per-cpu variable cpu_number to store the cpu id. >> >> >> >> >> >> Here is how the BPF and ARM64 JITed assembly changes after this co= mmit: >> >> >> >> >> >> BPF >> >> >> =3D=3D=3D=3D=3D >> >> >> BEFORE AFTER >> >> >> -------- ------- >> >> >> >> >> >> int cpu =3D bpf_get_smp_processor_id(); int cpu =3D bpf_= get_smp_processor_id(); >> >> >> (85) call bpf_get_smp_processor_id#229032 (18) r0 =3D 0xffff= 800082072008 >> >> >> (bf) r0 =3D r0 >> >> > >> >> > nit: hmm, you are probably using a bit outdated bpftool, it should = be >> >> > emitted as: >> >> > >> >> > (bf) r0 =3D &(void __percpu *)(r0) >> >> >> >> Yes, I was using the bpftool shipped with the distro. I tried it again >> >> with the latest bpftool and it emitted this as expected. >> > >> > Cool, would be nice to update the commit message with the right syntax >> > for next revision, thanks! >> > >> >> Sure, will do. >> >> >> >> >> > >> >> >> (61) r0 =3D *(u32 = *)(r0 +0) >> >> >> >> >> >> ARM64 JIT >> >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D >> >> >> >> >> >> BEFORE AFTER >> >> >> -------- ------- >> >> >> >> >> >> int cpu =3D bpf_get_smp_processor_id(); int cpu =3D bpf_get_s= mp_processor_id(); >> >> >> mov x10, #0xfffffffffffff4d0 mov x7, #0xffff8000= ffffffff >> >> >> movk x10, #0x802b, lsl #16 movk x7, #0x8207, ls= l #16 >> >> >> movk x10, #0x8000, lsl #32 movk x7, #0x2008 >> >> >> blr x10 mrs x10, tpidr_el1 >> >> >> add x7, x0, #0x0 add x7, x7, x10 >> >> >> ldr w7, [x7] >> >> >> >> >> >> Performance improvement using benchmark[1] >> >> >> >> >> >> BEFORE AFTER >> >> >> -------- ------- >> >> >> >> >> >> glob-arr-inc : 23.817 =C2=B1 0.019M/s glob-arr-inc : = 24.631 =C2=B1 0.027M/s >> >> >> arr-inc : 23.253 =C2=B1 0.019M/s arr-inc : = 23.742 =C2=B1 0.023M/s >> >> >> hash-inc : 12.258 =C2=B1 0.010M/s hash-inc : = 12.625 =C2=B1 0.004M/s >> >> >> >> >> >> [1] https://github.com/anakryiko/linux/commit/8dec900975ef >> >> >> >> >> >> Signed-off-by: Puranjay Mohan >> >> >> --- >> >> >> kernel/bpf/verifier.c | 11 ++++++++++- >> >> >> 1 file changed, 10 insertions(+), 1 deletion(-) >> >> >> >> >> > >> >> > Besides the nits, lgtm. >> >> > >> >> > Acked-by: Andrii Nakryiko >> >> > >> >> >> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c >> >> >> index 9715c88cc025..3373be261889 100644 >> >> >> --- a/kernel/bpf/verifier.c >> >> >> +++ b/kernel/bpf/verifier.c >> >> >> @@ -20205,7 +20205,7 @@ static int do_misc_fixups(struct bpf_verif= ier_env *env) >> >> >> goto next_insn; >> >> >> } >> >> >> >> >> >> -#ifdef CONFIG_X86_64 >> >> >> +#if defined(CONFIG_X86_64) || defined(CONFIG_ARM64) >> >> > >> >> > I think you can drop this, we are protected by >> >> > bpf_jit_supports_percpu_insn() check and newly added inner #if/#elif >> >> > checks? >> >> >> >> If I remove this and later add support of percpu_insn on RISCV without >> >> inlining bpf_get_smp_processor_id() then it will cause problems here >> >> right? because then the last 5-6 lines inside this if(){} will be >> >> executed for RISCV. >> > >> > Just add >> > >> > #else >> > return -EFAULT; >> >> I don't think we can return. > > ah, because it's not an error condition, right > >> >> > #endif >> > >> > ? >> > >> > I'm trying to avoid this duplication of the defined(CONFIG_xxx) checks >> > for supported architectures. >> >> Does the following look correct? >> >> I will do it like this: >> >> /* Implement bpf_get_smp_processor_id() inline. */ >> if (insn->imm =3D=3D BPF_FUNC_get_smp_processor_id && >> prog->jit_requested && bpf_jit_supports_percpu_insn(= )) { >> /* BPF_FUNC_get_smp_processor_id inlining is an >> * optimization, so if pcpu_hot.cpu_number is ev= er >> * changed in some incompatible and hard to supp= ort >> * way, it's fine to back out this inlining logic >> */ >> #if defined(CONFIG_X86_64) >> insn_buf[0] =3D BPF_MOV32_IMM(BPF_REG_0, (u32)(u= nsigned long)&pcpu_hot.cpu_number); >> insn_buf[1] =3D BPF_MOV64_PERCPU_REG(BPF_REG_0, = BPF_REG_0); >> insn_buf[2] =3D BPF_LDX_MEM(BPF_W, BPF_REG_0, BP= F_REG_0, 0); >> cnt =3D 3; >> #elif defined(CONFIG_ARM64) >> struct bpf_insn cpu_number_addr[2] =3D { BPF_LD_= IMM64(BPF_REG_0, (u64)&cpu_number) }; >> >> insn_buf[0] =3D cpu_number_addr[0]; >> insn_buf[1] =3D cpu_number_addr[1]; >> insn_buf[2] =3D BPF_MOV64_PERCPU_REG(BPF_REG_0, = BPF_REG_0); >> insn_buf[3] =3D BPF_LDX_MEM(BPF_W, BPF_REG_0, BP= F_REG_0, 0); >> cnt =3D 4; >> #else >> goto next_insn; >> #endif > > yep, I just wrote a large comment about goto next_insns above and then > saw you already proposed that :) Yep, I think this is the way. > >> new_prog =3D bpf_patch_insn_data(env, i + delta,= insn_buf, cnt); >> if (!new_prog) >> return -ENOMEM; >> >> delta +=3D cnt - 1; >> env->prog =3D prog =3D new_prog; >> insn =3D new_prog->insnsi + i + delta; >> goto next_insn; >> } >> >> >> >> >> >> > >> >> >> /* Implement bpf_get_smp_processor_id() inline. */ >> >> >> if (insn->imm =3D=3D BPF_FUNC_get_smp_processor_id= && >> >> >> prog->jit_requested && bpf_jit_supports_percpu= _insn()) { >> >> >> @@ -20214,11 +20214,20 @@ static int do_misc_fixups(struct bpf_ver= ifier_env *env) >> >> >> * changed in some incompatible and hard t= o support >> >> >> * way, it's fine to back out this inlinin= g logic >> >> >> */ >> >> >> +#if defined(CONFIG_X86_64) >> >> >> insn_buf[0] =3D BPF_MOV32_IMM(BPF_REG_0, (= u32)(unsigned long)&pcpu_hot.cpu_number); >> >> >> insn_buf[1] =3D BPF_MOV64_PERCPU_REG(BPF_R= EG_0, BPF_REG_0); >> >> >> insn_buf[2] =3D BPF_LDX_MEM(BPF_W, BPF_REG= _0, BPF_REG_0, 0); >> >> >> cnt =3D 3; >> >> >> +#elif defined(CONFIG_ARM64) >> >> >> + struct bpf_insn cpu_number_addr[2] =3D { B= PF_LD_IMM64(BPF_REG_0, (u64)&cpu_number) }; >> >> >> >> >> > >> >> > this &cpu_number offset is not guaranteed to be within 4GB on arm64? >> >> >> >> Unfortunately, the per-cpu section is not placed in the first 4GB and >> >> therefore the per-cpu pointers are not 32-bit on ARM64. >> > >> > I see. It might make sense to turn x86-64 code into using MOV64_IMM as >> > well to keep more of the logic common. Then it will be just the >> > difference of an offset that's loaded. Give it a try? >> >> I think MOV64_IMM would have more overhead than MOV32_IMM and if we can >> use it in x86-64 we should keep doing it that way. Wdyt? > > My assumption (which I didn't check) was that BPF JITs should optimize > such MOV64_IMM that have a constant fitting within 32-bits with a > faster and smaller instruction. But I'm fine leaving it as is, of > course. You are right. I verified that the JITs will optimize this if the imm is 32-bit. So, I will make it common in the next version. Also, for the readers, we are discussing: 1) BPF_MOV32_IMM : This moves a 32 bit imm into a register and zero-extends it. 2) BPF_LD_IMM64 : This moves(loads) a 64 bit imm into a register. The JITs will optimize this to a BPF_MOV32_IMM, if the imm is 32-bit. Not to be confused with : 3) BPF_MOV64_IMM: This also works with a 32-bit imm but will sign extend it to 64-bit rather than zero-extend. Thanks, Puranjay