Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a1745d9b-4bfc-50d2-8da6-7631ae2b24d0@meta.com>
Date:   Wed, 30 Nov 2022 08:52:27 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.5.0
Subject: Re: [PATCH bpf v3] selftests/bpf: Fix "missing ENDBR" BUG for
 destructor kfunc
Content-Language: en-US
To:     Chen Hu <hu1.chen@intel.com>
Cc:     jpoimboe@kernel.org, memxor@gmail.com, bpf@vger.kernel.org,
        Pengfei Xu <pengfei.xu@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>,
        Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Martin KaFai Lau <martin.lau@linux.dev>,
        Song Liu <song@kernel.org>, Yonghong Song <yhs@fb.com>,
        John Fastabend <john.fastabend@gmail.com>,
        KP Singh <kpsingh@kernel.org>,
        Stanislav Fomichev <sdf@google.com>,
        Hao Luo <haoluo@google.com>, Jiri Olsa <jolsa@kernel.org>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Kees Cook <keescook@chromium.org>,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        netdev@vger.kernel.org
References: <20221130101135.26806-1-hu1.chen@intel.com>
From:   Yonghong Song <yhs@meta.com>
In-Reply-To: <20221130101135.26806-1-hu1.chen@intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?emMrWThpeUtJQzRYUTlzUmsvK2FNdWV1YUhydHpSRFFMa3ZJNEN6V1dXK0pT?=
 =?utf-8?B?c0dGRWRVZktCTGpWWVJBZFpaY3VNRUhEL0VuSzRWQnVpUGhuRUY0OUgwejZW?=
 =?utf-8?B?bmdhSTVRRDZWMkp4SjNlWVNvMWdBa09YTFgwQ1JiVTQ2VkFnRllrSmNQaWJK?=
 =?utf-8?B?R21PL252NHhMenh6YzlIckhYRzR1ZXBXY0lmdDdwbDVZOU4rWHFVT2FCNFlE?=
 =?utf-8?B?MkJDNVAwMFVFU29MQnlZVG1mcENZdHcvOWt3Q0R5UWt0Q1hreHl5RDEwb1hM?=
 =?utf-8?B?b1FUUDFadis1TnNzc3BHSlpuWklaRy9PRWZyMjVJSHQrRGFDMXpQZURGalZZ?=
 =?utf-8?B?MVcveDVrc1VEcEYrS2E4YlNQQ3k0ODU2cURyT1g4TmpnbTlrQ2J3RXZ3M1RQ?=
 =?utf-8?B?VjM5b1ArLzE3YkRXQXlRWHNRZ1I0ZC9LQWZWMW5DbVBzV2VWS0ZSVkttM3hv?=
 =?utf-8?B?WmVwN3JqU283RFczUzlJOFpsZ3ZSNUJkZkJXVDdTUnhHMWRIRHY0Q2xDeGZm?=
 =?utf-8?B?dnNMbHpZR20xbXZjRU1jcnNxY25qUTNTeTdzaXdwN3VTTWRRSy9SZ1YvcFBX?=
 =?utf-8?B?VkNxVWV2UHJ6amdoSXZsdEJkZTMxU1pJSEExSUlCR0FzTUdrdnAxTWNZQVJD?=
 =?utf-8?B?YjFHU2lyWXI3Nm5keXBJR0dZMlZVMS9yNFdEbE1DMndrTHZENWZGWEpwaGJ3?=
 =?utf-8?B?S09LVzN6WWJIMUhmcit0amk0RnNLdS9RUTdPWEEvU2tPYlhPeXlNb1h0V09o?=
 =?utf-8?B?RkJYS25ZTmtKeGlUbEV0THVFc2hBOVhsQXY2QkRJSk1IR1h5a081cXlOdzJx?=
 =?utf-8?B?UnNJU0NWdFM2OTNyU3lNdEIzYWk4RGgxSzVSUEE2MnJoS3ZBcXExQjd5M0hU?=
 =?utf-8?B?ejkwOTlwQ2lJcGlFU0lsMXU5SkRtbk1GMTNIVjBVSWt2WU8wSW5nN3ZocUd1?=
 =?utf-8?B?QVp0eG5sLzhwU0hPZlNoTEFGYjZlcTk1RHloc1cyQzVXZEZPZVpWNStDVkts?=
 =?utf-8?B?YVZoRkFjai9wWDk1cXdUbmRLWW9WZzZ5VGE5TEoyVVI5bG9CSHRwUEhHRXQy?=
 =?utf-8?B?L041VDNrbHhFNndyL0hEWXZMYVFxVmVZdHMrU2NwZXRmSytHdGd5ci9UNDJ0?=
 =?utf-8?B?c3NQVWVobVdBR1FjT3U5ZjRuSStiOVNTV0ZLMjFxZDBUM25ORGp0SzBEWVJz?=
 =?utf-8?B?N24zeVBrOHVmNG1tblFlSXdBWEhHTzBFcStlQk9FYmRTY0Z6RGtUSFdqQWdY?=
 =?utf-8?B?L21adFVMZFQrVmQxQXJGd1FISnh4d0xqZDJ5b3ZRa3F6K0FrZ3hzSWtCSVEx?=
 =?utf-8?B?NCsxaTYwd0k1MGxNMmhpUHB4b0dyanVudEx2eDNDaDB1ZUd4cjh4ODhYRVp3?=
 =?utf-8?B?bUlWMEZkVDVvMFdvMkYzeFlReE9HZU9nV3lON1MveDB0RkV3TEpRZGtGUm5a?=
 =?utf-8?B?S2NNdFBMSy9xY2MxNXlaRXc2TkswdWFENnR1eklONnYzd1h0OWd1Rm9pVVdQ?=
 =?utf-8?B?d0JZVUdiYW83TWo2VnNnYlVPbzB6SUJRRlpYWFVpd1VkZ1dPazNodzZCVFpS?=
 =?utf-8?B?SFFtWnFrNnhzRkZuc0xyd0JqSDdoLytTci9uSnYzRmZMbjJUWkxrWkJYUzZW?=
 =?utf-8?B?OXRHMWVGUzE0MndmNmJ2SmdBUlhkZ2lTUVZ0UjJzb1QrcU9CWjYxR3VwVHFG?=
 =?utf-8?B?aFU3Skx3TWJ2S1h3dDV3SzREOFFHNEtYUzU0V1AveW5SZUJtY1ZwK2lXbXBC?=
 =?utf-8?B?UzRsblkyMXAvYVBOelhHRHF4MDlhZHN2UE01Rm5pQ1BLL2o1UlRpSkZZaUlV?=
 =?utf-8?B?SG9wTmE0ZnR3djVjWGl0clo5Vmxna1lwKzlIZnI0S1FyRU9XdG9vdDA4bnNR?=
 =?utf-8?B?c0tKdlRXLzJQdnphaDZKakt2WWNJKzVuSGVTUXZKMDZ1ZTVIcEZ6cG1TUFBN?=
 =?utf-8?B?T3hmWERBR2VoMEU1TGw0emNhQkd1S3Z2a2NPeG9uTW5uRE9MQUVGVFdxRXhN?=
 =?utf-8?B?S0syRmZZdzJCM2l2WHN1Zno5d1gyanRRdmdEVEpNbDRKcjJ5Skdaa2FTRnN1?=
 =?utf-8?B?K3Voci95bjhJWG9KcmdXM21ZZTB4OWREclhwR3NRRXIvY29uZXZYVStLelBP?=
 =?utf-8?B?eVV2VVNBVzdjb0JtZUhZSUFWYjRpeHl3bVZrYzFXR0dRNEY2MXVrN1cydXIr?=
 =?utf-8?B?ZkE9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0019bde5-0609-4adc-acdb-08dad2f34535
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2022 16:52:30.8565
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vxmjD+cJJb1X5fu83O1iXpUfwvhT4KNRBkYILW2tAXb/6kg/rDpKzWtPTs94vlot
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1185
X-Proofpoint-ORIG-GUID: 85hyFoXotEJvcsBpBTFW5WOHOFHk5x8V
X-Proofpoint-GUID: 85hyFoXotEJvcsBpBTFW5WOHOFHk5x8V
Content-Transfer-Encoding: 7bit
X-Proofpoint-UnRewURL: 0 URL was un-rewritten
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-11-30_04,2022-11-30_02,2022-06-22_01
X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/30/22 2:11 AM, Chen Hu wrote:
> With CONFIG_X86_KERNEL_IBT enabled, the test_verifier triggers the
> following BUG:
> 
>    traps: Missing ENDBR: bpf_kfunc_call_test_release+0x0/0x30
>    ------------[ cut here ]------------
>    kernel BUG at arch/x86/kernel/traps.c:254!
>    invalid opcode: 0000 [#1] PREEMPT SMP
>    <TASK>
>     asm_exc_control_protection+0x26/0x50
>    RIP: 0010:bpf_kfunc_call_test_release+0x0/0x30
>    Code: 00 48 c7 c7 18 f2 e1 b4 e8 0d ca 8c ff 48 c7 c0 00 f2 e1 b4 c3
> 	0f 1f 44 00 00 66 0f 1f 00 0f 1f 44 00 00 0f 0b 31 c0 c3 66 90
>         <66> 0f 1f 00 0f 1f 44 00 00 48 85 ff 74 13 4c 8d 47 18 b8 ff ff ff
>     bpf_map_free_kptrs+0x2e/0x70
>     array_map_free+0x57/0x140
>     process_one_work+0x194/0x3a0
>     worker_thread+0x54/0x3a0
>     ? rescuer_thread+0x390/0x390
>     kthread+0xe9/0x110
>     ? kthread_complete_and_exit+0x20/0x20
> 
> It turns out that ENDBR in bpf_kfunc_call_test_release() is converted to
> NOP by apply_ibt_endbr().
> 
> The only text references to this function from kernel side are:
> 
>    $ grep -r bpf_kfunc_call_test_release
>    net/bpf/test_run.c:noinline void bpf_kfunc_call_test_release(...)
>    net/bpf/test_run.c:BTF_ID_FLAGS(func, bpf_kfunc_call_test_release, ...)
>    net/bpf/test_run.c:BTF_ID(func, bpf_kfunc_call_test_release)

We have some other function like this. For example, some newly added
functions like bpf_obj_new_impl(), bpf_obj_drop_impl(), do they have
the same missing endbr problem? If this is the case, we need a
general solution.

> 
> but it may be called from bpf program as kfunc. (no other caller from
> kernel)
> 
> This fix creates dummy references to destructor kfuncs so ENDBR stay
> there.
> 
> Also modify macro XXX_NOSEAL slightly:
> - ASM_IBT_NOSEAL now stands for pure asm
> - IBT_NOSEAL can be used directly in C
> 
> Signed-off-by: Chen Hu <hu1.chen@intel.com>
> Tested-by: Pengfei Xu <pengfei.xu@intel.com>
> ---
> v3:
> - Macro go to IBT related header as suggested by Jiri Olsa
> - Describe reference to the func clearly in commit message as suggested
>    by Peter Zijlstra and Jiri Olsa
>   
> v2: https://lore.kernel.org/all/20221122073244.21279-1-hu1.chen@intel.com/
> 
> v1: https://lore.kernel.org/all/20221121085113.611504-1-hu1.chen@intel.com/
> 
>   arch/x86/include/asm/ibt.h | 6 +++++-
>   arch/x86/kvm/emulate.c     | 2 +-
>   net/bpf/test_run.c         | 5 +++++
>   3 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/include/asm/ibt.h b/arch/x86/include/asm/ibt.h
> index 9b08082a5d9f..be86dc31661c 100644
> --- a/arch/x86/include/asm/ibt.h
> +++ b/arch/x86/include/asm/ibt.h
> @@ -36,11 +36,14 @@
>    * the function as needing to be "sealed" (i.e. ENDBR converted to NOP by
>    * apply_ibt_endbr()).
>    */
> -#define IBT_NOSEAL(fname)				\
> +#define ASM_IBT_NOSEAL(fname)				\
>   	".pushsection .discard.ibt_endbr_noseal\n\t"	\
>   	_ASM_PTR fname "\n\t"				\
>   	".popsection\n\t"
>   
> +#define IBT_NOSEAL(name)				\
> +	asm(ASM_IBT_NOSEAL(#name))
> +
>   static inline __attribute_const__ u32 gen_endbr(void)
>   {
>   	u32 endbr;
> @@ -94,6 +97,7 @@ extern __noendbr void ibt_restore(u64 save);
>   #ifndef __ASSEMBLY__
>   
>   #define ASM_ENDBR
> +#define ASM_IBT_NOSEAL(name)
>   #define IBT_NOSEAL(name)
>   
>   #define __noendbr
> diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
> index 4a43261d25a2..d870c8bb5831 100644
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
> @@ -327,7 +327,7 @@ static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop);
>   	".type " name ", @function \n\t" \
>   	name ":\n\t" \
>   	ASM_ENDBR \
> -	IBT_NOSEAL(name)
> +	ASM_IBT_NOSEAL(name)
>   
>   #define FOP_FUNC(name) \
>   	__FOP_FUNC(#name)
> diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
> index fcb3e6c5e03c..9e9c8e8d50d7 100644
> --- a/net/bpf/test_run.c
> +++ b/net/bpf/test_run.c
> @@ -601,6 +601,11 @@ noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p)
>   {
>   }
>   
> +#ifdef CONFIG_X86_KERNEL_IBT
> +IBT_NOSEAL(bpf_kfunc_call_test_release);
> +IBT_NOSEAL(bpf_kfunc_call_memb_release);
> +#endif
> +
>   noinline void bpf_kfunc_call_memb1_release(struct prog_test_member1 *p)
>   {
>   	WARN_ON_ONCE(1);