Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5c9d77bf-75f5-954a-c691-39869bb22127@meta.com>
Date:   Fri, 9 Dec 2022 12:31:06 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.5.1
Subject: Re: BUG: unable to handle kernel paging request in bpf_dispatcher_xdp
Content-Language: en-US
To:     Jiri Olsa <olsajiri@gmail.com>
Cc:     Alexei Starovoitov <alexei.starovoitov@gmail.com>,
        Song Liu <song@kernel.org>, Hao Sun <sunhao.th@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>,
        bpf <bpf@vger.kernel.org>, Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        John Fastabend <john.fastabend@gmail.com>,
        Andrii Nakryiko <andrii@kernel.org>,
        Martin KaFai Lau <martin.lau@linux.dev>,
        Yonghong Song <yhs@fb.com>, KP Singh <kpsingh@kernel.org>,
        Stanislav Fomichev <sdf@google.com>,
        Hao Luo <haoluo@google.com>,
        David Miller <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>,
        Jesper Dangaard Brouer <hawk@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        netdev <netdev@vger.kernel.org>,
        Thorsten Leemhuis <regressions@leemhuis.info>
References: <CACkBjsbD4SWoAmhYFR2qkP1b6JHO3Og0Vyve0=FO-Jb2JGGRfw@mail.gmail.com>
 <Y49dMUsX2YgHK0J+@krava>
 <CAADnVQ+w-xtH=oWPYszG-TqxcHmbrKJK10C=P-o2Ouicx-9OUA@mail.gmail.com>
 <CAADnVQJ+9oiPEJaSgoXOmZwUEq9FnyLR3Kp38E_vuQo2PmDsbg@mail.gmail.com>
 <Y5Inw4HtkA2ql8GF@krava> <Y5JkomOZaCETLDaZ@krava> <Y5JtACA8ay5QNEi7@krava>
 <Y5LfMGbOHpaBfuw4@krava> <Y5MaffJOe1QtumSN@krava> <Y5M9P95l85oMHki9@krava>
 <Y5NSStSi7h9Vdo/j@krava>
From:   Yonghong Song <yhs@meta.com>
In-Reply-To: <Y5NSStSi7h9Vdo/j@krava>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c01GdkNnZDVNcGh2d1JFVmVEcUNxQmhKN3htb2p3eTFYek5ianFzQ1V6L3lh?=
 =?utf-8?B?ZG1nalY3S2ZHZFlvNHFRSGVBL2s0L0NVdk0xeENYZUg0Mk9KNDVDV2RQR2Ju?=
 =?utf-8?B?OHZTZ0tmNCszaVhMNVdLVnRvSmtzY3RnTHNxSW9iSDVvQXNnc0tWNUh5UFUx?=
 =?utf-8?B?ckp2V3NQSUl0ZkdJaGpwUUhOalVDMnNIbnRiVXQ5bFZ1WTloNUN0amZjMms3?=
 =?utf-8?B?S0Y4UEhlSHh1RXMrQTBjS3ZVcXRuaUFTTkJHWGNCdFBLQzJDK3BIOGpTZEVw?=
 =?utf-8?B?U1NTM2VvVXp0L2xkL0dlWTdPRGk3aVNiUGpXdFNFL2tmR2dhY0MzaGxhZXJl?=
 =?utf-8?B?SnNDTUlPS0VZaCtwT2pzczh4SVg4d2poV3NJMm1oZDQ0SEo3ZnNQQldWM214?=
 =?utf-8?B?dzVyS25GN1FuNUYrK2xpRFNKSmM2VWlnNGowK3BDQW41aER3QTk3Ukx6Znly?=
 =?utf-8?B?SExMaFhadzhaaEozN3ZGL3pQUU1lUzBtNXlwVCtHcUxiRnVBeHdPYkRPRGQ5?=
 =?utf-8?B?WGNpV0JwTEVuQ2M1S3JqTEJ5YnNGbFRGVXovVndQdXhkdEV2MnpoMjUxOUZk?=
 =?utf-8?B?ck90ZldxNk04RzNRRmZtelZTZE9RNnoyY1MrVnQwUk41c2tSNkl1K3NzTURw?=
 =?utf-8?B?dHhkNVdXaS8zSlM3dUlqYUJHQjc5UnNobGRXUU9Eb2V1a2hJM1I5YVoycEV5?=
 =?utf-8?B?NVNVS1pscm5abEJCSFdQcmNlNVhXNlpCK3ZGWUdoY3dhbU1CMUZJdlhIckJx?=
 =?utf-8?B?U3pyRUk2TGJNWGtOWGk2bU5rYWhwcVRONU85K01JWm5XWDFjbVNCbnU1aGFF?=
 =?utf-8?B?ekgxMDhUcTd2d09yWE9UbFBZeml5NmFwelNtN2Q0Z3QwRHFlVFBwR0lVcUVM?=
 =?utf-8?B?Y05vWUlmSGxvVndpS1ZVMytWRzhwZnlETnpvdDIvM3Q0N1NUQmorWDlGblFO?=
 =?utf-8?B?elVoWmplb0Z5MW15M2l2dGpxZy95UDRaWUJlZVMvcmJMRHp4WjV5TkFkTWZy?=
 =?utf-8?B?MzloVWh2ZnRNT3N6dWlMN2xjMU00QnlZWE9YNGE0dnhxMlVlR2EzdEh2RXhG?=
 =?utf-8?B?aXo4SWVqL0JtMkxoaWFHMGI4c0FVSzgvdTAxdFM1anNxL2hPYmhTb1lPQmdQ?=
 =?utf-8?B?aDRjRElkK2UxdHB5djdWM2gyQnc5VFZXbFd3OEVOU3k3SmYrTUVwV2ZMSWJi?=
 =?utf-8?B?Y20xOVpNWERrNFpVYmgwV05aOHVlcFRCaHlKRDA0RHJvc0ZTU2twVVF4dnZW?=
 =?utf-8?B?NmI0OGpMVG5lQS8wUXdwR3BJbmpGSUIrU1l3MGhDUDhPOXNsZU94eDNWbjIz?=
 =?utf-8?B?RlpNaktVVytrUkVGSjloSkFRNncrTy9KNWlNbnl3SDhURk14SlpzTjRaeWRq?=
 =?utf-8?B?dy9McEY5amVpMFdUdm1PaHNIcFVyeU85SVlMUjlydnd2Tm11Mldid3dYWlZ6?=
 =?utf-8?B?cytpSTh2Y1lKYW5wMFN5bjJTeXlVbmlsSktIN0IwRVpKbnFySTBjeEdSRnpK?=
 =?utf-8?B?RkxzeUh0UFQ2blM3TWdwYkNjbjlhZXFqZVVwQ29xQmpkYXZUd3RGLzY4SkV1?=
 =?utf-8?B?SitIeUpsTmVaRDcvZGMxcG1zbzI3aXY1RUI2d01yVnFaVzZqWTVod3hrYUw0?=
 =?utf-8?B?S2NYTHRTM2Rhemh2NmhPdW1ET1dsQWRtaXV5VEcwOVNBc0h0eWpSdmNkdE9Q?=
 =?utf-8?B?cU9BaHoxVUZxaUk4M0tEUGY5YWhiUGNSMlRBSHNWU1pTdGJsN0dobVJhcml4?=
 =?utf-8?B?QVl3R1pUQ1BjeTlTNUxIMk9NYVJYMDdjVG9aYklIT3prTkpOUzNNSkpka0Rj?=
 =?utf-8?B?c0NiVExHMW0yekFqVS9zcFJmSUFUMjBBQ3IzS3pWM2dOYitiTHZzSVE0bnBh?=
 =?utf-8?B?ZmNOTUhiZ092QXEvRTczQ05EcDM1dnRZNzhCQ0Fra2IvOHE3Yy9qenA3bG5x?=
 =?utf-8?B?c1o5THJPS0V0T3NndG1haTYrVXlYUGtmbHA2QXJxaWNhYXZYMld0RE1XVUIw?=
 =?utf-8?B?MDdqVEoyUG5vSDdhYWlvTkVrTFgrM2V4K3RwK2toR09lck5Na0NzNjBBVWta?=
 =?utf-8?B?M2xTRWNJeHJ4ajNoQm9ZU2o1YkplVWZYdEFoS3ZxQ3dCc1hCdHBpeDZjUXJE?=
 =?utf-8?B?MVUrMXg1UzRKcWFPdkZncnRKeHJEeTB5MHRiSlFUTUdMaWU5RGJsbDBwaERV?=
 =?utf-8?B?THc9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 95fd33ee-9e06-44b9-912d-08dada244e87
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Dec 2022 20:31:10.0031
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hf71/CUXE6NScStoTxCURqcQaeYR+w2zMkbs41povS0A9n+PjlCXH9dRhJ7BfnYu
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR15MB3361
X-Proofpoint-GUID: mz7SmBgxeGU7_U8ae2Wf043NgQWK_G4g
X-Proofpoint-ORIG-GUID: mz7SmBgxeGU7_U8ae2Wf043NgQWK_G4g
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-12-09_11,2022-12-08_01,2022-06-22_01
X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 12/9/22 7:20 AM, Jiri Olsa wrote:
> On Fri, Dec 09, 2022 at 02:50:55PM +0100, Jiri Olsa wrote:
>> On Fri, Dec 09, 2022 at 12:22:37PM +0100, Jiri Olsa wrote:
>>
>> SBIP
>>
>>>>>>>>>
>>>>>>>>> I'm trying to understand the severity of the issues and
>>>>>>>>> whether we need to revert that commit asap since the merge window
>>>>>>>>> is about to start.
>>>>>>>>
>>>>>>>> Jiri, Peter,
>>>>>>>>
>>>>>>>> ping.
>>>>>>>>
>>>>>>>> cc-ing Thorsten, since he's tracking it now.
>>>>>>>>
>>>>>>>> The config has CONFIG_X86_KERNEL_IBT=y.
>>>>>>>> Is it related?
>>>>>>>
>>>>>>> sorry for late reply.. I still did not find the reason,
>>>>>>> but I did not try with IBT yet, will test now
>>>>>>
>>>>>> no difference with IBT enabled, can't reproduce the issue
>>>>>>
>>>>>
>>>>> ok, scratch that.. the reproducer got stuck on wifi init :-\
>>>>>
>>>>> after I fix that I can now reproduce on my local config with
>>>>> IBT enabled or disabled.. it's something else
>>>>
>>>> I'm getting the error also when reverting the static call change,
>>>> looking for good commit, bisecting
>>>>
>>>> I'm getting fail with:
>>>>     f0c4d9fc9cc9 (tag: v6.1-rc4) Linux 6.1-rc4
>>>>
>>>> v6.1-rc1 is ok
>>>
>>> so far I narrowed it down between rc1 and rc3.. bisect got me nowhere so far
>>>
>>> attaching some more logs
>>
>> looking at the code.. how do we ensure that code running through
>> bpf_prog_run_xdp will not get dispatcher image changed while
>> it's being exetuted
>>
>> we use 'the other half' of the image when we add/remove programs,
>> but could bpf_dispatcher_update race with bpf_prog_run_xdp like:
>>
>>
>> cpu 0:                                  cpu 1:
>>
>> bpf_prog_run_xdp
>>     ...
>>     bpf_dispatcher_xdp_func
>>        start exec image at offset 0x0
>>
>>                                          bpf_dispatcher_update
>>                                                  update image at offset 0x800
>>                                          bpf_dispatcher_update
>>                                                  update image at offset 0x0
>>
>>        still in image at offset 0x0
>>
>>
>> that might explain why I wasn't able to trigger that on
>> bare metal just in qemu
> 
> I tried patch below and it fixes the issue for me and seems
> to confirm the race above.. but not sure it's the best fix
> 
> jirka
> 
> 
> ---
> diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c
> index c19719f48ce0..6a2ced102fc7 100644
> --- a/kernel/bpf/dispatcher.c
> +++ b/kernel/bpf/dispatcher.c
> @@ -124,6 +124,7 @@ static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs)
>   	}
>   
>   	__BPF_DISPATCHER_UPDATE(d, new ?: (void *)&bpf_dispatcher_nop_func);
> +	synchronize_rcu_tasks();
>   
>   	if (new)
>   		d->image_off = noff;

This might work. In arch/x86/kernel/alternative.c, we have following
code and comments. For text_poke, synchronize_rcu_tasks() might be able
to avoid concurrent execution and update.

/**
  * text_poke_copy - Copy instructions into (an unused part of) RX memory
  * @addr: address to modify
  * @opcode: source of the copy
  * @len: length to copy, could be more than 2x PAGE_SIZE
  *
  * Not safe against concurrent execution; useful for JITs to dump
  * new code blocks into unused regions of RX memory. Can be used in
  * conjunction with synchronize_rcu_tasks() to wait for existing
  * execution to quiesce after having made sure no existing functions
  * pointers are live.
  */
void *text_poke_copy(void *addr, const void *opcode, size_t len)
{
         unsigned long start = (unsigned long)addr;
         size_t patched = 0;

         if (WARN_ON_ONCE(core_kernel_text(start)))
                 return NULL;

         mutex_lock(&text_mutex);
         while (patched < len) {
                 unsigned long ptr = start + patched;
                 size_t s;

                 s = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(ptr), 
len - patched);

                 __text_poke(text_poke_memcpy, (void *)ptr, opcode + 
patched, s);
                 patched += s;
         }
         mutex_unlock(&text_mutex);
         return addr;
}