Received: by 2002:a05:6a10:9848:0:0:0:0 with SMTP id x8csp3839910pxf; Mon, 15 Mar 2021 21:54:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzj18qgfiRVhpP+GopF/CzF6MEvFzDETvSCQNbU7aVymsuPa+Iayf4vfC0B/wUH2nepTMYE X-Received: by 2002:a05:6402:254f:: with SMTP id l15mr33833049edb.189.1615870475964; Mon, 15 Mar 2021 21:54:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1615870475; cv=none; d=google.com; s=arc-20160816; b=UrUmZYVT9AwRrfxNEa7lAnGHNYPhDOUzUwNfKG773/6E/5UwCnR+CoIgmC2Hkt/2Bf 9YgOI60CxBtth3DJ7ZGEU+3goyhNPGEAX35p9NXzGWoGvz3/7Yg3jL7srkn8+7z2NCqc epxknJnoeIyyqdmNTivjD+fC/CIeJ/Dyc3jq7NBMkGGNygbJ49xdEP2dI4/TPc3+g/iB 9zz5mlXDsT1E9I5niHFTMBTPEpmOyZz8YaATzVOmoOSBRptG6AppSFRblLUWff6Yc2Me IEHSM/xnNbfd0j+pWWCB37ELIFw+vVqXuXvPNoStAugkr2rkqyVnRCjm6BbXhjKtrzGl L9rQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:organization :from:references:cc:to:subject; bh=ngR5v0/fzzYbv068acA9tj4l+ouWezeaK6WzMmSMu+A=; b=1CyR/m3VzyivF4Mbq/gckxKh/nHF5nWVuVsKyKy9uAHJDOiFs1r9A2vkLOs9FKOYEa gMrHUbByBJBFQwkDSZdpo+ddYhNRbBsDW7liusbnBRrWALCarIo2u9KoJ5v5+kwK6M15 YoDFGF/oI9y+Wk1Ajo+ps9GSn/LpkIKElRZ1hbNwClVHTGe+uOkyD/g8pBS/YuiWi7gH lcWZ302PhMZFqa16iWVmvJJewAWcrgEvfHydyOVHIZ7Ye3rzmU5KMeL6Ay/CYy2y5wnw vTgrwI3u5SgFAtiQujtMWj0jvEeaIMW8nNZe7ldvR94XxOLDQPU0gnL9HGTzS5l6kmCq NxUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=codethink.co.uk Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id d21si12589293ejw.331.2021.03.15.21.54.13; Mon, 15 Mar 2021 21:54:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=codethink.co.uk Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233099AbhCOVjO (ORCPT + 99 others); Mon, 15 Mar 2021 17:39:14 -0400 Received: from imap2.colo.codethink.co.uk ([78.40.148.184]:50686 "EHLO imap2.colo.codethink.co.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232201AbhCOVjA (ORCPT ); Mon, 15 Mar 2021 17:39:00 -0400 Received: from cpc79921-stkp12-2-0-cust288.10-2.cable.virginm.net ([86.16.139.33] helo=[192.168.0.18]) by imap2.colo.codethink.co.uk with esmtpsa (Exim 4.92 #3 (Debian)) id 1lLuvB-0000qR-99; Mon, 15 Mar 2021 21:38:41 +0000 Subject: Re: [syzbot] BUG: unable to handle kernel access to user memory in schedule_tail To: Dmitry Vyukov Cc: Alex Ghiti , syzbot , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv , Daniel Bristot de Oliveira , Benjamin Segall , dietmar.eggemann@arm.com, Juri Lelli , LKML , Mel Gorman , Ingo Molnar , Peter Zijlstra , Steven Rostedt , syzkaller-bugs , Vincent Guittot References: <000000000000b74f1b05bd316729@google.com> <84b0471d-42c1-175f-ae1d-a18c310c7f77@codethink.co.uk> <795597a1-ec87-e09e-d073-3daf10422abb@ghiti.fr> From: Ben Dooks Organization: Codethink Limited. Message-ID: <12d4137e-6c14-bc41-4bbc-955ce46198d2@codethink.co.uk> Date: Mon, 15 Mar 2021 21:38:39 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 13/03/2021 07:20, Dmitry Vyukov wrote: > On Fri, Mar 12, 2021 at 9:12 PM Ben Dooks wrote: >> >> On 12/03/2021 16:25, Alex Ghiti wrote: >>> >>> >>> Le 3/12/21 à 10:12 AM, Dmitry Vyukov a écrit : >>>> On Fri, Mar 12, 2021 at 2:50 PM Ben Dooks >>>> wrote: >>>>> >>>>> On 10/03/2021 17:16, Dmitry Vyukov wrote: >>>>>> On Wed, Mar 10, 2021 at 5:46 PM syzbot >>>>>> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> syzbot found the following issue on: >>>>>>> >>>>>>> HEAD commit: 0d7588ab riscv: process: Fix no prototype for >>>>>>> arch_dup_tas.. >>>>>>> git tree: >>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes >>>>>>> console output: >>>>>>> https://syzkaller.appspot.com/x/log.txt?x=1212c6e6d00000 >>>>>>> kernel config: >>>>>>> https://syzkaller.appspot.com/x/.config?x=e3c595255fb2d136 >>>>>>> dashboard link: >>>>>>> https://syzkaller.appspot.com/bug?extid=e74b94fe601ab9552d69 >>>>>>> userspace arch: riscv64 >>>>>>> >>>>>>> Unfortunately, I don't have any reproducer for this issue yet. >>>>>>> >>>>>>> IMPORTANT: if you fix the issue, please add the following tag to >>>>>>> the commit: >>>>>>> Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com >>>>>> >>>>>> +riscv maintainers >>>>>> >>>>>> This is riscv64-specific. >>>>>> I've seen similar crashes in put_user in other places. It looks like >>>>>> put_user crashes in the user address is not mapped/protected (?). >>>>> >>>>> I've been having a look, and this seems to be down to access of the >>>>> tsk->set_child_tid variable. I assume the fuzzing here is to pass a >>>>> bad address to clone? >>>>> >>>>> From looking at the code, the put_user() code should have set the >>>>> relevant SR_SUM bit (the value for this, which is 1<<18 is in the >>>>> s2 register in the crash report) and from looking at the compiler >>>>> output from my gcc-10, the code looks to be dong the relevant csrs >>>>> and then csrc around the put_user >>>>> >>>>> So currently I do not understand how the above could have happened >>>>> over than something re-tried the code seqeunce and ended up retrying >>>>> the faulting instruction without the SR_SUM bit set. >>>> >>>> I would maybe blame qemu for randomly resetting SR_SUM, but it's >>>> strange that 99% of these crashes are in schedule_tail. If it would be >>>> qemu, then they would be more evenly distributed... >>>> >>>> Another observation: looking at a dozen of crash logs, in none of >>>> these cases fuzzer was actually trying to fuzz clone with some insane >>>> arguments. So it looks like completely normal clone's (e..g coming >>>> from pthread_create) result in this crash. >>>> >>>> I also wonder why there is ret_from_exception, is it normal? I see >>>> handle_exception disables SR_SUM: >>> >>> csrrc does the right thing: it cleans SR_SUM bit in status but saves the >>> previous value that will get correctly restored. >>> >>> ("The CSRRC (Atomic Read and Clear Bits in CSR) instruction reads the >>> value of the CSR, zero-extends the value to XLEN bits, and writes it to >>> integer registerrd. The initial value in integerregisterrs1is treated >>> as a bit mask that specifies bit positions to be cleared in the CSR. Any >>> bitthat is high inrs1will cause the corresponding bit to be cleared in >>> the CSR, if that CSR bit iswritable. Other bits in the CSR are >>> unaffected.") >> >> I think there may also be an understanding issue on what the SR_SUM >> bit does. I thought if it is set, M->U accesses would fault, which is >> why it gets set early on. But from reading the uaccess code it looks >> like the uaccess code sets it on entry and then clears on exit. >> >> I am very confused. Is there a master reference for rv64? >> >> https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.pdf >> seems to state PUM is the SR_SUM bit, and that (if set) disabled >> >> Quote: >> The PUM (Protect User Memory) bit modifies the privilege with which >> S-mode loads, stores, and instruction fetches access virtual memory. >> When PUM=0, translation and protection behave as normal. When PUM=1, >> S-mode memory accesses to pages that are accessible by U-mode (U=1 in >> Figure 4.19) will fault. PUM has no effect when executing in U-mode >> >> >>>> https://elixir.bootlin.com/linux/v5.12-rc2/source/arch/riscv/kernel/entry.S#L73 >>>> >>> >>> Still no luck for the moment, can't reproduce it locally, my test is >>> maybe not that good (I created threads all day long in order to trigger >>> the put_user of schedule_tail). >> >> It may of course depend on memory and other stuff. I did try to see if >> it was possible to clone() with the child_tid address being a valid but >> not mapped page... >> >>> Given that the path you mention works most of the time, and that the >>> status register in the stack trace shows the SUM bit is not set whereas >>> it is set in put_user, I'm leaning toward some race condition (maybe an >>> interrupt that arrives at the "wrong" time) or a qemu issue as you >>> mentioned. >> >> I suppose this is possible. From what I read it should get to the >> point of being there with the SUM flag cleared, so either something >> went wrong in trying to fix the instruction up or there's some other >> error we're missing. >> >>> To eliminate qemu issues, do you have access to some HW ? Or to >>> different qemu versions ? >> >> I do have access to a Microchip Polarfire board. I just need the >> instructions on how to setup the test-code to make it work on the >> hardware. > > For full syzkaller support, it would need to know how to reboot these > boards and get access to the console. > syzkaller has a stop-gap VM backend which just uses ssh to a physical > machine and expects the kernel to reboot on its own after any crashes. > > But I actually managed to reproduce it in an even simpler setup. > Assuming you have Go 1.15 and riscv64 cross-compiler gcc installed > > $ go get -u -d github.com/google/syzkaller/... > $ cd $GOPATH/src/github.com/google/syzkaller > $ make stress executor TARGETARCH=riscv64 > $ scp bin/linux_riscv64/syz-execprog bin/linux_riscv64/syz-executor > your_machine:/ > > Then run ./syz-stress on the machine. > On the first run it crashed it with some other bug, on the second run > I got the crash in schedule_tail. > With qemu tcg I also added -slowdown=10 flag to syz-stress to scale > all timeouts, if native execution is faster, then you don't need it. Ok, not sure what's going on. I get a lot of errors similar to: > > 2021/03/15 21:35:20 transitively unsupported: ioctl$SNAPSHOT_CREATE_IMAGE: no syscalls can create resource fd_snapshot, enable some syscalls that can create it [openat$snapshot] Followed by: > 2021/03/15 21:35:48 executed 0 programs > 2021/03/15 21:35:48 failed to create execution environment: failed to mmap shm file: invalid argument The qemu is 5.2.0 and root is Debian/unstable riscv64 (same as chroot used to build the syz tools) -- Ben Dooks http://www.codethink.co.uk/ Senior Engineer Codethink - Providing Genius https://www.codethink.co.uk/privacy.html