Received: by 2002:ac0:bc90:0:0:0:0:0 with SMTP id a16csp428390img; Wed, 20 Mar 2019 03:43:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqz05mnKhP7vbLVh+gRi33bfaUHn1q51RrHrIe1zWdyCujV8uT+H615Oy8cTJgSnhE4+bPax X-Received: by 2002:a65:5184:: with SMTP id h4mr6738000pgq.32.1553078622726; Wed, 20 Mar 2019 03:43:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553078622; cv=none; d=google.com; s=arc-20160816; b=kg9WHG0evZcVHCXMBnXWFYMRO1HAiQydC8c+G3V5y4QOzaZ87a1OnTJBXr8hCxo/0K GZJFAl7EPOXVGKnKjg1O3r/4aXdB5nIj1l+uqOEM2tuLJewQx11n6AsMo9QcliAZ9P2P TlnZt6fxf88DivXEAdbBgCCkRq8AwMxc1o7xZkHhqkd9p7Y14jsNSAFXvolE+9d2UFaM abLx8d2T5B0ERgoS62NT2jxosbzlRMffQpaSO0xc29mnx6D3ESXcvHuRAUVvDYSsjV5e ViaFezhj97MyV4d72Wt9tSnj5bD2D3HQJa7re+exY0/RGe7Vho+VDlGuJ4CasEq6nDiR GlHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=hcRJyjpf1Os+dJaolVe49RgmT+5cszRgDCVwAxJ0n0o=; b=uCkRFD1ld6fNkcZcVAr7qFQgslTe4aFyukp+xofiMhZI3ERDHJuY2DX2gvcSTc/+mL IElsiNH/CkS8sht4GM1vtPian92gTDKYuzMnQ+86uT1wWDA2fAUUz7LjljELHTWqPoC5 EDGBCmdrmCdwtIjJMgGdxOcl7/qk8DHWHAY4nETpjbp+fjD3tfq82oEwDiOGJD/8zRu7 7x8tqlPESEsI2ad2G9I4ZEs9MMd0Nuf354IDI8MmprMeM8WfrpviWQg1GVBD5fCQahgt vd6fW1laHkSX6/wkLhdj6HG71sH2JbxCcBQZBOFn4563ucJZ/xjIKnQvokSP7djspbSr eYGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="dCn/aejq"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 124si1425850pfw.148.2019.03.20.03.43.27; Wed, 20 Mar 2019 03:43:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="dCn/aejq"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726981AbfCTKmt (ORCPT + 99 others); Wed, 20 Mar 2019 06:42:49 -0400 Received: from mail-io1-f66.google.com ([209.85.166.66]:42744 "EHLO mail-io1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727215AbfCTKms (ORCPT ); Wed, 20 Mar 2019 06:42:48 -0400 Received: by mail-io1-f66.google.com with SMTP id c4so1480801ioh.9 for ; Wed, 20 Mar 2019 03:42:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hcRJyjpf1Os+dJaolVe49RgmT+5cszRgDCVwAxJ0n0o=; b=dCn/aejqzrOp2WKxTwq/E2q4ZQe5ezVUloPkEz/OKE/qjj1mOYEqYUmimO810pIJ+U RIF9iElA5ZHfbnOZM8VkouD8O4dDbFRtiqz/FYb+Dgg4XN27hOJOQTcuWabryiYXrcCw mHShwR4k8ruf6hD6aJk6HxpZBKoWmmN+5Myg9rR5x0ZBEHYD+gvxN95MIC4SLvTnefzU liUpF4/YunuRUFelrz9xfIaZxT3AMlEaGvPtzAZj1trQzA0F0lWmreA80Mmm+omZvGar 0WmQYvG5ZEuPtACYumZ36jDhhRZOoQwpcMAp+ZPPCvXtIPvUhB0cTQtQzWzMUkWVrMgp 1JLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hcRJyjpf1Os+dJaolVe49RgmT+5cszRgDCVwAxJ0n0o=; b=BK4ve+kzR9AKoHydryeB+6qOd8g9gu1hCsu55RmdiEBupZHPYRoIeHwsI2NdRkKuYF kaG+G7cjfPaTGVBif9PnxlnnQuNSfWfYYglBVmar2mtJiWzKlUJWq5wjtArK4Kt97WT0 mMCXW+Ed1k8ps1JoEP4jhud3WVdsUbERA/Amjwu1gIaoUY13nEu8xwwaUjZRZ94L4ohc 1d5eAFpQbWQfIN8vIE95n3TBW0/d7ZRFu1IGbVHhfESULQadgFiUpqiciNfNmkKv6tA6 Ym3PnYoLHFO2fh34xUG6kcO86FjJl+rFZXkhcSI6WMNQuABempcTJGWcwywNtYlTw8IG irNw== X-Gm-Message-State: APjAAAUOXFeloUDFTyl6M41gbPqFEyiT9S9Q5/UKTIPoTfJcw93nPP/f 0XBm4NyxH5Ud80e549Mer+QPw7Wal6NUqWn2kLg/ew== X-Received: by 2002:a5d:834a:: with SMTP id q10mr4426278ior.271.1553078567797; Wed, 20 Mar 2019 03:42:47 -0700 (PDT) MIME-Version: 1.0 References: <000000000000db3d130584506672@google.com> <426293c3-bf63-88ad-06fb-83927ab0d7c0@I-love.SAKURA.ne.jp> In-Reply-To: From: Dmitry Vyukov Date: Wed, 20 Mar 2019 11:42:36 +0100 Message-ID: Subject: Re: kernel panic: corrupted stack end in wb_workfn To: Tetsuo Handa Cc: Andrey Ryabinin , syzbot , Andrew Morton , Qian Cai , David Miller , guro@fb.com, Johannes Weiner , Josef Bacik , Kirill Tkhai , LKML , Linux-MM , linux-sctp@vger.kernel.org, Mel Gorman , Michal Hocko , netdev , Neil Horman , Shakeel Butt , syzkaller-bugs , Al Viro , Vladislav Yasevich , Matthew Wilcox , Xin Long Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Mar 20, 2019 at 11:38 AM Dmitry Vyukov wrote: > > On Wed, Mar 20, 2019 at 11:24 AM Tetsuo Handa > wrote: > > > > On 2019/03/20 18:59, Dmitry Vyukov wrote: > > >> From bisection log: > > >> > > >> testing release v4.17 > > >> testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0 > > >> run #0: crashed: kernel panic: corrupted stack end in wb_workfn > > >> run #1: crashed: kernel panic: corrupted stack end in worker_thread > > >> run #2: crashed: kernel panic: Out of memory and no killable processes... > > >> run #3: crashed: kernel panic: corrupted stack end in wb_workfn > > >> run #4: crashed: kernel panic: corrupted stack end in wb_workfn > > >> run #5: crashed: kernel panic: corrupted stack end in wb_workfn > > >> run #6: crashed: kernel panic: corrupted stack end in wb_workfn > > >> run #7: crashed: kernel panic: corrupted stack end in wb_workfn > > >> run #8: crashed: kernel panic: Out of memory and no killable processes... > > >> run #9: crashed: kernel panic: corrupted stack end in wb_workfn > > >> testing release v4.16 > > >> testing commit 0adb32858b0bddf4ada5f364a84ed60b196dbcda with gcc (GCC) 8.1.0 > > >> run #0: OK > > >> run #1: OK > > >> run #2: OK > > >> run #3: OK > > >> run #4: OK > > >> run #5: crashed: kernel panic: Out of memory and no killable processes... > > >> run #6: OK > > >> run #7: crashed: kernel panic: Out of memory and no killable processes... > > >> run #8: OK > > >> run #9: OK > > >> testing release v4.15 > > >> testing commit d8a5b80568a9cb66810e75b182018e9edb68e8ff with gcc (GCC) 8.1.0 > > >> all runs: OK > > >> # git bisect start v4.16 v4.15 > > >> > > >> Why bisect started between 4.16 4.15 instead of 4.17 4.16? > > > > > > Because 4.16 was still crashing and 4.15 was not crashing. 4.15..4.16 > > > looks like the right range, no? > > > > No, syzbot should bisect between 4.16 and 4.17 regarding this bug, for > > "Stack corruption" can't manifest as "Out of memory and no killable processes". > > > > "kernel panic: Out of memory and no killable processes..." is completely > > unrelated to "kernel panic: corrupted stack end in wb_workfn". > > > Do you think this predicate is possible to code? Looking at the > examples we have, distinguishing different bugs does not look feasible > to me. If the predicate is not accurate, you just trade one set of > false positives to another set of false positives and then you at the > beginning of an infinite slippery slope refining it. > Also, if we see a different bug (assuming we can distinguish them), > does it mean that the original bug is not present? Or it's also > present, but we just hit the other one first? This also does not look > feasible to answer. And if you give a wrong answer, bisection goes the > wrong way and we are where we started. Just with more complex code and > things being even harder to explain to other people. > I mean, yes, I agree, kernel bug bisection won't be perfect. But do > you see anything actionable here? I see the larger long term bisection quality improvement (for syzbot and for everybody else) in doing some actual testing for each kernel commit before it's being merged into any kernel tree, so that we have less of these a single program triggers 3 different bugs, stray unrelated bugs, broken release boots, etc. I don't see how reliable bisection is possible without that.