Received: by 2002:ac0:bc90:0:0:0:0:0 with SMTP id a16csp568563img; Wed, 20 Mar 2019 06:34:59 -0700 (PDT) X-Google-Smtp-Source: APXvYqyfTCUeIL+kjvPbtBjOdQ+yk0rW6hsbjwh5Es/IZUsR0c4z7Cp7BDlRK7+zJa3NQSjVbrxy X-Received: by 2002:a17:902:7e0f:: with SMTP id b15mr7741651plm.124.1553088899748; Wed, 20 Mar 2019 06:34:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553088899; cv=none; d=google.com; s=arc-20160816; b=ejnNw10ycX7guCBWCdzid2ejWcYpREg3wLBNXj6sIkmDNLeEss1OJNcDOn/FhYzuLB 9OrDUKvaVlnOoW1+jZ2p72h+BwlLGuR5J/4GQ5EpnexSI14W1cFlTPMFh431D8SA05h2 VBu25WAUYOa+7ug9KNiyo6TuY7a90IBOd/CIv4waA8DvFsvgxKGI6l2DN6YjL7h6xCgV RtGAQO3spKMeBUMwW0heO/FCQsKge0avKZgSxaiwgt0Si+B5O82/q3Q3sS3/Aznzn2eN FYUvlAW7nxB5wn8i6vdNsgXk1Bs3G4pYookrD3QJpxR0RT8aj4vPPh7Nx3c8ZxxdeFEJ Xe+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:subject; bh=y93P8V0JEbhy51autrO0vk3y/+ZtbIz69pKtJ+mRUNo=; b=njJ9/kzudAZ1RQv0Rq8qjNtYlFgWoWMDu5pAM3rwhblVnfzjugCGS+fW/IY0utxKhC 2lUOsshYFVkE8HaSlYfpOcJC5B/2Rv6v595r1oFmOty7chN7sfkzfKN3bw5/AfCyeAwS +fzHKIMcCPvQUN0xRxL9IhpZ6GjUHuys0lsOD7dcbobaiWsUhFLnOBNz3GNcjU5gEpGo wVQcjOa+xisDygrCXFbwSS2NYpBaFqku/BB7w5caNFgSMx1kO4g3u6XVXtP36AlPiic6 jRfzxFJCh1y/OA6Z/LEnoHgqAbv3EsB7NvSePA1WuA78NuKY0hCDpnoC/VEh3Y8YBwZM saYw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=virtuozzo.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k7si595602pgi.451.2019.03.20.06.34.43; Wed, 20 Mar 2019 06:34:59 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=virtuozzo.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727537AbfCTNeH (ORCPT + 99 others); Wed, 20 Mar 2019 09:34:07 -0400 Received: from relay.sw.ru ([185.231.240.75]:59890 "EHLO relay.sw.ru" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726065AbfCTNeG (ORCPT ); Wed, 20 Mar 2019 09:34:06 -0400 Received: from [172.16.25.12] by relay.sw.ru with esmtp (Exim 4.91) (envelope-from ) id 1h6bLg-0000UM-PB; Wed, 20 Mar 2019 16:33:40 +0300 Subject: Re: kernel panic: corrupted stack end in wb_workfn To: Dmitry Vyukov , Tetsuo Handa Cc: syzbot , Andrew Morton , Qian Cai , David Miller , guro@fb.com, Johannes Weiner , Josef Bacik , Kirill Tkhai , LKML , Linux-MM , linux-sctp@vger.kernel.org, Mel Gorman , Michal Hocko , netdev , Neil Horman , Shakeel Butt , syzkaller-bugs , Al Viro , Vladislav Yasevich , Matthew Wilcox , Xin Long References: <000000000000db3d130584506672@google.com> <426293c3-bf63-88ad-06fb-83927ab0d7c0@I-love.SAKURA.ne.jp> From: Andrey Ryabinin Message-ID: <315c8ff3-fd03-f2ca-c546-ca7dc5c14669@virtuozzo.com> Date: Wed, 20 Mar 2019 16:34:11 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 3/20/19 1:38 PM, Dmitry Vyukov wrote: > On Wed, Mar 20, 2019 at 11:24 AM Tetsuo Handa > wrote: >> >> On 2019/03/20 18:59, Dmitry Vyukov wrote: >>>> From bisection log: >>>> >>>> testing release v4.17 >>>> testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0 >>>> run #0: crashed: kernel panic: corrupted stack end in wb_workfn >>>> run #1: crashed: kernel panic: corrupted stack end in worker_thread >>>> run #2: crashed: kernel panic: Out of memory and no killable processes... >>>> run #3: crashed: kernel panic: corrupted stack end in wb_workfn >>>> run #4: crashed: kernel panic: corrupted stack end in wb_workfn >>>> run #5: crashed: kernel panic: corrupted stack end in wb_workfn >>>> run #6: crashed: kernel panic: corrupted stack end in wb_workfn >>>> run #7: crashed: kernel panic: corrupted stack end in wb_workfn >>>> run #8: crashed: kernel panic: Out of memory and no killable processes... >>>> run #9: crashed: kernel panic: corrupted stack end in wb_workfn >>>> testing release v4.16 >>>> testing commit 0adb32858b0bddf4ada5f364a84ed60b196dbcda with gcc (GCC) 8.1.0 >>>> run #0: OK >>>> run #1: OK >>>> run #2: OK >>>> run #3: OK >>>> run #4: OK >>>> run #5: crashed: kernel panic: Out of memory and no killable processes... >>>> run #6: OK >>>> run #7: crashed: kernel panic: Out of memory and no killable processes... >>>> run #8: OK >>>> run #9: OK >>>> testing release v4.15 >>>> testing commit d8a5b80568a9cb66810e75b182018e9edb68e8ff with gcc (GCC) 8.1.0 >>>> all runs: OK >>>> # git bisect start v4.16 v4.15 >>>> >>>> Why bisect started between 4.16 4.15 instead of 4.17 4.16? >>> >>> Because 4.16 was still crashing and 4.15 was not crashing. 4.15..4.16 >>> looks like the right range, no? >> >> No, syzbot should bisect between 4.16 and 4.17 regarding this bug, for >> "Stack corruption" can't manifest as "Out of memory and no killable processes". >> >> "kernel panic: Out of memory and no killable processes..." is completely >> unrelated to "kernel panic: corrupted stack end in wb_workfn". > > > Do you think this predicate is possible to code? Something like bellow probably would work better than current behavior. For starters, is_duplicates() might just compare 'crash' title with 'target_crash' title and its duplicates titles. syzbot has some knowledge about duplicates with different crash titles when people use "syz dup" command. Also it might be worth to experiment with using neural networks to identify duplicates. target_crash = 'kernel panic: corrupted stack end in wb_workfn' test commit: bad = false; skip = true; foreach run: run_started, crashed, crash := run_repro(); //kernel built, booted, reproducer launched successfully if (run_started) skip = false; if (crashed && is_duplicates(crash, target_crash)) bad = true; if (skip) git bisect skip; else if (bad) git bisect bad; else git bisect good;