Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 4 Apr 2019 22:22:57 +0300
From:   Dmitrii Tcvetkov <demfloro@demfloro.ru>
To:     Paolo Valente <paolo.valente@linaro.org>
Cc:     Jens Axboe <axboe@kernel.dk>,
        linux-block <linux-block@vger.kernel.org>,
        linux-kernel@vger.kernel.org
Subject: Re: Bisected GFP in bfq_bfqq_expire on v5.1-rc1
Message-ID: <20190404222257.0cfb1130@fire.localdomain>
In-Reply-To: <EBC090F1-72C0-4794-91CC-37F9A2A1DEFB@linaro.org>
References: <20190329160227.7d55c8dd@fire.localdomain>
        <0e203a26-b941-cef4-dff1-013999d4b041@kernel.dk>
        <626EAE58-63C1-4ABA-9040-9D9A61F74A0D@linaro.org>
        <20190401115509.76310e03@fire.localdomain>
        <84B0CA50-0ED8-4171-8007-19EA43951735@linaro.org>
        <20190401122233.3e861312@fire.localdomain>
        <EBC090F1-72C0-4794-91CC-37F9A2A1DEFB@linaro.org>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MP_/IJ.6CVMsu2MF7LCxrnNrmoU"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

--MP_/IJ.6CVMsu2MF7LCxrnNrmoU
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On Mon, 1 Apr 2019 12:35:11 +0200
Paolo Valente <paolo.valente@linaro.org> wrote:

> 
> 
> > Il giorno 1 apr 2019, alle ore 11:22, Dmitrii Tcvetkov
> > <demfloro@demfloro.ru> ha scritto:
> > 
> > On Mon, 1 Apr 2019 11:01:27 +0200
> > Paolo Valente <paolo.valente@linaro.org> wrote:
> >> Ok, thank you. Could you please do a
> >> 
> >> list *(bfq_bfqq_expire+0x1f3)
> >> 
> >> for me?
> >> 
> >> Thanks,
> >> Paolo
> >> 
> >>> 
> >>> <gpf.txt><gpf-w-bfq-group-iosched.txt><config.txt>
> > 
> > Reading symbols from vmlinux...done.
> > (gdb) list *(bfq_bfqq_expire+0x1f3)
> > 0xffffffff813d02c3 is in bfq_bfqq_expire (block/bfq-iosched.c:3390).
> > 3385             * even in case bfqq and thus parent entities go on
> > receiving 3386             * service with the same budget.
> > 3387             */
> > 3388            entity = entity->parent;
> > 3389            for_each_entity(entity)
> > 3390                    entity->service = 0;
> > 3391    }
> > 3392
> > 3393    /*
> > 3394     * Budget timeout is not implemented through a dedicated
> > timer, but
> 
> Thank you very much.  Unfortunately this doesn't ring any bell.  I'm
> trying to reproduce the failure.  It will probably take a little
> time.  If I don't make it, I'll ask you to kindly retry after applying
> some instrumentation patch.
> 

I looked at what git is doing just before panic and it's doing a lot of
lstat() syscalls on working tree.

I've attached a python script which reproduces the crash in about
10 seconds after it prepares testdir, git checkout origin/linux-5.0.y
reproduces it in about 2 seconds. I have to use multiprocessing Pool as
I couldn't reproduce the crash using ThreadPool, probably due to Python
GIL.


--MP_/IJ.6CVMsu2MF7LCxrnNrmoU
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=crash.py

#!/usr/bin/env python3

from glob import glob
from os import lstat,mkdir
from random import randint
from os.path import isdir,exists
from pathlib import Path
from time import sleep
from subprocess import run
from multiprocessing import Pool

def drop_caches():
    with open('/proc/sys/vm/drop_caches','w') as f:
        f.write('3')

def enable_bfq():
    with open('/sys/block/sda/queue/scheduler','w') as f:
        f.write('bfq')

def sync():
    run(('sync'))

def prepare_tree(name):
    def populate(dir, depth=6):
        if not depth:
            return
        for fname in range(1,20):
            if randint(0,100) > 80:
                dirname = "{}{}/".format(dir,fname)
                mkdir(dirname)
                populate(dirname, depth - 1)
                continue
            fname = "{}{}".format(dir, fname)
            Path(fname).touch(exist_ok=True)
            
    if not isdir(name):
        mkdir(name)
    if not name.endswith('/'):
        name = '{}/'.format(name)
    populate(name)
    
        
def traverse(dir):
    drop_caches()
    for inode in glob("{}/*".format(dir)):
        if isdir(inode):
            traverse(inode)
        else:
            lstat(inode)
            if randint(0,10) > 6:
                sleep(0)

def main():
    nproc = 16
    dirname = 'testdir'
    if not exists(dirname):
        prepare_tree(dirname)
    sync()
    drop_caches()
    enable_bfq()
    drop_caches()

    with Pool(nproc) as pool:
        dirs = (dirname,) * nproc
        pool.map(traverse,dirs)
            
main()

--MP_/IJ.6CVMsu2MF7LCxrnNrmoU--