Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <0000000000007523a60576e80a47@google.com> <CACT4Y+b3AmVQMjPNsPHOXRZS4tNYb6Z9h5-c=1ZwZk0VR-5J5Q@mail.gmail.com>
 <20180928070042.GF3439@hirez.programming.kicks-ass.net> <CACT4Y+YFmSmXjs5EMNRPvsR-mLYeAYKypBppYq_M_boTi8a9uQ@mail.gmail.com>
 <CACT4Y+ZBYYUiJejNbPcZWS+aHehvkgKkTKm0gvuviXGGcirJ5g@mail.gmail.com>
 <CACT4Y+bTGp1J9Wn=93LUObdTcWPo2JrChYKF-1v6aXmtvoQgPQ@mail.gmail.com>
 <CAM_iQpVtcNFeEtW15z_nZoyC1Q-_pCq+UfZ4vYBB3Lb2CMm4Mg@mail.gmail.com>
 <CACT4Y+YuM55YUT37jwRP163J7ha25cN03sZ5WqTUPkz3e43Ggw@mail.gmail.com> <CAMArcTXq=nV5kNSdWPgvpMmYXuet9gxZgmxO2JJJ7_T3vLRoRg@mail.gmail.com>
In-Reply-To: <CAMArcTXq=nV5kNSdWPgvpMmYXuet9gxZgmxO2JJJ7_T3vLRoRg@mail.gmail.com>
From:   Dmitry Vyukov <dvyukov@google.com>
Date:   Thu, 16 Jan 2020 18:05:26 +0100
Message-ID: <CACT4Y+Y5+wma84Gofcm86KvOKw2niybPBTwz8iHzYW-ndFHH8g@mail.gmail.com>
Subject: Re: BUG: MAX_LOCKDEP_CHAINS too low!
To:     Taehee Yoo <ap420073@gmail.com>
Cc:     Cong Wang <xiyou.wangcong@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>,
        syzbot <syzbot+aaa6fa4949cc5d9b7b25@syzkaller.appspotmail.com>,
        Ingo Molnar <mingo@kernel.org>,
        Will Deacon <will.deacon@arm.com>,
        LKML <linux-kernel@vger.kernel.org>,
        syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
        Thomas Gleixner <tglx@linutronix.de>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Jan 16, 2020 at 4:33 PM Taehee Yoo <ap420073@gmail.com> wrote:
>
> On Thu, 16 Jan 2020 at 14:25, Dmitry Vyukov <dvyukov@google.com> wrote:
> >
>
> Hi Dmitry!
>
> > On Wed, Jan 15, 2020 at 10:53 PM Cong Wang <xiyou.wangcong@gmail.com> wrote:
> > > > +Taehee, Cong,
> > > >
> > > > In the other thread Taehee mentioned the creation of dynamic keys for
> > > > net devices that was added recently and that they are subject to some
> > > > limits.
> > > > syzkaller creates lots of net devices for isolation (several dozens
> > > > per test process, but then these can be created and destroyed
> > > > periodically). I wonder if it's the root cause of the lockdep limits
> > > > problems?
> > >
> > > Very possibly. In current code base, there are 4 lockdep keys
> > > per netdev:
> > >
> > >         struct lock_class_key   qdisc_tx_busylock_key;
> > >         struct lock_class_key   qdisc_running_key;
> > >         struct lock_class_key   qdisc_xmit_lock_key;
> > >         struct lock_class_key   addr_list_lock_key;
> > >
> > > so the number of lockdep keys is at least 4x number of network
> > > devices.
> >
> > And these are not freed/reused, right? So with dynamic keys LOCKDEP
> > inherently can't handle prolonged running, only O(1) work?
>
> When netdev interface is removed, these dynamic keys are removed.
> But if so many netdev interfaces are existing concurrently
> (more than 2000), lockdep will stop because of a lack of keys.
> In addition, as far as I know, freeing dynamic lockdep key routine is
> slower than creating a new dynamic lockdep key. If there are so many
> netdev interface add/delete operations, for a while, there may be no
> available lockdep key. At this moment, lockdep will stop.
>
> Thank you
> Taehee Yoo


This issues mostly stalled all syzbot testing by now, these LOCKDEP
bugs are the only bugs that are being detected...

2020/01/16 15:30:47 vm-2: crash: BUG: MAX_LOCKDEP_ENTRIES too low!
2020/01/16 15:38:28 vm-3: crash: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
2020/01/16 15:39:55 vm-0: crash: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
2020/01/16 15:41:19 vm-6: crash: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
2020/01/16 15:41:35 vm-1: crash: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
2020/01/16 15:51:16 vm-2: crash: BUG: MAX_LOCKDEP_ENTRIES too low!
2020/01/16 15:57:52 vm-4: crash: BUG: MAX_LOCKDEP_ENTRIES too low!
2020/01/16 16:00:57 vm-6: crash: BUG: MAX_LOCKDEP_ENTRIES too low!
2020/01/16 16:36:35 vm-1: crash: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
2020/01/16 16:37:04 vm-3: crash: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!