LinuxLists.cc - BUG: MAX_LOCKDEP_CHAIN

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Tue, Jul 26, 2022, at 12:42 PM, David Sterba wrote:
> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>> Hi guys.
>> Always with intensive writing on a btrfs volume, the message "BUG:
>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>
> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> tends to work.

Fedora is using 17. I'll make a request to bump it to 18. Thanks.

--
Chris Murphy

2022-07-26 20:00:52

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Tue, Jul 26, 2022, at 3:19 PM, Chris Murphy wrote:
> On Tue, Jul 26, 2022, at 12:42 PM, David Sterba wrote:
>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>>> Hi guys.
>>> Always with intensive writing on a btrfs volume, the message "BUG:
>>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>
>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>> tends to work.
>
> Fedora is using 17. I'll make a request to bump it to 18. Thanks.

Should it be 18 across all archs? Or is it OK to only bump x86_64?

--
Chris Murphy

2022-07-26 21:00:59

by David Sterba

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Tue, Jul 26, 2022 at 03:21:32PM -0400, Chris Murphy wrote:
>
>
> On Tue, Jul 26, 2022, at 3:19 PM, Chris Murphy wrote:
> > On Tue, Jul 26, 2022, at 12:42 PM, David Sterba wrote:
> >> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> >>> Hi guys.
> >>> Always with intensive writing on a btrfs volume, the message "BUG:
> >>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> >>
> >> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> >> tends to work.
> >
> > Fedora is using 17. I'll make a request to bump it to 18. Thanks.
>
> Should it be 18 across all archs? Or is it OK to only bump x86_64?

I think it applies to all achritectures equally but I'm no lockdep
expert.

2022-08-03 19:34:58

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
>
> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > Hi guys.
> > Always with intensive writing on a btrfs volume, the message "BUG:
> > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>
> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> tends to work.

I confirm that after bumping LOCKDEP_CHAINS_BITS to 18 several days of
continuous writing on the BTRFS partition with different files with a
total size of 10Tb I didn't see this kernel bug message again.
Tetsuo, I saw your commit 5dc33592e95534dc8455ce3e9baaaf3dae0fff82 [1]
set for LOCKDEP_CHAINS_BITS default value 16.
Why not increase LOCKDEP_CHAINS_BITS to 18 by default?
Thanks.

[1] https://github.com/torvalds/linux/blame/master/lib/Kconfig.debug#L1387

--
Best Regards,
Mike Gavrilov.

2022-08-03 20:15:20

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Wed, Aug 3, 2022, at 3:28 PM, Mikhail Gavrilov wrote:
> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
>>
>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>> > Hi guys.
>> > Always with intensive writing on a btrfs volume, the message "BUG:
>> > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>
>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>> tends to work.
>
> I confirm that after bumping LOCKDEP_CHAINS_BITS to 18 several days of
> continuous writing on the BTRFS partition with different files with a
> total size of 10Tb I didn't see this kernel bug message again.
> Tetsuo, I saw your commit 5dc33592e95534dc8455ce3e9baaaf3dae0fff82 [1]
> set for LOCKDEP_CHAINS_BITS default value 16.
> Why not increase LOCKDEP_CHAINS_BITS to 18 by default?

This will be making it into Fedora debug kernels, which have lockdep enabled on them, starting with 5.20 series, which are now building in koji.
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921

--
Chris Murphy

2022-08-04 08:12:22

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Thu, Aug 4, 2022 at 1:01 AM Chris Murphy <[email protected]> wrote:
>
> This will be making it into Fedora debug kernels, which have lockdep enabled on them, starting with 5.20 series, which are now building in koji.
> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921

I saw this change, but it would be good if users of all other
distributions will be happy too.

--
Best Regards,
Mike Gavrilov.

2022-08-04 11:29:17

by Tetsuo Handa

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On 2022/08/04 16:35, Mikhail Gavrilov wrote:
> On Thu, Aug 4, 2022 at 1:01 AM Chris Murphy <[email protected]> wrote:
>>
>> This will be making it into Fedora debug kernels, which have lockdep enabled on them, starting with 5.20 series, which are now building in koji.
>> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
>
> I saw this change, but it would be good if users of all other
> distributions will be happy too.
>

I'm not a lockdep maintainer.

Please submit a patch to lockdep maintainers and persuade lockdep maintainers
to change the default value. ;-)

2023-01-24 20:28:09

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
>
> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > Hi guys.
> > Always with intensive writing on a btrfs volume, the message "BUG:
> > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>
> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> tends to work.

Hi,
Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.

❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
CONFIG_LOCKDEP_CHAINS_BITS=18

[88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
[88685.088124] turning off the locking correctness validator.
[88685.088133] Please attach the output of /proc/lock_stat to the bug report
[88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
[88685.088154] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022

What's next? Increase this value to 19?
Actual full kernel log and lock_stat attached below.

--
Best Regards,
Mike Gavrilov.

Attachments:

dmesg.tar.xz (35.89 kB)
lock_stat.tar.xz (94.65 kB)
Download all attachments

2023-01-25 17:21:07

by David Sterba

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> >
> > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > Hi guys.
> > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> >
> > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > tends to work.
>
> Hi,
> Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
>
> ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> CONFIG_LOCKDEP_CHAINS_BITS=18
>
> [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> [88685.088124] turning off the locking correctness validator.
> [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> [88685.088154] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
>
> What's next? Increase this value to 19?

Yes, though increasing the value is a workaround so you may see the
warning again.

2023-01-26 09:48:02

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
>
> On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> > >
> > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > Hi guys.
> > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > >
> > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > tends to work.
> >
> > Hi,
> > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> >
> > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > CONFIG_LOCKDEP_CHAINS_BITS=18
> >
> > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > [88685.088124] turning off the locking correctness validator.
> > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> > ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > [88685.088154] Hardware name: System manufacturer System Product
> > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> >
> > What's next? Increase this value to 19?
>
> Yes, though increasing the value is a workaround so you may see the
> warning again.

Is there any sense in this WARNING if we would ignore it and every
time increase the threshold value?
May Be set 99 right away? Or remove such a check condition?

--
Best Regards,
Mike Gavrilov.

2023-01-26 17:39:28

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Cc lock folks]

On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
> >
> > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> > > >
> > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > Hi guys.
> > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > >
> > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > tends to work.
> > >
> > > Hi,
> > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > >
> > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > >
> > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > [88685.088124] turning off the locking correctness validator.
> > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> > > ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > [88685.088154] Hardware name: System manufacturer System Product
> > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > >
> > > What's next? Increase this value to 19?
> >
> > Yes, though increasing the value is a workaround so you may see the
> > warning again.
>
> Is there any sense in this WARNING if we would ignore it and every
> time increase the threshold value?

Lockdep uses static allocated array to track lock holdings chains to
avoid dynmaic memory allocation in its own code. So if you see the
warning it means your test has more combination of lock holdings than
the array can record. In other words, you reach the resource limitation,
and in that sense it makes sense to just ignore it and increase the
value: you want to give lockdep enough resource to work, right?

> May Be set 99 right away? Or remove such a check condition?

That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
chains array..

However, a few other options we can try in lockdep are:

* warn but not turn off the lockdep: the lock holding chain is
only a cache for what lock holding combination lockdep has ever
see, we also record the dependency in the graph. Without the
lock holding chain, lockdep can still work but just slower.

* allow dynmaic memory allocation in lockdep: I think this might
be OK since we have lockdep_recursion to avoid lockdep code ->
mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
missing something. And even we allow it, the use of memory
doesn't change, you will still need that amout of memory to
track lock holding chains.

I'm not sure whether these options are better than just increasing the
number, maybe to unblock your ASAP, you can try make it 30 and make sure
you have large enough memory to test.

Regards,
Boqun

>
> --
> Best Regards,
> Mike Gavrilov.

2023-01-26 18:31:37

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On 1/26/23 12:38, Boqun Feng wrote:
> [Cc lock folks]
>
> On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
>> On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
>>> On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
>>>> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
>>>>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>>>>>> Hi guys.
>>>>>> Always with intensive writing on a btrfs volume, the message "BUG:
>>>>>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>>>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>>>>> tends to work.
>>>> Hi,
>>>> Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
>>>> low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
>>>>
>>>> ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
>>>> CONFIG_LOCKDEP_CHAINS_BITS=18
>>>>
>>>> [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
>>>> [88685.088124] turning off the locking correctness validator.
>>>> [88685.088133] Please attach the output of /proc/lock_stat to the bug report
>>>> [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
>>>> ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
>>>> [88685.088154] Hardware name: System manufacturer System Product
>>>> Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
>>>>
>>>> What's next? Increase this value to 19?
>>> Yes, though increasing the value is a workaround so you may see the
>>> warning again.
>> Is there any sense in this WARNING if we would ignore it and every
>> time increase the threshold value?
> Lockdep uses static allocated array to track lock holdings chains to
> avoid dynmaic memory allocation in its own code. So if you see the
> warning it means your test has more combination of lock holdings than
> the array can record. In other words, you reach the resource limitation,
> and in that sense it makes sense to just ignore it and increase the
> value: you want to give lockdep enough resource to work, right?
>
>> May Be set 99 right away? Or remove such a check condition?
> That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> chains array..

Note that every increment of LOCKDEP_CHAINS_BITS double the storage
space. With 99, that will likely exceed the total amount of memory you
have in your system.

Boqun, where does the 5 figure come from. It is just a simple u16 array
of size MAX_LOCKDEP_CHAIN_HLOCKS. The chain_hlocks array stores the lock
chains that show up in the lockdep splats and in the /proc/lockdep*
files. Each chain is variable size. As we add new lock into the chain,
we have to repeatedly deallocate and reallocate a larger chain buffer.
That will cause fragmentation in the chain_hlocks[]. So if we have a
very long lock chain, the allocation may fail because the largest free
block is smaller than the requested chain length. There may be enough
free space in chain_hlocks, but it is just too fragmented to be useful.

Maybe we should figure out a better way to handle this fragmentation. In
the mean time, the easiest way forward is just to increase the
LOCKDEP_CHAINS_BITS by 1.

>
> However, a few other options we can try in lockdep are:
>
> * warn but not turn off the lockdep: the lock holding chain is
> only a cache for what lock holding combination lockdep has ever
> see, we also record the dependency in the graph. Without the
> lock holding chain, lockdep can still work but just slower.
>
> * allow dynmaic memory allocation in lockdep: I think this might
> be OK since we have lockdep_recursion to avoid lockdep code ->
> mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> missing something. And even we allow it, the use of memory
> doesn't change, you will still need that amout of memory to
> track lock holding chains.

It is not just the issue of calling the memory allocator. There is also
the issue of copying data from old chain_hlocks to new one while the old
one may be updated during the copying process unless we can freeze
everything else.

Cheers,
Longman

2023-01-26 19:00:38

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Thu, Jan 26, 2023 at 01:30:34PM -0500, Waiman Long wrote:
> On 1/26/23 12:38, Boqun Feng wrote:
> > [Cc lock folks]
> >
> > On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
> > > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> > > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > > Hi guys.
> > > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > > tends to work.
> > > > > Hi,
> > > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > > >
> > > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > > >
> > > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > > [88685.088124] turning off the locking correctness validator.
> > > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> > > > > ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > > >
> > > > > What's next? Increase this value to 19?
> > > > Yes, though increasing the value is a workaround so you may see the
> > > > warning again.
> > > Is there any sense in this WARNING if we would ignore it and every
> > > time increase the threshold value?
> > Lockdep uses static allocated array to track lock holdings chains to
> > avoid dynmaic memory allocation in its own code. So if you see the
> > warning it means your test has more combination of lock holdings than
> > the array can record. In other words, you reach the resource limitation,
> > and in that sense it makes sense to just ignore it and increase the
> > value: you want to give lockdep enough resource to work, right?
> >
> > > May Be set 99 right away? Or remove such a check condition?
> > That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> > chains array..
>
> Note that every increment of LOCKDEP_CHAINS_BITS double the storage space.
> With 99, that will likely exceed the total amount of memory you have in your
> system.
>
> Boqun, where does the 5 figure come from. It is just a simple u16 array of

#define MAX_LOCKDEP_CHAINS_BITS CONFIG_LOCKDEP_CHAINS_BITS
#define MAX_LOCKDEP_CHAINS (1UL << MAX_LOCKDEP_CHAINS_BITS)

#define MAX_LOCKDEP_CHAIN_HLOCKS (MAX_LOCKDEP_CHAINS*5)

I think the last one means we think the average length of a lock chain
is 5, in other words, in average, a task hold at most 5 locks. I don't
know where the 5 came from either, but it's there ;-)

Regards,
Boqun

> size MAX_LOCKDEP_CHAIN_HLOCKS. The chain_hlocks array stores the lock chains
> that show up in the lockdep splats and in the /proc/lockdep* files. Each
> chain is variable size. As we add new lock into the chain, we have to
> repeatedly deallocate and reallocate a larger chain buffer. That will cause
> fragmentation in the chain_hlocks[]. So if we have a very long lock chain,
> the allocation may fail because the largest free block is smaller than the
> requested chain length. There may be enough free space in chain_hlocks, but
> it is just too fragmented to be useful.
>
> Maybe we should figure out a better way to handle this fragmentation. In the
> mean time, the easiest way forward is just to increase the
> LOCKDEP_CHAINS_BITS by 1.
>
> >
> > However, a few other options we can try in lockdep are:
> >
> > * warn but not turn off the lockdep: the lock holding chain is
> > only a cache for what lock holding combination lockdep has ever
> > see, we also record the dependency in the graph. Without the
> > lock holding chain, lockdep can still work but just slower.
> >
> > * allow dynmaic memory allocation in lockdep: I think this might
> > be OK since we have lockdep_recursion to avoid lockdep code ->
> > mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> > missing something. And even we allow it, the use of memory
> > doesn't change, you will still need that amout of memory to
> > track lock holding chains.
>
> It is not just the issue of calling the memory allocator. There is also the
> issue of copying data from old chain_hlocks to new one while the old one may
> be updated during the copying process unless we can freeze everything else.
>
> Cheers,
> Longman
>

2023-01-26 19:08:18

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On 1/26/23 13:59, Boqun Feng wrote:
> On Thu, Jan 26, 2023 at 01:30:34PM -0500, Waiman Long wrote:
>> On 1/26/23 12:38, Boqun Feng wrote:
>>> [Cc lock folks]
>>>
>>> On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
>>>> On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
>>>>> On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
>>>>>> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
>>>>>>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>>>>>>>> Hi guys.
>>>>>>>> Always with intensive writing on a btrfs volume, the message "BUG:
>>>>>>>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>>>>>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>>>>>>> tends to work.
>>>>>> Hi,
>>>>>> Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
>>>>>> low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
>>>>>>
>>>>>> ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
>>>>>> CONFIG_LOCKDEP_CHAINS_BITS=18
>>>>>>
>>>>>> [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
>>>>>> [88685.088124] turning off the locking correctness validator.
>>>>>> [88685.088133] Please attach the output of /proc/lock_stat to the bug report
>>>>>> [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
>>>>>> ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
>>>>>> [88685.088154] Hardware name: System manufacturer System Product
>>>>>> Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
>>>>>>
>>>>>> What's next? Increase this value to 19?
>>>>> Yes, though increasing the value is a workaround so you may see the
>>>>> warning again.
>>>> Is there any sense in this WARNING if we would ignore it and every
>>>> time increase the threshold value?
>>> Lockdep uses static allocated array to track lock holdings chains to
>>> avoid dynmaic memory allocation in its own code. So if you see the
>>> warning it means your test has more combination of lock holdings than
>>> the array can record. In other words, you reach the resource limitation,
>>> and in that sense it makes sense to just ignore it and increase the
>>> value: you want to give lockdep enough resource to work, right?
>>>
>>>> May Be set 99 right away? Or remove such a check condition?
>>> That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
>>> chains array..
>> Note that every increment of LOCKDEP_CHAINS_BITS double the storage space.
>> With 99, that will likely exceed the total amount of memory you have in your
>> system.
>>
>> Boqun, where does the 5 figure come from. It is just a simple u16 array of
> #define MAX_LOCKDEP_CHAINS_BITS CONFIG_LOCKDEP_CHAINS_BITS
> #define MAX_LOCKDEP_CHAINS (1UL << MAX_LOCKDEP_CHAINS_BITS)
>
> #define MAX_LOCKDEP_CHAIN_HLOCKS (MAX_LOCKDEP_CHAINS*5)
>
> I think the last one means we think the average length of a lock chain
> is 5, in other words, in average, a task hold at most 5 locks. I don't
> know where the 5 came from either, but it's there ;-)

You are right. I missed that when I looked. So 5 is assumed to the
average length of a lock chain.

Thanks,
Longman

2023-01-26 22:43:10

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Thu, Jan 26, 2023 at 10:39 PM Boqun Feng <[email protected]> wrote:
>
> [Cc lock folks]
>
> On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
> > >
> > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> > > > >
> > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > Hi guys.
> > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > >
> > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > tends to work.
> > > >
> > > > Hi,
> > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > >
> > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > >
> > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > [88685.088124] turning off the locking correctness validator.
> > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> > > > ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > >
> > > > What's next? Increase this value to 19?
> > >
> > > Yes, though increasing the value is a workaround so you may see the
> > > warning again.
> >
> > Is there any sense in this WARNING if we would ignore it and every
> > time increase the threshold value?
>
> Lockdep uses static allocated array to track lock holdings chains to
> avoid dynmaic memory allocation in its own code. So if you see the
> warning it means your test has more combination of lock holdings than
> the array can record. In other words, you reach the resource limitation,
> and in that sense it makes sense to just ignore it and increase the
> value: you want to give lockdep enough resource to work, right?

It is needed for correct working btrfs. David, am I right?

>
> > May Be set 99 right away? Or remove such a check condition?
>
> That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> chains array..
>
> However, a few other options we can try in lockdep are:
>
> * warn but not turn off the lockdep: the lock holding chain is
> only a cache for what lock holding combination lockdep has ever
> see, we also record the dependency in the graph. Without the
> lock holding chain, lockdep can still work but just slower.
>
> * allow dynmaic memory allocation in lockdep: I think this might
> be OK since we have lockdep_recursion to avoid lockdep code ->
> mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> missing something. And even we allow it, the use of memory
> doesn't change, you will still need that amout of memory to
> track lock holding chains.
>
> I'm not sure whether these options are better than just increasing the
> number, maybe to unblock your ASAP, you can try make it 30 and make sure
> you have large enough memory to test.

About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
be done? In vanilla kernel on kernel.org? In a specific distribution?
or the user must rebuild the kernel himself? Maybe increase
LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
to distribute to end users because the meaning of using packaged
distributions is lost (user should change LOCKDEP_CHAINS_BITS in
config and rebuild the kernel by yourself).

It would be great if the chosen value would simply work always
everywhere. 30? ok! But as I understand, btrfs does not have any
guarantees for this. David, am I right?

Anyway, thank you for keeping the conversation going.

--
Best Regards,
Mike Gavrilov.

2023-01-26 22:52:42

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Fri, Jan 27, 2023 at 03:42:52AM +0500, Mikhail Gavrilov wrote:
> On Thu, Jan 26, 2023 at 10:39 PM Boqun Feng <[email protected]> wrote:
> >
> > [Cc lock folks]
> >
> > On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> > > > > >
> > > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > > Hi guys.
> > > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > > >
> > > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > > tends to work.
> > > > >
> > > > > Hi,
> > > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > > >
> > > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > > >
> > > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > > [88685.088124] turning off the locking correctness validator.
> > > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> > > > > ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > > >
> > > > > What's next? Increase this value to 19?
> > > >
> > > > Yes, though increasing the value is a workaround so you may see the
> > > > warning again.
> > >
> > > Is there any sense in this WARNING if we would ignore it and every
> > > time increase the threshold value?
> >
> > Lockdep uses static allocated array to track lock holdings chains to
> > avoid dynmaic memory allocation in its own code. So if you see the
> > warning it means your test has more combination of lock holdings than
> > the array can record. In other words, you reach the resource limitation,
> > and in that sense it makes sense to just ignore it and increase the
> > value: you want to give lockdep enough resource to work, right?
>
> It is needed for correct working btrfs. David, am I right?
>
> >
> > > May Be set 99 right away? Or remove such a check condition?
> >
> > That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> > chains array..
> >
> > However, a few other options we can try in lockdep are:
> >
> > * warn but not turn off the lockdep: the lock holding chain is
> > only a cache for what lock holding combination lockdep has ever
> > see, we also record the dependency in the graph. Without the
> > lock holding chain, lockdep can still work but just slower.
> >
> > * allow dynmaic memory allocation in lockdep: I think this might
> > be OK since we have lockdep_recursion to avoid lockdep code ->
> > mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> > missing something. And even we allow it, the use of memory
> > doesn't change, you will still need that amout of memory to
> > track lock holding chains.
> >
> > I'm not sure whether these options are better than just increasing the
> > number, maybe to unblock your ASAP, you can try make it 30 and make sure
> > you have large enough memory to test.
>
> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> be done? In vanilla kernel on kernel.org? In a specific distribution?
> or the user must rebuild the kernel himself? Maybe increase
> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> to distribute to end users because the meaning of using packaged
> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> config and rebuild the kernel by yourself).
>

Lockdep is a dev tool to help finding out deadlocks, and it introduces
cost when enabled, although it's possible, I doubt no one will run
LOCKDEP enabled kernel in production environment. In other words, it's
a debug/test-kernel-only option.

Regards,
Boqun

> It would be great if the chosen value would simply work always
> everywhere. 30? ok! But as I understand, btrfs does not have any
> guarantees for this. David, am I right?
>
> Anyway, thank you for keeping the conversation going.
>
> --
> Best Regards,
> Mike Gavrilov.

2023-01-26 23:50:25

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Fri, Jan 27, 2023 at 03:42:52AM +0500, Mikhail Gavrilov wrote:
> On Thu, Jan 26, 2023 at 10:39 PM Boqun Feng <[email protected]> wrote:
> >
> > [Cc lock folks]
> >
> > On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <[email protected]> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <[email protected]> wrote:
> > > > > >
> > > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > > Hi guys.
> > > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > > >
> > > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > > tends to work.
> > > > >
> > > > > Hi,
> > > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > > >
> > > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > > >
> > > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > > [88685.088124] turning off the locking correctness validator.
> > > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G W L
> > > > > ------- --- 6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > > >
> > > > > What's next? Increase this value to 19?
> > > >
> > > > Yes, though increasing the value is a workaround so you may see the
> > > > warning again.
> > >
> > > Is there any sense in this WARNING if we would ignore it and every
> > > time increase the threshold value?
> >
> > Lockdep uses static allocated array to track lock holdings chains to
> > avoid dynmaic memory allocation in its own code. So if you see the
> > warning it means your test has more combination of lock holdings than
> > the array can record. In other words, you reach the resource limitation,
> > and in that sense it makes sense to just ignore it and increase the
> > value: you want to give lockdep enough resource to work, right?
>
> It is needed for correct working btrfs. David, am I right?
>

Lockdep is not needed for correct working btrfs in production. It's a
tool to help btrfs developers to find deadlocks in
development/test/debug environment. End users, i.e. the users of linux
kernel don't need it.

Regards,
Boqun

> >
> > > May Be set 99 right away? Or remove such a check condition?
> >
> > That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> > chains array..
> >
> > However, a few other options we can try in lockdep are:
> >
> > * warn but not turn off the lockdep: the lock holding chain is
> > only a cache for what lock holding combination lockdep has ever
> > see, we also record the dependency in the graph. Without the
> > lock holding chain, lockdep can still work but just slower.
> >
> > * allow dynmaic memory allocation in lockdep: I think this might
> > be OK since we have lockdep_recursion to avoid lockdep code ->
> > mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> > missing something. And even we allow it, the use of memory
> > doesn't change, you will still need that amout of memory to
> > track lock holding chains.
> >
> > I'm not sure whether these options are better than just increasing the
> > number, maybe to unblock your ASAP, you can try make it 30 and make sure
> > you have large enough memory to test.
>
> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> be done? In vanilla kernel on kernel.org? In a specific distribution?
> or the user must rebuild the kernel himself? Maybe increase
> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> to distribute to end users because the meaning of using packaged
> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> config and rebuild the kernel by yourself).
>
> It would be great if the chosen value would simply work always
> everywhere. 30? ok! But as I understand, btrfs does not have any
> guarantees for this. David, am I right?
>
> Anyway, thank you for keeping the conversation going.
>
> --
> Best Regards,
> Mike Gavrilov.

2023-01-27 00:21:52

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On 1/26/23 17:42, Mikhail Gavrilov wrote:
>> I'm not sure whether these options are better than just increasing the
>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>> you have large enough memory to test.
> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> be done? In vanilla kernel on kernel.org? In a specific distribution?
> or the user must rebuild the kernel himself? Maybe increase
> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> to distribute to end users because the meaning of using packaged
> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> config and rebuild the kernel by yourself).

Note that lockdep is typically only enabled in a debug kernel shipped by
a distro because of the high performance overhead. The non-debug kernel
doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
when testing on the debug kernel, you can file a ticket to the distro
asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.

Cheers,
Longman

2023-01-27 03:38:26

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
> On 1/26/23 17:42, Mikhail Gavrilov wrote:
>>> I'm not sure whether these options are better than just increasing the
>>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>>> you have large enough memory to test.
>> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
>> be done? In vanilla kernel on kernel.org? In a specific distribution?
>> or the user must rebuild the kernel himself? Maybe increase
>> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
>> to distribute to end users because the meaning of using packaged
>> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
>> config and rebuild the kernel by yourself).
>
> Note that lockdep is typically only enabled in a debug kernel shipped by
> a distro because of the high performance overhead. The non-debug kernel
> doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
> when testing on the debug kernel, you can file a ticket to the distro
> asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
> your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.

Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921

If 19 the recommended value I don't mind sending an MR for it. But if the idea is we're going to be back here talking about bumping it to 20 in six months, I'd like to avoid that.

--
Chris Murphy

2023-01-27 04:08:16

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
>
>
> On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
> > On 1/26/23 17:42, Mikhail Gavrilov wrote:
> >>> I'm not sure whether these options are better than just increasing the
> >>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
> >>> you have large enough memory to test.
> >> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> >> be done? In vanilla kernel on kernel.org? In a specific distribution?
> >> or the user must rebuild the kernel himself? Maybe increase
> >> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> >> to distribute to end users because the meaning of using packaged
> >> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> >> config and rebuild the kernel by yourself).
> >
> > Note that lockdep is typically only enabled in a debug kernel shipped by
> > a distro because of the high performance overhead. The non-debug kernel
> > doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
> > when testing on the debug kernel, you can file a ticket to the distro
> > asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
> > your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
>
> Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
>
> If 19 the recommended value I don't mind sending an MR for it. But if
> the idea is we're going to be back here talking about bumping it to 20
> in six months, I'd like to avoid that.
>

How about a boot parameter then?

Regards,
Boqun

>
>
> --
> Chris Murphy

2023-01-27 05:36:37

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On Fri, Jan 27, 2023 at 9:08 AM Boqun Feng <[email protected]> wrote:
>
> On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
> >
> >
> > On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
> > > On 1/26/23 17:42, Mikhail Gavrilov wrote:
> > >>> I'm not sure whether these options are better than just increasing the
> > >>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
> > >>> you have large enough memory to test.
> > >> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> > >> be done? In vanilla kernel on kernel.org? In a specific distribution?
> > >> or the user must rebuild the kernel himself? Maybe increase
> > >> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> > >> to distribute to end users because the meaning of using packaged
> > >> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> > >> config and rebuild the kernel by yourself).
> > >
> > > Note that lockdep is typically only enabled in a debug kernel shipped by
> > > a distro because of the high performance overhead. The non-debug kernel
> > > doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
> > > when testing on the debug kernel, you can file a ticket to the distro
> > > asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
> > > your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
> >
> > Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
> > https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
> >
> > If 19 the recommended value I don't mind sending an MR for it. But if
> > the idea is we're going to be back here talking about bumping it to 20
> > in six months, I'd like to avoid that.
> >
>
> How about a boot parameter then?

I would like this option.
This is better than rebuilding the kernel yourself and asking the
distribution's maintainers to increase this value.

Thanks.

--
Best Regards,
Mike Gavrilov.

2023-01-27 14:27:14

[permalink] [raw]

Subject: Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

On 1/26/23 23:07, Boqun Feng wrote:
> On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
>>
>> On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
>>> On 1/26/23 17:42, Mikhail Gavrilov wrote:
>>>>> I'm not sure whether these options are better than just increasing the
>>>>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>>>>> you have large enough memory to test.
>>>> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
>>>> be done? In vanilla kernel on kernel.org? In a specific distribution?
>>>> or the user must rebuild the kernel himself? Maybe increase
>>>> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
>>>> to distribute to end users because the meaning of using packaged
>>>> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
>>>> config and rebuild the kernel by yourself).
>>> Note that lockdep is typically only enabled in a debug kernel shipped by
>>> a distro because of the high performance overhead. The non-debug kernel
>>> doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
>>> when testing on the debug kernel, you can file a ticket to the distro
>>> asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
>>> your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
>> Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
>> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
>>
>> If 19 the recommended value I don't mind sending an MR for it. But if
>> the idea is we're going to be back here talking about bumping it to 20
>> in six months, I'd like to avoid that.
>>
> How about a boot parameter then?

A boot parameter doesn't work for a statically allocated array which is
determined at compile time. Dynamic memory allocation isn't enabled yet
at early boot when lockdep will be used.

Cheers,
Longman

2023-01-27 15:34:06