2022-11-04 11:14:48

by Mirsad Todorovac

[permalink] [raw]
Subject: BUG: kworker memory leak found in 6.1.0-rc3

Dear Sirs,

When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
CONFIG_DEBUG_KMEMLEAK=y
and the result showed an unreferenced object in kworker process:

# cat /sys/kernel/debug/kmemleak
unreferenced object 0xffffa01dabff6100 (size 16):
  comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
  hex dump (first 16 bytes):
    6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
  backtrace:
    [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
    [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
    [<0000000005472512>] kstrdup+0x36/0x70
    [<000000002f797ac4>] kstrdup_const+0x28/0x30
    [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
    [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
    [<000000004158a6c0>] dev_set_name+0x53/0x70
    [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
    [<00000000122bb894>] process_one_work+0x214/0x3f0
    [<00000000fcf282cc>] worker_thread+0x34/0x3d0
    [<0000000002409855>] kthread+0xed/0x120
    [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
unreferenced object 0xffffa01dabff6ec0 (size 16):
  comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
  hex dump (first 16 bytes):
    6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
  backtrace:
    [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
    [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
    [<0000000005472512>] kstrdup+0x36/0x70
    [<000000002f797ac4>] kstrdup_const+0x28/0x30
    [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
    [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
    [<000000004158a6c0>] dev_set_name+0x53/0x70
    [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
    [<00000000122bb894>] process_one_work+0x214/0x3f0
    [<00000000fcf282cc>] worker_thread+0x34/0x3d0
    [<0000000002409855>] kthread+0xed/0x120
    [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
#

Please fing the build config and lshw output attached.

dmesg is useless, as it is filled with events like:

[ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
[ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
[ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
[ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
[ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
[ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
[ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
[ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
[ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
[ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
[ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0

Thank you,

Mirsad

--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
--
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia


Attachments:
lshw-20221104.txt (22.89 kB)
config-6.1.0-rc3.gz (64.72 kB)
Download all attachments

2022-11-10 05:28:18

by Mirsad Todorovac

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:

> Dear Sirs,
>
> When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
> CONFIG_DEBUG_KMEMLEAK=y
> and the result showed an unreferenced object in kworker process:
>
> # cat /sys/kernel/debug/kmemleak
> unreferenced object 0xffffa01dabff6100 (size 16):
>   comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
>   hex dump (first 16 bytes):
>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>   backtrace:
>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>     [<0000000005472512>] kstrdup+0x36/0x70
>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>     [<0000000002409855>] kthread+0xed/0x120
>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
> unreferenced object 0xffffa01dabff6ec0 (size 16):
>   comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
>   hex dump (first 16 bytes):
>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>   backtrace:
>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>     [<0000000005472512>] kstrdup+0x36/0x70
>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>     [<0000000002409855>] kthread+0xed/0x120
>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
> #
>
> Please fing the build config and lshw output attached.
>
> dmesg is useless, as it is filled with events like:
>
> [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
> [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
> [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
> [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
> [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
> [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
> [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
> [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0

This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
kernel memory leaks, potentially exposing race conditions or other more
serious bug.

The bug is now also confirmed and now manifested also in the Ubuntu
22.04 LTS jammy 6.1-rc4 build.

Here is the kmemleak output:

unreferenced object 0xffff9242b13b3980 (size 64):
  comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
  hex dump (first 32 bytes):
    80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ....B...........
    20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B...........
  backtrace:
    [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
    [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
    [<000000004c09eee5>] xhci_alloc_command+0x6e/0x180
    [<0000000099436a99>] xhci_alloc_command_with_ctx+0x1d/0x60
    [<0000000070f45e17>] xhci_change_max_exit_latency+0x2e/0x1c0
    [<000000001b13cf46>] xhci_disable_usb3_lpm_timeout+0x77/0xb0
    [<00000000ff561da4>] usb_disable_link_state+0x53/0xd0
    [<0000000068febd40>] usb_disable_lpm+0x82/0xc0
    [<00000000dffec01f>] usb_unlocked_disable_lpm+0x2d/0x50
    [<0000000061a59294>] usb_disable_device+0x12f/0x250
    [<0000000044e2299d>] usb_set_configuration+0x65c/0x9a0
    [<000000001af32470>] driver_set_config_work+0x78/0xa0
    [<00000000320216c3>] process_one_work+0x214/0x3f0
    [<00000000694e932f>] worker_thread+0x34/0x3d0
    [<0000000002e30775>] kthread+0xed/0x120
    [<00000000512e8066>] ret_from_fork+0x1f/0x30
unreferenced object 0xffff9242f0a08620 (size 32):
  comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    30 86 a0 f0 42 92 ff ff 30 86 a0 f0 42 92 ff ff 0...B...0...B...
  backtrace:
    [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
    [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
    [<00000000459908cf>] xhci_alloc_command+0xe6/0x180
    [<0000000099436a99>] xhci_alloc_command_with_ctx+0x1d/0x60
    [<0000000070f45e17>] xhci_change_max_exit_latency+0x2e/0x1c0
    [<000000001b13cf46>] xhci_disable_usb3_lpm_timeout+0x77/0xb0
    [<00000000ff561da4>] usb_disable_link_state+0x53/0xd0
    [<0000000068febd40>] usb_disable_lpm+0x82/0xc0
    [<00000000dffec01f>] usb_unlocked_disable_lpm+0x2d/0x50
    [<0000000061a59294>] usb_disable_device+0x12f/0x250
    [<0000000044e2299d>] usb_set_configuration+0x65c/0x9a0
    [<000000001af32470>] driver_set_config_work+0x78/0xa0
    [<00000000320216c3>] process_one_work+0x214/0x3f0
    [<00000000694e932f>] worker_thread+0x34/0x3d0
    [<0000000002e30775>] kthread+0xed/0x120
    [<00000000512e8066>] ret_from_fork+0x1f/0x30

Please find the complete kmemleak output attached:

Thanks,
Mirsad

--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
--
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
The European Union


Attachments:
thermald_systemd-dev_kworker.memleak.lrz (2.66 kB)

2022-11-10 09:03:59

by Thorsten Leemhuis

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 10.11.22 05:57, Mirsad Goran Todorovac wrote:
> On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:

> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
> kernel memory leaks, potentially exposing race conditions or other more
> serious bug.

Maybe, but that sadly is also true for a lot of other known issues, for
example those in this quite long list:
https://syzkaller.appspot.com/upstream#open

It would help if you could pinpoint the problem, then we know who should
look into this. You CCed me and the regression list, so I assume it's a
regression. Hence: Could you try to bisect it?

Ciao, Thorsten

2022-11-10 09:40:46

by Greg Kroah-Hartman

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote:
> On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:
>
> > Dear Sirs,
> >
> > When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
> > CONFIG_DEBUG_KMEMLEAK=y
> > and the result showed an unreferenced object in kworker process:
> >
> > # cat /sys/kernel/debug/kmemleak
> > unreferenced object 0xffffa01dabff6100 (size 16):
> > ? comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
> > ? hex dump (first 16 bytes):
> > ??? 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
> > ? backtrace:
> > ??? [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
> > ??? [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
> > ??? [<0000000005472512>] kstrdup+0x36/0x70
> > ??? [<000000002f797ac4>] kstrdup_const+0x28/0x30
> > ??? [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
> > ??? [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
> > ??? [<000000004158a6c0>] dev_set_name+0x53/0x70
> > ??? [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
> > ??? [<00000000122bb894>] process_one_work+0x214/0x3f0
> > ??? [<00000000fcf282cc>] worker_thread+0x34/0x3d0
> > ??? [<0000000002409855>] kthread+0xed/0x120
> > ??? [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
> > unreferenced object 0xffffa01dabff6ec0 (size 16):
> > ? comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
> > ? hex dump (first 16 bytes):
> > ??? 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
> > ? backtrace:
> > ??? [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
> > ??? [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
> > ??? [<0000000005472512>] kstrdup+0x36/0x70
> > ??? [<000000002f797ac4>] kstrdup_const+0x28/0x30
> > ??? [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
> > ??? [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
> > ??? [<000000004158a6c0>] dev_set_name+0x53/0x70
> > ??? [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
> > ??? [<00000000122bb894>] process_one_work+0x214/0x3f0
> > ??? [<00000000fcf282cc>] worker_thread+0x34/0x3d0
> > ??? [<0000000002409855>] kthread+0xed/0x120
> > ??? [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
> > #
> >
> > Please fing the build config and lshw output attached.
> >
> > dmesg is useless, as it is filled with events like:
> >
> > [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
> > [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
> > [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
> > [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
> > [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
> > [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
> > [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
> > [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>
> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
> kernel memory leaks, potentially exposing race conditions or other more
> serious bug.

How is a memory leak a race condition?

> The bug is now also confirmed and now manifested also in the Ubuntu 22.04
> LTS jammy 6.1-rc4 build.
>
> Here is the kmemleak output:
>
> unreferenced object 0xffff9242b13b3980 (size 64):
> ? comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
> ? hex dump (first 32 bytes):
> ??? 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ....B...........
> ??? 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B...........
> ? backtrace:
> ??? [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
> ??? [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
> ??? [<000000004c09eee5>] xhci_alloc_command+0x6e/0x180

This is a totally different backtrace from above, how are they related?

This looks like a potential xhci issue. Can you use 'git bisect' to
track down the offending change that caused this?

thanks,

greg k-h

2022-11-11 09:13:30

by Mirsad Todorovac

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 10.11.2022. 9:28, Thorsten Leemhuis wrote:

> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
> kernel memory leaks, potentially exposing race conditions or other more
> serious bug.
>> Maybe, but that sadly is also true for a lot of other known issues, for
>> example those in this quite long list:
>> https://syzkaller.appspot.com/upstream#open
>>
>> It would help if you could pinpoint the problem, then we know who should
>> look into this. You CCed me and the regression list, so I assume it's a
>> regression. Hence: Could you try to bisect it?
>>
>> Ciao, Thorsten

Hi, Thorsten!

I am taking this task, however, it might not be a relatively easy bisect
like the past
two or three (probably were beginner's luck).

It appears to predate 4.19:

$ head -13 memleak-kwork-4.19.0-08-84df9525b0c2-memlk-menu-al.log
unreferenced object 0xffff8ff917c79c40 (size 16):
  comm "kworker/u12:4", pid 422, jiffies 4294672757 (age 380.342s)
  hex dump (first 16 bytes):
    6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
  backtrace:
    [<000000004f7bc99c>] kobject_set_name_vargs+0x1e/0x90
    [<00000000f00ab312>] dev_set_name+0x57/0x70
    [<000000001fd9518e>] memstick_check+0x99/0x330 [memstick]
    [<000000008c797991>] process_one_work+0x1a7/0x3a0
    [<00000000405c5133>] worker_thread+0x30/0x390
    [<0000000060a58c2e>] kthread+0x112/0x130
    [<000000004b138871>] ret_from_fork+0x35/0x40
    [<0000000029f3e0aa>] 0xffffffffffffffff
$

Cheers,
Mirsad

--
Mirsad Todorovac
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb
Republic of Croatia, the European Union
--
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu


2022-11-11 10:19:09

by Mirsad Todorovac

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 10.11.2022. 9:28, Thorsten Leemhuis wrote:

> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
> kernel memory leaks, potentially exposing race conditions or other more
> serious bug.
>> Maybe, but that sadly is also true for a lot of other known issues, for
>> example those in this quite long list:
>> https://syzkaller.appspot.com/upstream#open
>>
>> It would help if you could pinpoint the problem, then we know who should
>> look into this. You CCed me and the regression list, so I assume it's a
>> regression. Hence: Could you try to bisect it?
>>
>> Ciao, Thorsten

Hi, Thorsten!

I am taking this task, however, it might not be a relatively easy bisect
like the past
two or three (probably were beginner's luck).

It appears to predate 4.19:

$ head -13 memleak-kwork-4.19.0-08-84df9525b0c2-memlk-menu-al.log
unreferenced object 0xffff8ff917c79c40 (size 16):
  comm "kworker/u12:4", pid 422, jiffies 4294672757 (age 380.342s)
  hex dump (first 16 bytes):
    6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
  backtrace:
    [<000000004f7bc99c>] kobject_set_name_vargs+0x1e/0x90
    [<00000000f00ab312>] dev_set_name+0x57/0x70
    [<000000001fd9518e>] memstick_check+0x99/0x330 [memstick]
    [<000000008c797991>] process_one_work+0x1a7/0x3a0
    [<00000000405c5133>] worker_thread+0x30/0x390
    [<0000000060a58c2e>] kthread+0x112/0x130
    [<000000004b138871>] ret_from_fork+0x35/0x40
    [<0000000029f3e0aa>] 0xffffffffffffffff
$

Cheers,
Mirsad

P.S. This mail is re-sent because of forgetting to set "Plain text" mode
to Thunderbird.

--
Mirsad Todorovac
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb
Republic of Croatia, the European Union
--
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu


2022-11-17 15:40:37

by Mirsad Todorovac

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 10.11.2022. 10:20, Greg KH wrote:
> On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote:
>> On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:
>>
>>> Dear Sirs,
>>>
>>> When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
>>> CONFIG_DEBUG_KMEMLEAK=y
>>> and the result showed an unreferenced object in kworker process:
>>>
>>> # cat /sys/kernel/debug/kmemleak
>>> unreferenced object 0xffffa01dabff6100 (size 16):
>>>   comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
>>>   hex dump (first 16 bytes):
>>>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>>>   backtrace:
>>>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>>>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>>>     [<0000000005472512>] kstrdup+0x36/0x70
>>>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>>>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>>>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>>>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>>>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>>>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>>>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>>>     [<0000000002409855>] kthread+0xed/0x120
>>>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
>>> unreferenced object 0xffffa01dabff6ec0 (size 16):
>>>   comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
>>>   hex dump (first 16 bytes):
>>>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>>>   backtrace:
>>>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>>>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>>>     [<0000000005472512>] kstrdup+0x36/0x70
>>>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>>>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>>>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>>>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>>>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>>>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>>>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>>>     [<0000000002409855>] kthread+0xed/0x120
>>>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
>>> #
>>>
>>> Please fing the build config and lshw output attached.
>>>
>>> dmesg is useless, as it is filled with events like:
>>>
>>> [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
>>> [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>> [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
>>> [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
>>> [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>> [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
>>> [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
>>> [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>> [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
>>> [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
>>> [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>
>> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
>> kernel memory leaks, potentially exposing race conditions or other more
>> serious bug.
>
> How is a memory leak a race condition?
Probably I connected it with spurious memleak in thermald that occurs
only under the torture tests.

>> The bug is now also confirmed and now manifested also in the Ubuntu 22.04
>> LTS jammy 6.1-rc4 build.
>>
>> Here is the kmemleak output:
>>
>> unreferenced object 0xffff9242b13b3980 (size 64):
>>   comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
>>   hex dump (first 32 bytes):
>>     80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ....B...........
>>     20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B...........
>>   backtrace:
>>     [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
>>     [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
>>     [<000000004c09eee5>] xhci_alloc_command+0x6e/0x180
>
> This is a totally different backtrace from above, how are they related?
>
> This looks like a potential xhci issue. Can you use 'git bisect' to
> track down the offending change that caused this?
>
> thanks,
>
> greg k-h

Hi, Greg, I saw this email only now. I will try to bisect the bug.
I was trying to bisect the first instance that occurs only on AlmaLinux
and it is introduced before 4.17. Here is the progress of the bisect on
AlmaLinux 8.6 (a.k.a. Centos 8):

mtodorov@domac:~/linux/kernel/linux_stable$ git bisect log
git bisect start
# bad: [f0c4d9fc9cc9462659728d168387191387e903cc] Linux 6.1-rc4
git bisect bad f0c4d9fc9cc9462659728d168387191387e903cc
# bad: [fbd56ddcecab5a3623a89c8e941fdbcc55b41045] Linux 6.0.1
git bisect bad fbd56ddcecab5a3623a89c8e941fdbcc55b41045
# bad: [7e18e42e4b280c85b76967a9106a13ca61c16179] Linux 6.0-rc4
git bisect bad 7e18e42e4b280c85b76967a9106a13ca61c16179
# bad: [568035b01cfb107af8d2e4bd2fb9aea22cf5b868] Linux 6.0-rc1
git bisect bad 568035b01cfb107af8d2e4bd2fb9aea22cf5b868
# bad: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux 4.19
git bisect bad 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d
# bad: [94710cac0ef4ee177a63b5227664b38c95bbf703] Linux 4.18
git bisect bad 94710cac0ef4ee177a63b5227664b38c95bbf703
# bad: [29dcea88779c856c7dc92040a0c01233263101d4] Linux 4.17
git bisect bad 29dcea88779c856c7dc92040a0c01233263101d4
mtodorov@domac:~/linux/kernel/linux_stable$

I failed to notice the different nature of the second memory leak.
Of course, if it is reproducible always in the same way, I'd agree that
it doesn't look like a race condition, but from one occurrence it is
hard to tell (I generalised from thermald/acpi bug bisect).

Please give me some grace period for I am only 2 1/2 months in the
kernel bisecting ...

Cheers,
Mirsad

--
Mirsad Todorovac
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb
Republic of Croatia, the European Union
--
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu

2022-11-29 03:43:33

by Mirsad Todorovac

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 10. 11. 2022. 10:20, Greg KH wrote:
> On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote:
>> On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:
>>
>>> Dear Sirs,
>>>
>>> When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
>>> CONFIG_DEBUG_KMEMLEAK=y
>>> and the result showed an unreferenced object in kworker process:
>>>
>>> # cat /sys/kernel/debug/kmemleak
>>> unreferenced object 0xffffa01dabff6100 (size 16):
>>>   comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
>>>   hex dump (first 16 bytes):
>>>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>>>   backtrace:
>>>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>>>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>>>     [<0000000005472512>] kstrdup+0x36/0x70
>>>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>>>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>>>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>>>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>>>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>>>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>>>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>>>     [<0000000002409855>] kthread+0xed/0x120
>>>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
>>> unreferenced object 0xffffa01dabff6ec0 (size 16):
>>>   comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
>>>   hex dump (first 16 bytes):
>>>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>>>   backtrace:
>>>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>>>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>>>     [<0000000005472512>] kstrdup+0x36/0x70
>>>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>>>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>>>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>>>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>>>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>>>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>>>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>>>     [<0000000002409855>] kthread+0xed/0x120
>>>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
>>> #
>>>
>>> Please fing the build config and lshw output attached.
>>>
>>> dmesg is useless, as it is filled with events like:
>>>
>>> [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
>>> [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>> [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
>>> [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
>>> [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>> [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
>>> [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
>>> [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>> [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
>>> [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
>>> [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>
>> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
>> kernel memory leaks, potentially exposing race conditions or other more
>> serious bug.
>
> How is a memory leak a race condition?
>
>> The bug is now also confirmed and now manifested also in the Ubuntu 22.04
>> LTS jammy 6.1-rc4 build.
>>
>> Here is the kmemleak output:
>>
>> unreferenced object 0xffff9242b13b3980 (size 64):
>>   comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
>>   hex dump (first 32 bytes):
>>     80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ....B...........
>>     20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B...........
>>   backtrace:
>>     [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
>>     [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
>>     [<000000004c09eee5>] xhci_alloc_command+0x6e/0x180
>
> This is a totally different backtrace from above, how are they related?
>
> This looks like a potential xhci issue. Can you use 'git bisect' to
> track down the offending change that caused this?
>
> thanks,
>
> greg k-h

Hello, Greg, Thorsten!

After multiple attempts, my box's UEFI refuses to run pre-4.17 kernels.
The bisect shows the problem appeared before 4.17, so unless I find what is
causing black screen when booting pre-4.17 kernels, it's a no-go ... :(

Thanks,
Mirsad

--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
--
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
The European Union

2022-11-29 09:09:09

by Greg Kroah-Hartman

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On Tue, Nov 29, 2022 at 04:35:10AM +0100, Mirsad Goran Todorovac wrote:
> On 10. 11. 2022. 10:20, Greg KH wrote:
> > On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote:
> > > On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:
> > >
> > > > Dear Sirs,
> > > >
> > > > When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
> > > > CONFIG_DEBUG_KMEMLEAK=y
> > > > and the result showed an unreferenced object in kworker process:
> > > >
> > > > # cat /sys/kernel/debug/kmemleak
> > > > unreferenced object 0xffffa01dabff6100 (size 16):
> > > > ? comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
> > > > ? hex dump (first 16 bytes):
> > > > ??? 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
> > > > ? backtrace:
> > > > ??? [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
> > > > ??? [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
> > > > ??? [<0000000005472512>] kstrdup+0x36/0x70
> > > > ??? [<000000002f797ac4>] kstrdup_const+0x28/0x30
> > > > ??? [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
> > > > ??? [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
> > > > ??? [<000000004158a6c0>] dev_set_name+0x53/0x70
> > > > ??? [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
> > > > ??? [<00000000122bb894>] process_one_work+0x214/0x3f0
> > > > ??? [<00000000fcf282cc>] worker_thread+0x34/0x3d0
> > > > ??? [<0000000002409855>] kthread+0xed/0x120
> > > > ??? [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
> > > > unreferenced object 0xffffa01dabff6ec0 (size 16):
> > > > ? comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
> > > > ? hex dump (first 16 bytes):
> > > > ??? 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
> > > > ? backtrace:
> > > > ??? [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
> > > > ??? [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
> > > > ??? [<0000000005472512>] kstrdup+0x36/0x70
> > > > ??? [<000000002f797ac4>] kstrdup_const+0x28/0x30
> > > > ??? [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
> > > > ??? [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
> > > > ??? [<000000004158a6c0>] dev_set_name+0x53/0x70
> > > > ??? [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
> > > > ??? [<00000000122bb894>] process_one_work+0x214/0x3f0
> > > > ??? [<00000000fcf282cc>] worker_thread+0x34/0x3d0
> > > > ??? [<0000000002409855>] kthread+0xed/0x120
> > > > ??? [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
> > > > #
> > > >
> > > > Please fing the build config and lshw output attached.
> > > >
> > > > dmesg is useless, as it is filled with events like:
> > > >
> > > > [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
> > > > [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > > > [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
> > > > [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
> > > > [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > > > [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
> > > > [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
> > > > [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > > > [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
> > > > [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
> > > > [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
> > >
> > > This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
> > > kernel memory leaks, potentially exposing race conditions or other more
> > > serious bug.
> >
> > How is a memory leak a race condition?
> >
> > > The bug is now also confirmed and now manifested also in the Ubuntu 22.04
> > > LTS jammy 6.1-rc4 build.
> > >
> > > Here is the kmemleak output:
> > >
> > > unreferenced object 0xffff9242b13b3980 (size 64):
> > > ? comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
> > > ? hex dump (first 32 bytes):
> > > ??? 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ....B...........
> > > ??? 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B...........
> > > ? backtrace:
> > > ??? [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
> > > ??? [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
> > > ??? [<000000004c09eee5>] xhci_alloc_command+0x6e/0x180
> >
> > This is a totally different backtrace from above, how are they related?
> >
> > This looks like a potential xhci issue. Can you use 'git bisect' to
> > track down the offending change that caused this?
> >
> > thanks,
> >
> > greg k-h
>
> Hello, Greg, Thorsten!
>
> After multiple attempts, my box's UEFI refuses to run pre-4.17 kernels.
> The bisect shows the problem appeared before 4.17, so unless I find what is
> causing black screen when booting pre-4.17 kernels, it's a no-go ... :(

Ok, so I guess this has always been an issue, and is not a regression,
which is good. Can you work with the memstick developers to find a
solution?

thanks,

greg k-h

2022-11-29 10:10:13

by Mirsad Todorovac

[permalink] [raw]
Subject: Re: BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4

On 29.11.2022. 9:36, Greg KH wrote:
> On Tue, Nov 29, 2022 at 04:35:10AM +0100, Mirsad Goran Todorovac wrote:
>> On 10. 11. 2022. 10:20, Greg KH wrote:
>>> On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote:
>>>> On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote:
>>>>
>>>>> Dear Sirs,
>>>>>
>>>>> When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled
>>>>> CONFIG_DEBUG_KMEMLEAK=y
>>>>> and the result showed an unreferenced object in kworker process:
>>>>>
>>>>> # cat /sys/kernel/debug/kmemleak
>>>>> unreferenced object 0xffffa01dabff6100 (size 16):
>>>>>   comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s)
>>>>>   hex dump (first 16 bytes):
>>>>>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>>>>>   backtrace:
>>>>>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>>>>>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>>>>>     [<0000000005472512>] kstrdup+0x36/0x70
>>>>>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>>>>>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>>>>>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>>>>>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>>>>>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>>>>>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>>>>>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>>>>>     [<0000000002409855>] kthread+0xed/0x120
>>>>>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
>>>>> unreferenced object 0xffffa01dabff6ec0 (size 16):
>>>>>   comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s)
>>>>>   hex dump (first 16 bytes):
>>>>>     6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
>>>>>   backtrace:
>>>>>     [<000000009ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0
>>>>>     [<00000000451f4268>] __kmalloc_node_track_caller+0x55/0x150
>>>>>     [<0000000005472512>] kstrdup+0x36/0x70
>>>>>     [<000000002f797ac4>] kstrdup_const+0x28/0x30
>>>>>     [<00000000e3f86581>] kvasprintf_const+0x78/0xa0
>>>>>     [<00000000e15920f7>] kobject_set_name_vargs+0x23/0xa0
>>>>>     [<000000004158a6c0>] dev_set_name+0x53/0x70
>>>>>     [<000000001a120541>] memstick_check+0xff/0x384 [memstick]
>>>>>     [<00000000122bb894>] process_one_work+0x214/0x3f0
>>>>>     [<00000000fcf282cc>] worker_thread+0x34/0x3d0
>>>>>     [<0000000002409855>] kthread+0xed/0x120
>>>>>     [<000000007b02b4a3>] ret_from_fork+0x1f/0x30
>>>>> #
>>>>>
>>>>> Please fing the build config and lshw output attached.
>>>>>
>>>>> dmesg is useless, as it is filled with events like:
>>>>>
>>>>> [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0
>>>>> [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>>>> [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
>>>>> [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1
>>>>> [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>>>> [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762
>>>>> [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0
>>>>> [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>>>> [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792
>>>>> [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1
>>>>> [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0
>>>>
>>>> This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev"
>>>> kernel memory leaks, potentially exposing race conditions or other more
>>>> serious bug.
>>>
>>> How is a memory leak a race condition?
>>>
>>>> The bug is now also confirmed and now manifested also in the Ubuntu 22.04
>>>> LTS jammy 6.1-rc4 build.
>>>>
>>>> Here is the kmemleak output:
>>>>
>>>> unreferenced object 0xffff9242b13b3980 (size 64):
>>>>   comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s)
>>>>   hex dump (first 32 bytes):
>>>>     80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ....B...........
>>>>     20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B...........
>>>>   backtrace:
>>>>     [<00000000c5dea4db>] __kmem_cache_alloc_node+0x380/0x4e0
>>>>     [<000000002b17af47>] kmalloc_node_trace+0x27/0xa0
>>>>     [<000000004c09eee5>] xhci_alloc_command+0x6e/0x180
>>>
>>> This is a totally different backtrace from above, how are they related?
>>>
>>> This looks like a potential xhci issue. Can you use 'git bisect' to
>>> track down the offending change that caused this?
>>>
>>> thanks,
>>>
>>> greg k-h
>>
>> Hello, Greg, Thorsten!
>>
>> After multiple attempts, my box's UEFI refuses to run pre-4.17 kernels.
>> The bisect shows the problem appeared before 4.17, so unless I find what is
>> causing black screen when booting pre-4.17 kernels, it's a no-go ... :(
>
> Ok, so I guess this has always been an issue, and is not a regression,
> which is good. Can you work with the memstick developers to find a
> solution?

Hi, Greg,

Of course, I will gladly cooperate with the memstick team.
I will CC: everyone with commits to the memstick driver, I hope that's
not too awkward.

So far, the Code of Conduct says to inform the maintainers about the bug.

BTW, the bug is confirmed as unfixed in 6.1-rc7:

# cat /sys/kernel/debug/kmemleak
unreferenced object 0xffff93e548ab1e90 (size 16):
comm "kworker/u12:5", pid 405, jiffies 4294894087 (age 65919.068s)
hex dump (first 16 bytes):
6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0.......
backtrace:
[<00000000942f1553>] __kmem_cache_alloc_node+0x380/0x4e0
[<00000000555b3e8a>] __kmalloc_node_track_caller+0x55/0x140
[<000000000b60a98a>] kstrdup+0x36/0x70
[<00000000f9a4a52a>] kstrdup_const+0x28/0x30
[<000000005c5ca378>] kvasprintf_const+0x78/0xa0
[<00000000b8f94e41>] kobject_set_name_vargs+0x23/0xa0
[<00000000b7a2c8ea>] dev_set_name+0x53/0x70
[<00000000291af717>] memstick_check+0xff/0x384 [memstick]
[<000000007b776e48>] process_one_work+0x214/0x3f0
[<000000005791f9b2>] worker_thread+0x34/0x3d0
[<00000000df696ef8>] kthread+0xed/0x120
[<0000000016f05dd5>] ret_from_fork+0x1f/0x30
#

Thanks,
Mirsad

--
Mirsad Todorovac
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb
Republic of Croatia, the European Union
--
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu