Hi everyone!
Recently I got next trace on one of machines (backtrace can be not in correct
order since it captured via netconsole):
[7932653.070316] ------------[ cut here ]------------
[7932653.070359] kernel BUG at fs/jbd2/journal.c:759!
[7932653.070392] invalid opcode: 0000 [#1] SMP
[7932653.070424] Modules linked in: ...
[7932653.071108] CPU: 0 PID: 356 Comm: jbd2/md0-8 Not tainted 4.8.1 #1
[7932653.071164] Hardware name: Supermicro Super Server/X10DRW-i, BIOS 1.1 08/13/2015
[7932653.071222] task: ffff881ff1babfc0 task.stack: ffff881fef890000
[<ffffffffa0454728>] jbd2_journal_next_log_block+0x68/0x70 [jbd2]
[7932653.071258] RIP: 0010:[<ffffffffa0454728>]
[7932653.071338] RSP: 0018:ffff881fef893cc8 EFLAGS: 00010246
[7932653.071374] RAX: 0000000000000001 RBX: ffff883fed5a8800 RCX: 00000000000007c4
[7932653.071432] RDX: 00000000000000ff RSI: ffff881fef893d68 RDI: ffff883fed5a8824
[7932653.071489] RBP: ffff881fef893d68 R08: 0000000000000000 R09: 0000000000000000
[7932653.071546] R10: 0000000000000001 R11: 0000000002408840 R12: ffff8822a92aad00
[7932653.071603] R13: ffff882dfc7b083c R14: ffff883fed5a8800 R15: ffff883fecb5ad00
[7932653.071659] FS: 0000000000000000(0000) GS:ffff881fffc00000(0000) knlGS:0000000000000000
[7932653.071718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[7932653.071753] CR2: 00007f2d363fc730 CR3: 0000000001806000 CR4: 00000000001406f0
[7932653.071809] Stack:
0000000800000018
ffffffffa044d7e5
ffff881fef893e60
[7932653.071843] ffff881274fca7f8
ffff883fecb5ad9c
ffff881fe4e26f00
001c2ecad4ef6dba
[7932653.071913] ffff881ff1babfc0
ffff882dfc7b083c
ffff883ff10c3000
000007c400000000
[7932653.071982] ffff881ff1babfc0
[7932653.072051] Call Trace:
[7932653.072082] [<ffffffffa044d7e5>] ? jbd2_journal_commit_transaction+0xa65/0x18a0 [jbd2]
[7932653.072143] [<ffffffffa04530f2>] ? kjournald2+0xb2/0x240 [jbd2]
[7932653.072182] [<ffffffff81088e90>] ? wake_atomic_t_function+0x50/0x50
[7932653.072220] [<ffffffffa0453040>] ? commit_timeout+0x10/0x10 [jbd2]
[7932653.072258] [<ffffffff8106e17d>] ? kthread+0xbd/0xe0
[7932653.072294] [<ffffffff814638cf>] ? ret_from_fork+0x1f/0x40
[7932653.072330] [<ffffffff8106e0c0>] ? kthread_worker_fn+0x160/0x160
...
[<ffffffffa0454728>] jbd2_journal_next_log_block+0x68/0x70 [jbd2]
[7932653.072732] RIP
[7932653.072793] RSP <ffff881fef893cc8>
[7932653.073307] ---[ end trace 3f9eca026a57b5c2 ]---
And a lot of WARN_ON(blk_needs_flush_plug(tsk)) after:
[7932653.115857] WARNING: CPU: 0 PID: 356 at kernel/exit.c:737 do_exit+0x50/0xad0
...
[7932653.123101] Call Trace:
[7932653.123188] [<ffffffff8122fd58>] ? dump_stack+0x46/0x5e
[7932653.123280] [<ffffffff8105312e>] ? __warn+0xbe/0xe0
[7932653.123371] [<ffffffff810557b0>] ? do_exit+0x50/0xad0
[7932653.123462] [<ffffffff81464a37>] ? rewind_stack_do_exit+0x17/0x20
[7932653.123556] [<ffffffff8106e0c0>] ? kthread_worker_fn+0x160/0x160
[7932653.123648] ---[ end trace 3f9eca026a57b5c4 ]---
[7932653.123737] Fixing recursive fault but reboot is needed!
[7932653.123830] BUG: unable to handle kernel
I looked through the code on and on, mailing-lists ([1] and [2]), fixed bugs
(seems that [3] cannot make this), but couldn't find anything that can be
relevant.
And there is a comment in jbd2_journal_commit_transaction():
/*
* start_this_handle() uses t_outstanding_credits to determine
* the free space in the log, but this counter is changed
* by jbd2_journal_next_log_block() also.
*/
atomic_dec(&commit_transaction->t_outstanding_credits);
And it is there over decades (since "fs/jbd"), but nothing in
jbd2_journal_next_log_block() touches commit_transaction, maybe it is about
"j_free" (since jbd2_log_space_left() uses t_outstanding_credits and j_free).
[1]: https://lkml.org/lkml/2005/1/19/144
[2]: https://lkml.org/lkml/2008/1/11/235
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=559cce698eaf4ccecb2213b2519ea3a0413e5155
Right now I don't have any reasons to think that this is the hardware
problem, but it is SAMSUNG MZ7KM960HAHP-0Z005 (SSD).
Also this is not reproducible, I got it once (a few days ago, and it works well
before for a year or so), and since than I tried to reproduce it without any
luck (with kvm and without).
It will be great if you can give me some hints in debugging this.
Thanks,
Azat.