Hello,
We're running Linux kernel 4.14.x and our systems occasionally suffer a
bug which is already fixed:
https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
applies cleanly. The two files "fs/jbd2/checkpoint.c" and
"fs/jbd2/journal.c" seem pretty identical in the affected sections
compared to kernel 5.4 where we have this bugfix already applied.
Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
system in 4.14 and 5.4 fundamentally different and this would lead to
data loss or other problems?
Thank you.
Best regards.
--Ivan
On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
> Hello,
>
> We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
> which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
>
> This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
> applies cleanly. The two files "fs/jbd2/checkpoint.c" and
> "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
> to kernel 5.4 where we have this bugfix already applied.
>
> Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
> that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
> system in 4.14 and 5.4 fundamentally different and this would lead to data
> loss or other problems?
The commit was over two years ago, so my memory is not going to be
perfect. However, Jan had made a comment suggesting the approach in
this commit because it should be easier to backport into older stble
kernels[1].
"Since proper locking change is going to be a bit more involved, can you
perhaps fix this deadlock by just dropping j_checkpoint_mutex in
log_do_checkpoint() when we are going to wait for transaction commit. I've
checked and that should be fine and that is going to be much easier change
to backport into stable kernels..."
[1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2
So I suspect it was just that I failed to remember to add a "Cc:
[email protected]" and so it was never automatically backported into
4.14 or 4.19.
Do you have a reliable reproduction which is triggering the deadlock
on your kernels? If so, have you tried applying the patch and does it
make the problem go away for you?
Cheers,
- Ted
Out of thousand machines, one would trigger the problem about every 1 to
10 days. Some machines trigger the problem much often than others. So I
can say that we have a way to verify quickly if applying the patch will
fix this for us.
The most important question is: Is it safe to apply the patch on
production machines with kernel 4.14?
We can't risk data loss. And I lack the expertise to asses what risks
this small patch brings.
Best regards.
--Ivan
On 8.7.2021 г. 2:52, Theodore Ts'o wrote:
> On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
>> Hello,
>>
>> We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
>> which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
>>
>> This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
>> applies cleanly. The two files "fs/jbd2/checkpoint.c" and
>> "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
>> to kernel 5.4 where we have this bugfix already applied.
>>
>> Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
>> that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
>> system in 4.14 and 5.4 fundamentally different and this would lead to data
>> loss or other problems?
> The commit was over two years ago, so my memory is not going to be
> perfect. However, Jan had made a comment suggesting the approach in
> this commit because it should be easier to backport into older stble
> kernels[1].
>
> "Since proper locking change is going to be a bit more involved, can you
> perhaps fix this deadlock by just dropping j_checkpoint_mutex in
> log_do_checkpoint() when we are going to wait for transaction commit. I've
> checked and that should be fine and that is going to be much easier change
> to backport into stable kernels..."
>
> [1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2
>
> So I suspect it was just that I failed to remember to add a "Cc:
> [email protected]" and so it was never automatically backported into
> 4.14 or 4.19.
>
> Do you have a reliable reproduction which is triggering the deadlock
> on your kernels? If so, have you tried applying the patch and does it
> make the problem go away for you?
>
> Cheers,
>
> - Ted
On Thu 08-07-21 06:45:35, Ivan Zahariev wrote:
> Out of thousand machines, one would trigger the problem about every 1 to 10
> days. Some machines trigger the problem much often than others. So I can say
> that we have a way to verify quickly if applying the patch will fix this for
> us.
>
> The most important question is: Is it safe to apply the patch on production
> machines with kernel 4.14?
>
> We can't risk data loss. And I lack the expertise to asses what risks this
> small patch brings.
The fix should work correctly even for older kernels. I'm not aware of any
changes in this area in the past that could conflict...
Honza
> On 8.7.2021 г. 2:52, Theodore Ts'o wrote:
> > On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
> > > Hello,
> > >
> > > We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
> > > which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
> > >
> > > This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
> > > applies cleanly. The two files "fs/jbd2/checkpoint.c" and
> > > "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
> > > to kernel 5.4 where we have this bugfix already applied.
> > >
> > > Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
> > > that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
> > > system in 4.14 and 5.4 fundamentally different and this would lead to data
> > > loss or other problems?
> > The commit was over two years ago, so my memory is not going to be
> > perfect. However, Jan had made a comment suggesting the approach in
> > this commit because it should be easier to backport into older stble
> > kernels[1].
> >
> > "Since proper locking change is going to be a bit more involved, can you
> > perhaps fix this deadlock by just dropping j_checkpoint_mutex in
> > log_do_checkpoint() when we are going to wait for transaction commit. I've
> > checked and that should be fine and that is going to be much easier change
> > to backport into stable kernels..."
> >
> > [1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2
> >
> > So I suspect it was just that I failed to remember to add a "Cc:
> > [email protected]" and so it was never automatically backported into
> > 4.14 or 4.19.
> >
> > Do you have a reliable reproduction which is triggering the deadlock
> > on your kernels? If so, have you tried applying the patch and does it
> > make the problem go away for you?
> >
> > Cheers,
> >
> > - Ted
--
Jan Kara <[email protected]>
SUSE Labs, CR