The __log_wait_for_space function sits in a loop checkpointing transactions
until there is sufficient space free in the journal. However, if there are
no transactions to be processed (e.g. because the free space calculation is
wrong due to a corrupted filesystem) it will never progress.
Check for space being required when no transactions are outstanding and
abort the journal instead of endlessly looping.
This patch fixes the bug reported by Sami Liedes at:
http://bugzilla.kernel.org/show_bug.cgi?id=10976
Signed-off-by: Duane Griffin <[email protected]>
Tested-by: Sami Liedes <[email protected]>
---
diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
index a5432bb..9fac177 100644
--- a/fs/jbd/checkpoint.c
+++ b/fs/jbd/checkpoint.c
@@ -126,14 +126,29 @@ void __log_wait_for_space(journal_t *journal)
/*
* Test again, another process may have checkpointed while we
- * were waiting for the checkpoint lock
+ * were waiting for the checkpoint lock. If there are no
+ * outstanding transactions there is nothing to checkpoint and
+ * we can't make progress. Abort the journal in this case.
*/
spin_lock(&journal->j_state_lock);
+ spin_lock(&journal->j_list_lock);
nblocks = jbd_space_needed(journal);
if (__log_space_left(journal) < nblocks) {
+ int chkpt = journal->j_checkpoint_transactions != NULL;
+
+ spin_unlock(&journal->j_list_lock);
spin_unlock(&journal->j_state_lock);
- log_do_checkpoint(journal);
+ if (chkpt) {
+ log_do_checkpoint(journal);
+ } else {
+ printk(KERN_ERR "%s: no transactions\n",
+ __func__);
+ journal_abort(journal, 0);
+ }
+
spin_lock(&journal->j_state_lock);
+ } else {
+ spin_unlock(&journal->j_list_lock);
}
mutex_unlock(&journal->j_checkpoint_mutex);
}
The __jbd2_log_wait_for_space function sits in a loop checkpointing
transactions until there is sufficient space free in the journal. However,
if there are no transactions to be processed (e.g. because the free space
calculation is wrong due to a corrupted filesystem) it will never progress.
Check for space being required when no transactions are outstanding and
abort the journal instead of endlessly looping.
This patch fixes the bug reported by Sami Liedes at:
http://bugzilla.kernel.org/show_bug.cgi?id=10976
Signed-off-by: Duane Griffin <[email protected]>
---
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 91389c8..af4651b 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -126,14 +126,29 @@ void __jbd2_log_wait_for_space(journal_t *journal)
/*
* Test again, another process may have checkpointed while we
- * were waiting for the checkpoint lock
+ * were waiting for the checkpoint lock. If there are no
+ * outstanding transactions there is nothing to checkpoint and
+ * we can't make progress. Abort the journal in this case.
*/
spin_lock(&journal->j_state_lock);
+ spin_lock(&journal->j_list_lock);
nblocks = jbd_space_needed(journal);
if (__jbd2_log_space_left(journal) < nblocks) {
+ int chkpt = journal->j_checkpoint_transactions != NULL;
+
+ spin_unlock(&journal->j_list_lock);
spin_unlock(&journal->j_state_lock);
- jbd2_log_do_checkpoint(journal);
+ if (chkpt) {
+ jbd2_log_do_checkpoint(journal);
+ } else {
+ printk(KERN_ERR "%s: no transactions\n",
+ __func__);
+ jbd2_journal_abort(journal, 0);
+ }
+
spin_lock(&journal->j_state_lock);
+ } else {
+ spin_unlock(&journal->j_list_lock);
}
mutex_unlock(&journal->j_checkpoint_mutex);
}
On Tue, 5 Aug 2008 00:51:34 +0100 "Duane Griffin" <[email protected]> wrote:
> The __log_wait_for_space function sits in a loop checkpointing transactions
> until there is sufficient space free in the journal. However, if there are
> no transactions to be processed (e.g. because the free space calculation is
> wrong due to a corrupted filesystem) it will never progress.
>
> Check for space being required when no transactions are outstanding and
> abort the journal instead of endlessly looping.
>
> This patch fixes the bug reported by Sami Liedes at:
> http://bugzilla.kernel.org/show_bug.cgi?id=10976
>
> Signed-off-by: Duane Griffin <[email protected]>
> Tested-by: Sami Liedes <[email protected]>
> ---
> diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
> index a5432bb..9fac177 100644
> --- a/fs/jbd/checkpoint.c
> +++ b/fs/jbd/checkpoint.c
> @@ -126,14 +126,29 @@ void __log_wait_for_space(journal_t *journal)
>
> /*
> * Test again, another process may have checkpointed while we
> - * were waiting for the checkpoint lock
> + * were waiting for the checkpoint lock. If there are no
> + * outstanding transactions there is nothing to checkpoint and
> + * we can't make progress. Abort the journal in this case.
> */
> spin_lock(&journal->j_state_lock);
> + spin_lock(&journal->j_list_lock);
> nblocks = jbd_space_needed(journal);
> if (__log_space_left(journal) < nblocks) {
> + int chkpt = journal->j_checkpoint_transactions != NULL;
> +
> + spin_unlock(&journal->j_list_lock);
> spin_unlock(&journal->j_state_lock);
> - log_do_checkpoint(journal);
> + if (chkpt) {
> + log_do_checkpoint(journal);
> + } else {
> + printk(KERN_ERR "%s: no transactions\n",
> + __func__);
> + journal_abort(journal, 0);
> + }
> +
> spin_lock(&journal->j_state_lock);
> + } else {
> + spin_unlock(&journal->j_list_lock);
> }
> mutex_unlock(&journal->j_checkpoint_mutex);
> }
I don't expect that the additional taking of j_list_lock in here does
anything useful.
Plus.. after j_list_lock has been dropped, new transactions could
theoretically appear at journal->j_checkpoint_transactions, so we
_could_ reclaim more journal space. But a) that probably couldn't
happen due to ->j_state_lock and lots of other things and b) it's
hopelessly theoretical even if it _could_ happen, methinks. Just
sayin'..
2008/8/5 Andrew Morton <[email protected]>:
> On Tue, 5 Aug 2008 00:51:34 +0100 "Duane Griffin" <[email protected]> wrote:
>
>> The __log_wait_for_space function sits in a loop checkpointing transactions
>> until there is sufficient space free in the journal. However, if there are
>> no transactions to be processed (e.g. because the free space calculation is
>> wrong due to a corrupted filesystem) it will never progress.
>>
>> Check for space being required when no transactions are outstanding and
>> abort the journal instead of endlessly looping.
>>
>> This patch fixes the bug reported by Sami Liedes at:
>> http://bugzilla.kernel.org/show_bug.cgi?id=10976
>>
>> Signed-off-by: Duane Griffin <[email protected]>
>> Tested-by: Sami Liedes <[email protected]>
>> ---
>> diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
>> index a5432bb..9fac177 100644
>> --- a/fs/jbd/checkpoint.c
>> +++ b/fs/jbd/checkpoint.c
>> @@ -126,14 +126,29 @@ void __log_wait_for_space(journal_t *journal)
>>
>> /*
>> * Test again, another process may have checkpointed while we
>> - * were waiting for the checkpoint lock
>> + * were waiting for the checkpoint lock. If there are no
>> + * outstanding transactions there is nothing to checkpoint and
>> + * we can't make progress. Abort the journal in this case.
>> */
>> spin_lock(&journal->j_state_lock);
>> + spin_lock(&journal->j_list_lock);
>> nblocks = jbd_space_needed(journal);
>> if (__log_space_left(journal) < nblocks) {
>> + int chkpt = journal->j_checkpoint_transactions != NULL;
>> +
>> + spin_unlock(&journal->j_list_lock);
>> spin_unlock(&journal->j_state_lock);
>> - log_do_checkpoint(journal);
>> + if (chkpt) {
>> + log_do_checkpoint(journal);
>> + } else {
>> + printk(KERN_ERR "%s: no transactions\n",
>> + __func__);
>> + journal_abort(journal, 0);
>> + }
>> +
>> spin_lock(&journal->j_state_lock);
>> + } else {
>> + spin_unlock(&journal->j_list_lock);
>> }
>> mutex_unlock(&journal->j_checkpoint_mutex);
>> }
>
> I don't expect that the additional taking of j_list_lock in here does
> anything useful.
>
> Plus.. after j_list_lock has been dropped, new transactions could
> theoretically appear at journal->j_checkpoint_transactions, so we
> _could_ reclaim more journal space. But a) that probably couldn't
> happen due to ->j_state_lock and lots of other things and b) it's
> hopelessly theoretical even if it _could_ happen, methinks. Just
> sayin'..
Fair enough. I was just trying to be extra careful in taking the lock,
so I'm happy to drop it if you think it is safe. It will simplify the
patch significantly.
Cheers,
Duane.
--
"I never could learn to drink that blood and call it wine" - Bob Dylan
Hi,
On Tue, 2008-08-05 at 00:51 +0100, Duane Griffin wrote:
> The __log_wait_for_space function sits in a loop checkpointing transactions
> until there is sufficient space free in the journal. However, if there are
> no transactions to be processed (e.g. because the free space calculation is
> wrong due to a corrupted filesystem) it will never progress.
>
> Check for space being required when no transactions are outstanding and
> abort the journal instead of endlessly looping.
I'm not sure this is the right fix --- it seems like we're fixing the
symptoms, not the problem.
The journal free space fields are reset in journal_reset() when we load
the journal, so we can't get this situation of j_free being insufficient
on an idle filesystem unless the main journal start/end pointers are
corrupt.
Surely we'd be better off detecting this in the first place at mount
time, not later on during checkpoint?
Cheers,
Stephen
2008/8/5 Stephen C. Tweedie <[email protected]>:
> On Tue, 2008-08-05 at 00:51 +0100, Duane Griffin wrote:
>> The __log_wait_for_space function sits in a loop checkpointing transactions
>> until there is sufficient space free in the journal. However, if there are
>> no transactions to be processed (e.g. because the free space calculation is
>> wrong due to a corrupted filesystem) it will never progress.
>>
>> Check for space being required when no transactions are outstanding and
>> abort the journal instead of endlessly looping.
>
> I'm not sure this is the right fix --- it seems like we're fixing the
> symptoms, not the problem.
>
> The journal free space fields are reset in journal_reset() when we load
> the journal, so we can't get this situation of j_free being insufficient
> on an idle filesystem unless the main journal start/end pointers are
> corrupt.
>
> Surely we'd be better off detecting this in the first place at mount
> time, not later on during checkpoint?
Sounds sensible. In fact I've got another patch, waiting for feedback
from the reporter, that adds some very basic validation there (i.e.
first > 0 && last >= first). Not enough, I suspect. I guess we could
do much better?
> Cheers,
> Stephen
Cheers,
Duane.
--
"I never could learn to drink that blood and call it wine" - Bob Dylan
Hi,
On Thu, 2008-08-07 at 01:47 +0100, Duane Griffin wrote:
> > Surely we'd be better off detecting this in the first place at mount
> > time, not later on during checkpoint?
>
> Sounds sensible. In fact I've got another patch, waiting for feedback
> from the reporter, that adds some very basic validation there (i.e.
> first > 0 && last >= first). Not enough, I suspect. I guess we could
> do much better?
Right: in journal.c we initialise the maximum size of a transaction to
journal->j_max_transaction_buffers = journal->j_maxlen / 4;
(the logic being that we need the journal to be able to hold an absolute
minimum of one full transaction being checkpointed, one being committed,
and one being live concurrently for the transaction engine to work
correctly, which gives three outstanding transactions; we up that to
four to protect against rounding errors and to ensure space for the
sequence and commit blocks that take up log space in addition to the
journaled buffers themselves.)
If, during journal load, that's not enough for a minimum-sized single
update, we'll never be able to start some transactions, so that would be
a good place to check that we're starting off with a large enough
journal.
--Stephen