From: Li Nan <[email protected]>
Li Nan (2):
md/raid10: fix taks hung in raid10d
md/raid10: fix null-ptr-deref in raid10_sync_request
drivers/md/raid10.c | 26 +++++++++++++++++---------
1 file changed, 17 insertions(+), 9 deletions(-)
--
2.31.1
From: Li Nan <[email protected]>
init_resync() init mempool and set conf->have_replacemnt at the begaining
of sync, close_sync() free the mempool when sync is completed.
After commit 7e83ccbecd60 ("md/raid10: Allow skipping recovery when clean
arrays are assembled"), recovery might skipped and init_resync() is called
but close_sync() is not. null-ptr-deref occurs as below:
1) creat a array, wait for resync to complete, mddev->recovery_cp is set
to MaxSector.
2) recovery is woken and it is skipped. conf->have_replacement is set to
0 in init_resync(). close_sync() not called.
3) some io errors and rdev A is set to WantReplacement.
4) a new device is added and set to A's replacement.
5) recovery is woken, A have replacement, but conf->have_replacemnt is
0. r10bio->dev[i].repl_bio will not be alloced and null-ptr-deref
occurs.
Fix it by not init_resync() if recovery skipped.
Fixes: 7e83ccbecd60 md/raid10: Allow skipping recovery when clean arrays are assembled")
Signed-off-by: Li Nan <[email protected]>
---
drivers/md/raid10.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index db9ee3b637d6..9e0e7bf524aa 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -3297,10 +3297,6 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
sector_t chunk_mask = conf->geo.chunk_mask;
int page_idx = 0;
- if (!mempool_initialized(&conf->r10buf_pool))
- if (init_resync(conf))
- return 0;
-
/*
* Allow skipping a full rebuild for incremental assembly
* of a clean array, like RAID1 does.
@@ -3316,6 +3312,10 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
return mddev->dev_sectors - sector_nr;
}
+ if (!mempool_initialized(&conf->r10buf_pool))
+ if (init_resync(conf))
+ return 0;
+
skipped:
max_sector = mddev->dev_sectors;
if (test_bit(MD_RECOVERY_SYNC, &mddev->recovery) ||
--
2.31.1
From: Li Nan <[email protected]>
commit fe630de009d0 ("md/raid10: avoid deadlock on recovery.") allowed
normal io and sync io to exist at the same time. Task hung will occur as
below:
T1 T2 T3 T4
raid10d
handle_read_error
allow_barrier
conf->nr_pending--
-> 0
//submit sync io
raid10_sync_request
raise_barrier
->will not be blocked
...
//submit to drivers
raid10_read_request
wait_barrier
conf->nr_pending++
-> 1
//retry read fail
raid10_end_read_request
reschedule_retry
add to retry_list
conf->nr_queued++
-> 1
//sync io fail
end_sync_read
__end_sync_read
reschedule_retry
add to retry_list
conf->nr_queued++
-> 2
...
handle_read_error
get form retry_list
conf->nr_queued--
freeze_array
wait nr_pending == nr_queued+1
->1 ->2
//task hung
retry read and sync io will be added to retry_list(nr_queued->2) if they
fails. raid10d() called handle_read_error() and hung in freeze_array().
nr_queued will not decrease because raid10d is blocked, nr_pending will
not increase because conf->barrier is not released.
Fix it by moving allow_barrier() after raid10_read_request().
raise_barrier() will wait for nr_waiting to become 0. Therefore, sync io
and regular io will not be issued at the same time.
We also removed the check of nr_queued. It can be 0 but don't need to be
blocked. MD_RECOVERY_RUNNING always is set after this patch, because all
sync io is waitting in raise_barrier(), remove it, too.
Fixes: fe630de009d0 ("md/raid10: avoid deadlock on recovery.")
Signed-off-by: Li Nan <[email protected]>
---
drivers/md/raid10.c | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 6c66357f92f5..db9ee3b637d6 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -995,11 +995,15 @@ static bool stop_waiting_barrier(struct r10conf *conf)
(!bio_list_empty(&bio_list[0]) || !bio_list_empty(&bio_list[1])))
return true;
- /* move on if recovery thread is blocked by us */
- if (conf->mddev->thread->tsk == current &&
- test_bit(MD_RECOVERY_RUNNING, &conf->mddev->recovery) &&
- conf->nr_queued > 0)
+ /*
+ * move on if io is issued from raid10d(), nr_pending is not released
+ * from original io(see handle_read_error()). All raise barrier is
+ * blocked until this io is done.
+ */
+ if (conf->mddev->thread->tsk == current) {
+ WARN_ON_ONCE(atomic_read(&conf->nr_pending) == 0);
return true;
+ }
return false;
}
@@ -2978,9 +2982,13 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)
md_error(mddev, rdev);
rdev_dec_pending(rdev, mddev);
- allow_barrier(conf);
r10_bio->state = 0;
raid10_read_request(mddev, r10_bio->master_bio, r10_bio);
+ /*
+ * allow_barrier after re-submit to ensure no sync io
+ * can be issued while regular io pending.
+ */
+ allow_barrier(conf);
}
static void handle_write_completed(struct r10conf *conf, struct r10bio *r10_bio)
--
2.31.1
Hi,
friendly ping ...
Thanks,
Nan
在 2023/2/22 12:09, [email protected] 写道:
> From: Li Nan <[email protected]>
>
> Li Nan (2):
> md/raid10: fix taks hung in raid10d
> md/raid10: fix null-ptr-deref in raid10_sync_request
>
> drivers/md/raid10.c | 26 +++++++++++++++++---------
> 1 file changed, 17 insertions(+), 9 deletions(-)
>
On Mon, Mar 13, 2023 at 5:00 AM Li Nan <[email protected]> wrote:
>
> Hi,
>
> friendly ping ...
>
> Thanks,
> Nan
Sorry for the delay. Applied the set to md-next (as these are not urgent).
Thanks,
Song
>
> 在 2023/2/22 12:09, [email protected] 写道:
> > From: Li Nan <[email protected]>
> >
> > Li Nan (2):
> > md/raid10: fix taks hung in raid10d
> > md/raid10: fix null-ptr-deref in raid10_sync_request
> >
> > drivers/md/raid10.c | 26 +++++++++++++++++---------
> > 1 file changed, 17 insertions(+), 9 deletions(-)
> >
>