Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
MIME-Version: 1.0
References: <20220519191311.17119-1-logang@deltatee.com> <20220519191311.17119-13-logang@deltatee.com>
 <62b09487-9223-db3d-2165-789a51230060@molgen.mpg.de>
In-Reply-To: <62b09487-9223-db3d-2165-789a51230060@molgen.mpg.de>
From:   Song Liu <song@kernel.org>
Date:   Sun, 22 May 2022 23:47:41 -0700
Message-ID: <CAPhsuW5FKh7VKgU+=eU4_ad4btFMwU7ymMNMZ5TZVQbv82MaHg@mail.gmail.com>
Subject: Re: [PATCH v1 12/15] md/raid5-cache: Add RCU protection to conf->log accesses
To:     Donald Buczek <buczek@molgen.mpg.de>
Cc:     Logan Gunthorpe <logang@deltatee.com>,
        open list <linux-kernel@vger.kernel.org>,
        linux-raid <linux-raid@vger.kernel.org>,
        Christoph Hellwig <hch@infradead.org>,
        Guoqing Jiang <guoqing.jiang@linux.dev>,
        Xiao Ni <xni@redhat.com>, Stephen Bates <sbates@raithlin.com>,
        Martin Oliveira <Martin.Oliveira@eideticom.com>,
        David Sloan <David.Sloan@eideticom.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Sun, May 22, 2022 at 12:32 AM Donald Buczek <buczek@molgen.mpg.de> wrote=
:
>
> On 19.05.22 21:13, Logan Gunthorpe wrote:
> > The mdadm test 21raid5cache randomly fails with NULL pointer accesses
> > conf->log when run repeatedly. conf->log was sort of protected with
> > a RCU, but most dereferences were not done with the correct functions.
> >
> > Add rcu_read_locks() and rcu_access_pointers() to the appropriate
> > places.
> >
> > Signed-off-by: Logan Gunthorpe <logang@deltatee.com>

[...]

> > diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h
> > index f26e6f4c7f9a..24b4dbd5b25c 100644
> > --- a/drivers/md/raid5-log.h
> > +++ b/drivers/md/raid5-log.h
> > @@ -58,7 +58,7 @@ static inline int log_stripe(struct stripe_head *sh, =
struct stripe_head_state *s
> >   {
> >       struct r5conf *conf =3D sh->raid_conf;
> >
> > -     if (conf->log) {
> > +     if (rcu_access_pointer(conf->log)) {
>
>
> A problem here is that `struct r5l_log` of `conf->log` is private to raid=
5-cache.c and gcc below version 10 (wrongly) regards the `typeof(*p) *local=
` declaration of __rcu_access_pointer as a dereference:
>
>    CC      drivers/md/raid5.o
>
> In file included from ./include/linux/rculist.h:11:0,
>
>                   from ./include/linux/dcache.h:8,
>
>                   from ./include/linux/fs.h:8,
>
>                   from ./include/linux/highmem.h:5,
>
>                   from ./include/linux/bvec.h:10,
>
>                   from ./include/linux/blk_types.h:10,
>
>                   from ./include/linux/blkdev.h:9,
>
>                   from drivers/md/raid5.c:38:
>
> drivers/md/raid5-log.h: In function =E2=80=98log_stripe=E2=80=99:
>
> ./include/linux/rcupdate.h:384:9: error: dereferencing pointer to incompl=
ete type =E2=80=98struct r5l_log=E2=80=99
>
>    typeof(*p) *local =3D (typeof(*p) *__force)READ_ONCE(p); \
>
>           ^
>
> ./include/linux/rcupdate.h:495:31: note: in expansion of macro =E2=80=98_=
_rcu_access_pointer=E2=80=99
>
>   #define rcu_access_pointer(p) __rcu_access_pointer((p), __UNIQUE_ID(rcu=
), __rcu)
>
>                                 ^~~~~~~~~~~~~~~~~~~~
>
> drivers/md/raid5-log.h:61:6: note: in expansion of macro =E2=80=98rcu_acc=
ess_pointer=E2=80=99
>
>    if (rcu_access_pointer(conf->log)) {
>
>        ^~~~~~~~~~~~~~~~~~
>
> make[2]: *** [scripts/Makefile.build:288: drivers/md/raid5.o] Error 1
>
> make[1]: *** [scripts/Makefile.build:550: drivers/md] Error 2
>
> make: *** [Makefile:1834: drivers] Error 2

This is annoying.. And there are a few other cases in raid5-log.h and
raid5.c.

Maybe we should move the definition of r5l_log to raid5-log.h?

Thanks,
Song


>
>
> See https://godbolt.org/z/TPP8MdKbc to test compiler versions with this c=
onstruct.
>
> Best
>
>    Donald
>
>
> >               if (!test_bit(STRIPE_R5C_CACHING, &sh->state)) {
> >                       /* writing out phase */
> >                       if (s->waiting_extra_page)
> > @@ -79,7 +79,7 @@ static inline void log_stripe_write_finished(struct s=
tripe_head *sh)
> >   {
> >       struct r5conf *conf =3D sh->raid_conf;
> >
> > -     if (conf->log)
> > +     if (rcu_access_pointer(conf->log))
> >               r5l_stripe_write_finished(sh);
> >       else if (raid5_has_ppl(conf))
> >               ppl_stripe_write_finished(sh);
> > @@ -87,7 +87,7 @@ static inline void log_stripe_write_finished(struct s=
tripe_head *sh)
> >
> >   static inline void log_write_stripe_run(struct r5conf *conf)
> >   {
> > -     if (conf->log)
> > +     if (rcu_access_pointer(conf->log))
> >               r5l_write_stripe_run(conf);
> >       else if (raid5_has_ppl(conf))
> >               ppl_write_stripe_run(conf);
> > @@ -95,7 +95,7 @@ static inline void log_write_stripe_run(struct r5conf=
 *conf)
> >
> >   static inline void log_flush_stripe_to_raid(struct r5conf *conf)
> >   {
> > -     if (conf->log)
> > +     if (rcu_access_pointer(conf->log))
> >               r5l_flush_stripe_to_raid(conf);
> >       else if (raid5_has_ppl(conf))
> >               ppl_write_stripe_run(conf);
> > @@ -105,7 +105,7 @@ static inline int log_handle_flush_request(struct r=
5conf *conf, struct bio *bio)
> >   {
> >       int ret =3D -ENODEV;
> >
> > -     if (conf->log)
> > +     if (rcu_access_pointer(conf->log))
> >               ret =3D r5l_handle_flush_request(conf, bio);
> >       else if (raid5_has_ppl(conf))
> >               ret =3D ppl_handle_flush_request(bio);
> > @@ -115,7 +115,7 @@ static inline int log_handle_flush_request(struct r=
5conf *conf, struct bio *bio)
> >
> >   static inline void log_quiesce(struct r5conf *conf, int quiesce)
> >   {
> > -     if (conf->log)
> > +     if (rcu_access_pointer(conf->log))
> >               r5l_quiesce(conf, quiesce);
> >       else if (raid5_has_ppl(conf))
> >               ppl_quiesce(conf, quiesce);
> > @@ -123,7 +123,7 @@ static inline void log_quiesce(struct r5conf *conf,=
 int quiesce)
> >
> >   static inline void log_exit(struct r5conf *conf)
> >   {
> > -     if (conf->log)
> > +     if (rcu_access_pointer(conf->log))
> >               r5l_exit_log(conf);
> >       else if (raid5_has_ppl(conf))
> >               ppl_exit_log(conf);
> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c

[...]