On Mon, 2022-03-28 at 09:43 +0800, Xiaomeng Tong wrote:
> On Sun, 27 Mar 2022 15:20:42 +0000, Trond Myklebust wrote:
> > On Sun, 2022-03-27 at 16:02 +0800, Xiaomeng Tong wrote:
> > > The bug is here:
> > > if (!server ||
> > > server->pnfs_curr_ld->id != dev->cbd_layout_type) {
> > >
> > > The list iterator value 'server' will *always* be set and non-
> > > NULL
> > > by list_for_each_entry_rcu, so it is incorrect to assume that the
> > > iterator value will be NULL if the list is empty or no element is
> > > found (In fact, it will be a bogus pointer to an invalid struct
> > > object containing the HEAD, which is used for above check at next
> > > outer loop). Otherwise it may bypass the check in theory (iif
> > > server->pnfs_curr_ld->id == dev->cbd_layout_type, 'server' now is
> > > a bogus pointer) and lead to invalid memory access passing the
> > > check.
> > >
> > > To fix the bug, use a new variable 'iter' as the list iterator,
> > > while use the original variable 'server' as a dedicated pointer
> > > to
> > > point to the found element.
> > >
> > > Cc: [email protected]
> > > Fixes: 1be5683b03a76 ("pnfs: CB_NOTIFY_DEVICEID")
> > > Signed-off-by: Xiaomeng Tong <[email protected]>
> > > ---
> > > fs/nfs/callback_proc.c | 9 +++++----
> > > 1 file changed, 5 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
> > > index c343666d9a42..84779785dc8d 100644
> > > --- a/fs/nfs/callback_proc.c
> > > +++ b/fs/nfs/callback_proc.c
> > > @@ -361,7 +361,7 @@ __be32 nfs4_callback_devicenotify(void *argp,
> > > void *resp,
> > > uint32_t i;
> > > __be32 res = 0;
> > > struct nfs_client *clp = cps->clp;
> > > - struct nfs_server *server = NULL;
> > > + struct nfs_server *server = NULL, *iter;
> > >
> > > if (!clp) {
> > > res = cpu_to_be32(NFS4ERR_OP_NOT_IN_SESSION);
> > > @@ -374,10 +374,11 @@ __be32 nfs4_callback_devicenotify(void
> > > *argp,
> > > void *resp,
> > > if (!server ||
> > > server->pnfs_curr_ld->id != dev-
> > > >cbd_layout_type)
> > > {
> > > rcu_read_lock();
> > > - list_for_each_entry_rcu(server, &clp-
> > > > cl_superblocks, client_link)
> > > - if (server->pnfs_curr_ld &&
> > > - server->pnfs_curr_ld->id ==
> > > dev-
> > > > cbd_layout_type) {
> > > + list_for_each_entry_rcu(iter, &clp-
> > > > cl_superblocks, client_link)
> > > + if (iter->pnfs_curr_ld &&
> > > + iter->pnfs_curr_ld->id ==
> > > dev-
> > > > cbd_layout_type) {
> > > rcu_read_unlock();
> > > + server = iter;
> >
> > Hmm... We're not holding any locks on the super block for 'iter'
> > here,
> > so nothing is preventing it from going away while we're.
> >
>
> ok, i am not a 'rcu lock' expert, i will make it hold the
> rcu_read_lock()
> if necessary.
>
> > Given that we really only want a pointer to the struct
> > pnfs_layoutdriver_type anyway, why not just convert the code to
> > save a
> > pointer to that (and do it while holding the rcu_read_lock())?
> >
>
> Maybe it's not that simple. If you only save a pointer to that and
> still
> use 'server' as the list iterator of list_for_each_entry_rcu, there
> could
> be problem.
>
> I.e., if no element found in list_for_each_entry_rcu in the first
> outer
> 'for' loop, and now 'server' is a bogus pointer to an invalid struct,
> and
> continue to go into the second outer 'for' loop, and the check below
> will
> lead to invalid memory access (server->pnfs_curr_ld->id), even can
> potentialy
> be bypassed with crafted data to make the condition false and
> mistakely run
> nfs4_delete_deviceid(server->pnfs_curr_ld, clp, &dev->cbd_dev_id);
> with bogus
> 'server'.
>
> if (!server ||
> server->pnfs_curr_ld->id != dev->cbd_layout_type) {
>
> > The struct pnfs_layoutdriver is always expected to be a statically
> > allocated structure, so it won't go away as long as the pNFS driver
> > module remains loaded.
>
Let's just do the following.
8<-----------------------------------------------
From 7c9d845f0612e5bcd23456a2ec43be8ac43458f1 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Mon, 28 Mar 2022 08:36:34 -0400
Subject: [PATCH] NFSv4/pNFS: Fix another issue with a list iterator pointing
to the head
In nfs4_callback_devicenotify(), if we don't find a matching entry for
the deviceid, we're left with a pointer to 'struct nfs_server' that
actually points to the list of super blocks associated with our struct
nfs_client.
Furthermore, even if we have a valid pointer, nothing pins the super
block, and so the struct nfs_server could end up getting freed while
we're using it.
Since all we want is a pointer to the struct pnfs_layoutdriver_type,
let's skip all the iteration over super blocks, and just use APIs to
find the layout driver directly.
Reported-by: Xiaomeng Tong <[email protected]>
Fixes: 1be5683b03a7 ("pnfs: CB_NOTIFY_DEVICEID")
Signed-off-by: Trond Myklebust <[email protected]>
---
fs/nfs/callback_proc.c | 27 +++++++++------------------
fs/nfs/pnfs.c | 11 +++++++++++
fs/nfs/pnfs.h | 2 ++
3 files changed, 22 insertions(+), 18 deletions(-)
diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
index 39d1ec870d90..c8520284dda7 100644
--- a/fs/nfs/callback_proc.c
+++ b/fs/nfs/callback_proc.c
@@ -358,12 +358,11 @@ __be32 nfs4_callback_devicenotify(void *argp, void *resp,
struct cb_process_state *cps)
{
struct cb_devicenotifyargs *args = argp;
+ const struct pnfs_layoutdriver_type *ld = NULL;
uint32_t i;
__be32 res = 0;
- struct nfs_client *clp = cps->clp;
- struct nfs_server *server = NULL;
- if (!clp) {
+ if (!cps->clp) {
res = cpu_to_be32(NFS4ERR_OP_NOT_IN_SESSION);
goto out;
}
@@ -371,23 +370,15 @@ __be32 nfs4_callback_devicenotify(void *argp, void *resp,
for (i = 0; i < args->ndevs; i++) {
struct cb_devicenotifyitem *dev = &args->devs[i];
- if (!server ||
- server->pnfs_curr_ld->id != dev->cbd_layout_type) {
- rcu_read_lock();
- list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link)
- if (server->pnfs_curr_ld &&
- server->pnfs_curr_ld->id == dev->cbd_layout_type) {
- rcu_read_unlock();
- goto found;
- }
- rcu_read_unlock();
- continue;
+ if (!ld || ld->id != dev->cbd_layout_type) {
+ pnfs_put_layoutdriver(ld);
+ ld = pnfs_find_layoutdriver(dev->cbd_layout_type);
+ if (!ld)
+ continue;
}
-
- found:
- nfs4_delete_deviceid(server->pnfs_curr_ld, clp, &dev->cbd_dev_id);
+ nfs4_delete_deviceid(ld, cps->clp, &dev->cbd_dev_id);
}
-
+ pnfs_put_layoutdriver(ld);
out:
kfree(args->devs);
return res;
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index de318bb5d349..856c962273c7 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -92,6 +92,17 @@ find_pnfs_driver(u32 id)
return local;
}
+const struct pnfs_layoutdriver_type *pnfs_find_layoutdriver(u32 id)
+{
+ return find_pnfs_driver(id);
+}
+
+void pnfs_put_layoutdriver(const struct pnfs_layoutdriver_type *ld)
+{
+ if (ld)
+ module_put(ld->owner);
+}
+
void
unset_pnfs_layoutdriver(struct nfs_server *nfss)
{
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index f4d7548d67b2..07f11489e4e9 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -234,6 +234,8 @@ struct pnfs_devicelist {
extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
+extern const struct pnfs_layoutdriver_type *pnfs_find_layoutdriver(u32 id);
+extern void pnfs_put_layoutdriver(const struct pnfs_layoutdriver_type *ld);
/* nfs4proc.c */
extern size_t max_response_pages(struct nfs_server *server);
--
2.35.1
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
On Mon, 28 Mar 2022 13:24:57 +0000, Trond Myklebust wrote:
> Let's just do the following.
>
> 8<-----------------------------------------------
> From 7c9d845f0612e5bcd23456a2ec43be8ac43458f1 Mon Sep 17 00:00:00 2001
> From: Trond Myklebust <[email protected]>
> Date: Mon, 28 Mar 2022 08:36:34 -0400
> Subject: [PATCH] NFSv4/pNFS: Fix another issue with a list iterator pointing
> to the head
>
> In nfs4_callback_devicenotify(), if we don't find a matching entry for
> the deviceid, we're left with a pointer to 'struct nfs_server' that
> actually points to the list of super blocks associated with our struct
> nfs_client.
> Furthermore, even if we have a valid pointer, nothing pins the super
> block, and so the struct nfs_server could end up getting freed while
> we're using it.
>
> Since all we want is a pointer to the struct pnfs_layoutdriver_type,
> let's skip all the iteration over super blocks, and just use APIs to
> find the layout driver directly.
>
> Reported-by: Xiaomeng Tong <[email protected]>
> Fixes: 1be5683b03a7 ("pnfs: CB_NOTIFY_DEVICEID")
> Signed-off-by: Trond Myklebust <[email protected]>
> ---
> fs/nfs/callback_proc.c | 27 +++++++++------------------
> fs/nfs/pnfs.c | 11 +++++++++++
> fs/nfs/pnfs.h | 2 ++
> 3 files changed, 22 insertions(+), 18 deletions(-)
>
> diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
> index 39d1ec870d90..c8520284dda7 100644
> --- a/fs/nfs/callback_proc.c
> +++ b/fs/nfs/callback_proc.c
> @@ -358,12 +358,11 @@ __be32 nfs4_callback_devicenotify(void *argp, void *resp,
> struct cb_process_state *cps)
> {
> struct cb_devicenotifyargs *args = argp;
> + const struct pnfs_layoutdriver_type *ld = NULL;
> uint32_t i;
> __be32 res = 0;
> - struct nfs_client *clp = cps->clp;
> - struct nfs_server *server = NULL;
>
> - if (!clp) {
> + if (!cps->clp) {
> res = cpu_to_be32(NFS4ERR_OP_NOT_IN_SESSION);
> goto out;
> }
> @@ -371,23 +370,15 @@ __be32 nfs4_callback_devicenotify(void *argp, void *resp,
> for (i = 0; i < args->ndevs; i++) {
> struct cb_devicenotifyitem *dev = &args->devs[i];
>
> - if (!server ||
> - server->pnfs_curr_ld->id != dev->cbd_layout_type) {
> - rcu_read_lock();
> - list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link)
> - if (server->pnfs_curr_ld &&
> - server->pnfs_curr_ld->id == dev->cbd_layout_type) {
> - rcu_read_unlock();
> - goto found;
> - }
> - rcu_read_unlock();
> - continue;
> + if (!ld || ld->id != dev->cbd_layout_type) {
> + pnfs_put_layoutdriver(ld);
> + ld = pnfs_find_layoutdriver(dev->cbd_layout_type);
> + if (!ld)
> + continue;
> }
> -
> - found:
> - nfs4_delete_deviceid(server->pnfs_curr_ld, clp, &dev->cbd_dev_id);
> + nfs4_delete_deviceid(ld, cps->clp, &dev->cbd_dev_id);
> }
> -
> + pnfs_put_layoutdriver(ld);
> out:
> kfree(args->devs);
> return res;
> diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
> index de318bb5d349..856c962273c7 100644
> --- a/fs/nfs/pnfs.c
> +++ b/fs/nfs/pnfs.c
> @@ -92,6 +92,17 @@ find_pnfs_driver(u32 id)
> return local;
> }
>
> +const struct pnfs_layoutdriver_type *pnfs_find_layoutdriver(u32 id)
> +{
> + return find_pnfs_driver(id);
> +}
> +
> +void pnfs_put_layoutdriver(const struct pnfs_layoutdriver_type *ld)
> +{
> + if (ld)
> + module_put(ld->owner);
> +}
> +
> void
> unset_pnfs_layoutdriver(struct nfs_server *nfss)
> {
> diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
> index f4d7548d67b2..07f11489e4e9 100644
> --- a/fs/nfs/pnfs.h
> +++ b/fs/nfs/pnfs.h
> @@ -234,6 +234,8 @@ struct pnfs_devicelist {
>
> extern int pnfs_register_layoutdriver(struct pnfs_layoutdriver_type *);
> extern void pnfs_unregister_layoutdriver(struct pnfs_layoutdriver_type *);
> +extern const struct pnfs_layoutdriver_type *pnfs_find_layoutdriver(u32 id);
> +extern void pnfs_put_layoutdriver(const struct pnfs_layoutdriver_type *ld);
>
> /* nfs4proc.c */
> extern size_t max_response_pages(struct nfs_server *server);
> --
Thank you, i have resend a PATCH v2 with fix as you suggested, and also with
some changes, please check it.
--
Xiaomeng Tong