Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: i6a694271:Fastmail
Date:   Thu, 13 Jul 2023 17:42:32 -0600
From:   Daniel Xu <dxu@dxuuu.xyz>
To:     Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc:     Andrii Nakryiko <andrii@kernel.org>,
        Alexei Starovoitov <ast@kernel.org>,
        Florian Westphal <fw@strlen.de>,
        "David S. Miller" <davem@davemloft.net>,
        Pablo Neira Ayuso <pablo@netfilter.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Jozsef Kadlecsik <kadlec@netfilter.org>,
        Martin KaFai Lau <martin.lau@linux.dev>,
        Song Liu <song@kernel.org>, Yonghong Song <yhs@fb.com>,
        John Fastabend <john.fastabend@gmail.com>,
        KP Singh <kpsingh@kernel.org>,
        Stanislav Fomichev <sdf@google.com>,
        Hao Luo <haoluo@google.com>, Jiri Olsa <jolsa@kernel.org>,
        bpf <bpf@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>,
        netfilter-devel <netfilter-devel@vger.kernel.org>,
        coreteam@netfilter.org,
        Network Development <netdev@vger.kernel.org>,
        David Ahern <dsahern@kernel.org>
Subject: Re: [PATCH bpf-next v4 2/6] netfilter: bpf: Support
 BPF_F_NETFILTER_IP_DEFRAG in netfilter link
Message-ID: <t6wypww537golmoosbikfuombrqq555fh5mbycwl4whto6joo4@hcqlospkgqyr>
References: <cover.1689203090.git.dxu@dxuuu.xyz>
 <d3b0ff95c58356192ea3b50824f8cdbf02c354e3.1689203090.git.dxu@dxuuu.xyz>
 <CAADnVQKKfEtZYZxihxvG3aQ34E1m95qTZ=jTD7yd0qvOASpAjQ@mail.gmail.com>
 <kwiwaeaijj6sxwz5fhtxyoquhz2kpujbsbeajysufgmdjgyx5c@f6lqrd23xr5f>
 <CAADnVQLcAoN5z+HD_44UKgJJc6t5TPW8+Ai9We0qJpau4NtEzA@mail.gmail.com>
 <wltfmammaf5g4gumsbna4kmwo6dtd24g472o7kgkug42dhwcy2@32fmd7q6kvg4>
 <CAADnVQJQZ2jQSWByVvi3N2ZOoL0XDSJzx5biSVvq=inS7OSW7A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAADnVQJQZ2jQSWByVvi3N2ZOoL0XDSJzx5biSVvq=inS7OSW7A@mail.gmail.com>
Precedence: bulk

On Thu, Jul 13, 2023 at 04:10:03PM -0700, Alexei Starovoitov wrote:
> On Wed, Jul 12, 2023 at 9:33 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> >
> > On Wed, Jul 12, 2023 at 06:26:13PM -0700, Alexei Starovoitov wrote:
> > > On Wed, Jul 12, 2023 at 6:22 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > > >
> > > > Hi Alexei,
> > > >
> > > > On Wed, Jul 12, 2023 at 05:43:49PM -0700, Alexei Starovoitov wrote:
> > > > > On Wed, Jul 12, 2023 at 4:44 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > > > > > +#if IS_ENABLED(CONFIG_NF_DEFRAG_IPV6)
> > > > > > +       case NFPROTO_IPV6:
> > > > > > +               rcu_read_lock();
> > > > > > +               v6_hook = rcu_dereference(nf_defrag_v6_hook);
> > > > > > +               if (!v6_hook) {
> > > > > > +                       rcu_read_unlock();
> > > > > > +                       err = request_module("nf_defrag_ipv6");
> > > > > > +                       if (err)
> > > > > > +                               return err < 0 ? err : -EINVAL;
> > > > > > +
> > > > > > +                       rcu_read_lock();
> > > > > > +                       v6_hook = rcu_dereference(nf_defrag_v6_hook);
> > > > > > +                       if (!v6_hook) {
> > > > > > +                               WARN_ONCE(1, "nf_defrag_ipv6_hooks bad registration");
> > > > > > +                               err = -ENOENT;
> > > > > > +                               goto out_v6;
> > > > > > +                       }
> > > > > > +               }
> > > > > > +
> > > > > > +               err = v6_hook->enable(link->net);
> > > > >
> > > > > I was about to apply, but luckily caught this issue in my local test:
> > > > >
> > > > > [   18.462448] BUG: sleeping function called from invalid context at
> > > > > kernel/locking/mutex.c:283
> > > > > [   18.463238] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid:
> > > > > 2042, name: test_progs
> > > > > [   18.463927] preempt_count: 0, expected: 0
> > > > > [   18.464249] RCU nest depth: 1, expected: 0
> > > > > [   18.464631] CPU: 15 PID: 2042 Comm: test_progs Tainted: G
> > > > > O       6.4.0-04319-g6f6ec4fa00dc #4896
> > > > > [   18.465480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > > > > BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
> > > > > [   18.466531] Call Trace:
> > > > > [   18.466767]  <TASK>
> > > > > [   18.466975]  dump_stack_lvl+0x32/0x40
> > > > > [   18.467325]  __might_resched+0x129/0x180
> > > > > [   18.467691]  mutex_lock+0x1a/0x40
> > > > > [   18.468057]  nf_defrag_ipv4_enable+0x16/0x70
> > > > > [   18.468467]  bpf_nf_link_attach+0x141/0x300
> > > > > [   18.468856]  __sys_bpf+0x133e/0x26d0
> > > > >
> > > > > You cannot call mutex under rcu_read_lock.
> > > >
> > > > Whoops, my bad. I think this patch should fix it:
> > > >
> > > > ```
> > > > From 7e8927c44452db07ddd7cf0e30bb49215fc044ed Mon Sep 17 00:00:00 2001
> > > > Message-ID: <7e8927c44452db07ddd7cf0e30bb49215fc044ed.1689211250.git.dxu@dxuuu.xyz>
> > > > From: Daniel Xu <dxu@dxuuu.xyz>
> > > > Date: Wed, 12 Jul 2023 19:17:35 -0600
> > > > Subject: [PATCH] netfilter: bpf: Don't hold rcu_read_lock during
> > > >  enable/disable
> > > >
> > > > ->enable()/->disable() takes a mutex which can sleep. You can't sleep
> > > > during RCU read side critical section.
> > > >
> > > > Our refcnt on the module will protect us from ->enable()/->disable()
> > > > from going away while we call it.
> > > >
> > > > Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> > > > ---
> > > >  net/netfilter/nf_bpf_link.c | 10 ++++++++--
> > > >  1 file changed, 8 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/net/netfilter/nf_bpf_link.c b/net/netfilter/nf_bpf_link.c
> > > > index 77ffbf26ba3d..79704cc596aa 100644
> > > > --- a/net/netfilter/nf_bpf_link.c
> > > > +++ b/net/netfilter/nf_bpf_link.c
> > > > @@ -60,9 +60,12 @@ static int bpf_nf_enable_defrag(struct bpf_nf_link *link)
> > > >                         goto out_v4;
> > > >                 }
> > > >
> > > > +               rcu_read_unlock();
> > > >                 err = v4_hook->enable(link->net);
> > > >                 if (err)
> > > >                         module_put(v4_hook->owner);
> > > > +
> > > > +               return err;
> > > >  out_v4:
> > > >                 rcu_read_unlock();
> > > >                 return err;
> > > > @@ -92,9 +95,12 @@ static int bpf_nf_enable_defrag(struct bpf_nf_link *link)
> > > >                         goto out_v6;
> > > >                 }
> > > >
> > > > +               rcu_read_unlock();
> > > >                 err = v6_hook->enable(link->net);
> > > >                 if (err)
> > > >                         module_put(v6_hook->owner);
> > > > +
> > > > +               return err;
> > > >  out_v6:
> > > >                 rcu_read_unlock();
> > > >                 return err;
> > > > @@ -114,11 +120,11 @@ static void bpf_nf_disable_defrag(struct bpf_nf_link *link)
> > > >         case NFPROTO_IPV4:
> > > >                 rcu_read_lock();
> > > >                 v4_hook = rcu_dereference(nf_defrag_v4_hook);
> > > > +               rcu_read_unlock();
> > > >                 if (v4_hook) {
> > > >                         v4_hook->disable(link->net);
> > > >                         module_put(v4_hook->owner);
> > > >                 }
> > > > -               rcu_read_unlock();
> > > >
> > > >                 break;
> > > >  #endif
> > > > @@ -126,11 +132,11 @@ static void bpf_nf_disable_defrag(struct bpf_nf_link *link)
> > > >         case NFPROTO_IPV6:
> > > >                 rcu_read_lock();
> > > >                 v6_hook = rcu_dereference(nf_defrag_v6_hook);
> > > > +               rcu_read_unlock();
> > >
> > > No. v6_hook is gone as soon as you unlock it.
> >
> > I think we're protected here by the try_module_get() on the enable path.
> > And we only disable defrag if enabling succeeds. The module shouldn't
> > be able to deregister its hooks until we call the module_put() later.
> >
> > I think READ_ONCE() would've been more appropriate but I wasn't sure if
> > that was ok given nf_defrag_v(4|6)_hook is written to by
> > rcu_assign_pointer() and I was assuming symmetry is necessary.
> 
> Why is rcu_assign_pointer() used?
> If it's not RCU protected, what is the point of rcu_*() accessors
> and rcu_read_lock() ?
> 
> In general, the pattern:
> rcu_read_lock();
> ptr = rcu_dereference(...);
> rcu_read_unlock();
> ptr->..
> is a bug. 100%.
> 

The reason I left it like this is b/c otherwise I think there is a race
with module unload and taking a refcnt. For example:

ptr = READ_ONCE(global_var)
                                             <module unload on other cpu>
// ptr invalid
try_module_get(ptr->owner) 

I think the the synchronize_rcu() call in
kernel/module/main.c:free_module() protects against that race based on
my reading.

Maybe the ->enable() path can store a copy of the hook ptr in
struct bpf_nf_link to get rid of the odd rcu_dereference()?

Open to other ideas too -- would appreciate any hints.

Thanks,
Daniel