Received-SPF: pass (google.com: domain of linux-nfs+bounces-2295-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <20240314210740.GA2823176@perftesting> <CANn89i+Bid4YkwFEmxSvF22Gk0jY+hH7P=mjEKR=LBPc+vG_PA@mail.gmail.com>
In-Reply-To: <CANn89i+Bid4YkwFEmxSvF22Gk0jY+hH7P=mjEKR=LBPc+vG_PA@mail.gmail.com>
From: Eric Dumazet <edumazet@google.com>
Date: Thu, 14 Mar 2024 23:10:45 +0100
Message-ID: <CANn89iJKfxnzWzoN=KkFERjRUYpyKHO4KLJytMBS69tnRWY9Jw@mail.gmail.com>
Subject: Re: [BUG] Panic in ipv6 on old NFS sockets from destroyed network namespace
To: Josef Bacik <josef@toxicpanda.com>
Cc: kuba@kernel.org, netdev@vger.kernel.org, linux-nfs@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Mar 14, 2024 at 10:47=E2=80=AFPM Eric Dumazet <edumazet@google.com>=
 wrote:
>
> On Thu, Mar 14, 2024 at 10:07=E2=80=AFPM Josef Bacik <josef@toxicpanda.co=
m> wrote:
> >
> > Hello,
> >
> > We've been hitting the following panic in production, and I've root cau=
sed
> > what's happening, but I'm at a loss on how to fix it.
> >
> > The panic we're seeing is this
> >
> >     BUG: kernel NULL pointer dereference, address: 0000000000000000
> >     RIP: 0010:ip6_pol_route+0x59/0x7a0
> >     Call Trace:
> >      <IRQ>
> >      ? __die+0x78/0xc0
> >      ? page_fault_oops+0x286/0x380
> >      ? fib6_table_lookup+0x95/0xf40
> >      ? exc_page_fault+0x5d/0x110
> >      ? asm_exc_page_fault+0x22/0x30
> >      ? ip6_pol_route+0x59/0x7a0
> >      ? unlink_anon_vmas+0x370/0x370
> >      fib6_rule_lookup+0x56/0x1b0
> >      ? update_blocked_averages+0x2c6/0x6a0
> >      ip6_route_output_flags+0xd2/0x130
> >      ip6_dst_lookup_tail+0x3b/0x220
> >      ip6_dst_lookup_flow+0x2c/0x80
> >      inet6_sk_rebuild_header+0x14c/0x1e0
> >      ? tcp_release_cb+0x150/0x150
> >      __tcp_retransmit_skb+0x68/0x6b0
> >      ? tcp_current_mss+0xca/0x150
> >      ? tcp_release_cb+0x150/0x150
> >      tcp_send_loss_probe+0x8e/0x220
> >      tcp_write_timer+0xbe/0x2d0
> >      run_timer_softirq+0x272/0x840
> >      ? hrtimer_interrupt+0x2c9/0x5f0
> >      ? sched_clock_cpu+0xc/0x170
> >      irq_exit_rcu+0x171/0x330
> >      sysvec_apic_timer_interrupt+0x6d/0x80
> >      </IRQ>
> >      <TASK>
> >      asm_sysvec_apic_timer_interrupt+0x16/0x20
> >     RIP: 0010:cpuidle_enter_state+0xe7/0x243
> >
> > Inspecting the vmcore with drgn you can see why this is a NULL pointer =
deref
> >
> >       >>> prog.crashed_thread().stack_trace()[0]
> >       #0 at 0xffffffff810bfa89 (ip6_pol_route+0x59/0x796) in ip6_pol_ro=
ute at net/ipv6/route.c:2212:40
> >
> >       2212        if (net->ipv6.devconf_all->forwarding =3D=3D 0)
> >       2213              strict |=3D RT6_LOOKUP_F_REACHABLE;
> >
> >       >>> prog.crashed_thread().stack_trace()[0]['net'].ipv6.devconf_al=
l
> >       (struct ipv6_devconf *)0x0
> >
> > Looking at the socket you can see that it's been closed
> >
> >       >>> decode_enum_type_flags(prog.crashed_thread().stack_trace()[11=
]['sk'].__sk_common.skc_flags, prog.type('enum sock_flags'))
> >       'SOCK_DEAD|SOCK_KEEPOPEN|SOCK_ZAPPED|SOCK_USE_WRITE_QUEUE'
> >       >>> decode_enum_type_flags(1 << prog.crashed_thread().stack_trace=
()[11]['sk'].__sk_common.skc_state.value_(), prog["TCPF_CLOSE"].type_, bit_=
numbers=3DFalse)
> >       'TCPF_FIN_WAIT1'
> >
> > The way this reproduces is with our NFS setup.  We have an NFS mount in=
side of a
> > container, which has it's own network namespace.  We setup the mount in=
side of
> > this network namespace.
> >
> > On container shutdown sometimes we trigger this panic, it's pretty reli=
ably
> > reproduced, with a stress tier of 200 machines I can usually trigger it=
 on ~10
> > machines by stopping the jobs.
> >
> > My initial thought was that NFS wasn't properly shutting down the socke=
ts, but
> > this doesn't appear to be the case.  The sock is always marked with SOC=
K_DEAD.
> > My second thought was that we had some pending timers when we call
> > kernel_sock_shutdown(), so I added tcp_clear_xmit_timers(sk); to tcp_sh=
utdown()
> > to make sure the timers were cleared.  This didn't fix the issue.
> >
> > I added some debugging to the socket and flagged the socket when NFS ca=
lled
> > kernel_sock_shutdown() and then had a WARN_ON(sock_flag(sk,
> > JOSEFS_SPECIAL_FLAG)) where we arm the timer, and that trips constantly=
.  So
> > we're definitely arming the sock after NFS has shutdown the socket.
> >
> > This is where we leave my ability to figure out what's going on and how=
 to fix
> > it.  What seems to be happening is this
> >
> > 1. NFS calls kernel_sock_shutdown() when we unmount.
> > 2. We get an ACK on the socket and the timer gets armed.
> > 3. We shutdown the container and tear down the network namespace.
> > 4. The timer fires and we try to send the loss probe and we panic becau=
se the
> >    network namespace teardown removes the devconf as part of its teardo=
wn.
> >
> > It appears to me that sock's will just hang around forever past the end=
 of an
> > application being done with it, tho I'm not sure if I'm correct in this=
.  If
> > that's the case then I don't know the correct way to handle this, other=
 than
> > adding an extra case for the timer to simply not run when SOCK_DEAD is =
set.  But
> > this seems to be done on purpose, so seems like that's a bad fix.
> >
> > Let me know if you have debug patches or other information you'd like f=
rom a
> > vmcore, I have plenty.  Like I said I can reproduce reliably, it does t=
ake a few
> > hours to deploy a test kernel, but I can have a turn around of about a =
day for
> > debug patches.  Thanks,
> >
> > Josef
>
>   If NFS is using kernel sockets, it is NFS responsibility to remove
> all of them when the netns is destroyed.
>
> Also look at recent relevant  patches
>
> 2a750d6a5b365265dbda33330a6188547ddb5c24 rds: tcp: Fix use-after-free
> of net in reqsk_timer_handler().
> 1c4e97dd2d3c9a3e84f7e26346aa39bc426d3249 tcp: Fix NEW_SYN_RECV
> handling in inet_twsk_purge()

Another relevant patch was

commit 3a58f13a881ed351198ffab4cf9953cf19d2ab3a
Author: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date:   Mon May 2 10:40:18 2022 +0900

    net: rds: acquire refcount on TCP sockets