Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 13 Jun 2019 21:21:25 -0700
From:   John Fastabend <john.fastabend@gmail.com>
To:     Hillf Danton <hdanton@sina.com>, Dmitry Vyukov <dvyukov@google.com>
Cc:     syzbot <syzbot+06537213db7ba2745c4a@syzkaller.appspotmail.com>,
        "aviadye@mellanox.com" <aviadye@mellanox.com>,
        "borisp@mellanox.com" <borisp@mellanox.com>,
        Daniel Borkmann <daniel@iogearbox.net>,
        "davejwatson@fb.com" <davejwatson@fb.com>,
        David Miller <davem@davemloft.net>,
        John Fastabend <john.fastabend@gmail.com>,
        LKML <linux-kernel@vger.kernel.org>,
        netdev <netdev@vger.kernel.org>,
        syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
        Jakub Kicinski <jakub.kicinski@netronome.com>,
        bpf <bpf@vger.kernel.org>, Eric Biggers <ebiggers@kernel.org>
Message-ID: <5d0320c540981_5ab92b1f436fc5c45b@john-XPS-13-9370.notmuch>
In-Reply-To: <20190612045828.10212-1-hdanton@sina.com>
References: <20190612045828.10212-1-hdanton@sina.com>
Subject: Re: memory leak in create_ctx
Mime-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hillf Danton wrote:
> 
> Hi Dmitry
> 
> On Tue, 11 Jun 2019 19:45:28 +0800 Dmitry Vyukov wrote:
> >
> > I've run the repro as "./syz-execprog -repeat=0 -procs=6 repro"  and
> > in 10 mins I got the following splat, which indeed suggests a bpf bug.
> > But we of course can have both bpf stack overflow and a memory leak in tls.
> >
> >
> >
> > 2019/06/11 10:26:52 executed programs: 887
> > 2019/06/11 10:26:57 executed programs: 899
> > 2019/06/11 10:27:02 executed programs: 916
> > [  429.171049][ T9870] BUG: stack guard page was hit at 00000000a78467b9 (stack is 000000001452e9df..000000004fb93e51)
> > [  429.173714][ T9870] kernel stack overflow (double-fault): 0000 [#1] PREEMPT SMP
> > [  429.174819][ T9870] CPU: 3 PID: 9870 Comm: syz-executor Not tainted 5.2.0-rc4+ #6
> > [  429.175901][ T9870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
> > [  429.177215][ T9870] RIP: 0010:tcp_bpf_unhash+0xc/0x80
> > [  429.177950][ T9870] Code: 28 4c 89 ee 48 89 df ff 10 e8 30 56 66 fe
> > 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 80 00 00 00 00 55 48 89 e5 41 55 41
> > 54 53 48 89 fb <e8> 0f 56 66 fe e8 6a bb 5f fe 4c 8b a3 80 02 00 00 4d
> > 85 e4 74 2f
> > [  429.180707][ T9870] RSP: 0018:ffffc90003690000 EFLAGS: 00010293
> > [  429.181562][ T9870] RAX: ffff888066a72000 RBX: ffff88806695b640 RCX: ffffffff82c82f80
> > [  429.182681][ T9870] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff88806695b640
> > [  429.183807][ T9870] RBP: ffffc90003690018 R08: 0000000000000000 R09: 0000000000000000
> > [  429.184931][ T9870] R10: ffffc90003693e70 R11: 0000000000000000 R12: ffffffff82c82f10
> > [  429.186104][ T9870] R13: 0000000000000007 R14: ffff88806695b710 R15: ffff88806695b710
> > [  429.187303][ T9870] FS:  00005555569fc940(0000) GS:ffff88807db80000(0000) knlGS:0000000000000000
> > [  429.188678][ T9870] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  429.189674][ T9870] CR2: ffffc9000368fff8 CR3: 00000000762bc002 CR4: 00000000007606e0
> > [  429.190880][ T9870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [  429.192094][ T9870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [  429.193295][ T9870] PKRU: 55555554
> > [  429.193829][ T9870] Call Trace:
> > [  429.194326][ T9870]  ? tcp_bpf_close+0xa0/0xa0
> > [  429.195020][ T9870]  tcp_bpf_unhash+0x76/0x80
> > [  429.195706][ T9870]  ? tcp_bpf_close+0xa0/0xa0
> > [  429.196400][ T9870]  tcp_bpf_unhash+0x76/0x80
> > [  429.197079][ T9870]  ? tcp_bpf_close+0xa0/0xa0
> > [  429.197773][ T9870]  tcp_bpf_unhash+0x76/0x80
> > [  429.651942][ T9870]  ? tcp_bpf_close+0xa0/0xa0
> >
> ... duplicated info trimed ...
> >
> > [  429.652512][ T9870]  tcp_bpf_unhash+0x76/0x80
> > [  429.656467][ T9870]  ? tcp_bpf_close+0xa0/0xa0
> > [  429.657037][ T9870]  tcp_bpf_unhash+0x76/0x80
> > [  429.657600][ T9870]  tcp_set_state+0x7b/0x220
> > [  429.658160][ T9870]  ? put_object+0x20/0x30
> > [  429.658699][ T9870]  ? debug_smp_processor_id+0x2b/0x130
> > [  429.659382][ T9870]  tcp_disconnect+0x518/0x610
> > [  429.659973][ T9870]  tcp_close+0x41d/0x540
> > [  429.660501][ T9870]  ? tcp_check_oom+0x180/0x180
> > [  429.661095][ T9870]  tls_sk_proto_close+0x86/0x2a0
> > [  429.661711][ T9870]  ? locks_remove_posix+0x114/0x1c0
> > [  429.662359][ T9870]  inet_release+0x44/0x80
> > [  429.662899][ T9870]  inet6_release+0x36/0x50
> > [  429.663453][ T9870]  __sock_release+0x4b/0x100
> > [  429.664024][ T9870]  ? __sock_release+0x100/0x100
> > [  429.664625][ T9870]  sock_close+0x19/0x20
> > [  429.665141][ T9870]  __fput+0xe7/0x2f0
> > [  429.665624][ T9870]  ____fput+0x15/0x20
> > [  429.666120][ T9870]  task_work_run+0xa4/0xd0
> > [  429.666671][ T9870]  exit_to_usermode_loop+0x16f/0x180
> > [  429.667329][ T9870]  do_syscall_64+0x187/0x1b0
> > [  429.667920][ T9870]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [  429.668654][ T9870] RIP: 0033:0x412451
> > [  429.669141][ T9870] Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff
> > ff 0f 83 94 1a 00 00 c3 48 83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03
> > 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4
> > 08 48 3d 01
> > [  429.671586][ T9870] RSP: 002b:00007ffde18b5470 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
> > [  429.672636][ T9870] RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000412451
> > [  429.673628][ T9870] RDX: 0000000000000000 RSI: 0000000000000081 RDI: 0000000000000004
> > [  429.674643][ T9870] RBP: 0000000000000000 R08: 0000000000721170 R09: 0000000000000000
> > [  429.675641][ T9870] R10: 00007ffde18b5580 R11: 0000000000000293 R12: 0000000000000000
> > [  429.676636][ T9870] R13: 000000000071bf00 R14: 00000000006e3140 R15: ffffffffffffffff
> > [  429.677630][ T9870] Modules linked in:
> > [  429.678119][ T9870] ---[ end trace a429c7ce256ca7bb ]---
> > [  429.678798][ T9870] RIP: 0010:tcp_bpf_unhash+0xc/0x80
> > [  429.679447][ T9870] Code: 28 4c 89 ee 48 89 df ff 10 e8 30 56 66 fe
> > 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 80 00 00 00 00 55 48 89 e5 41 55 41
> > 54 53 48 89 fb <e8> 0f 56 66 fe e8 6a bb 5f fe 4c 8b a3 80 02 00 00 4d
> > 85 e4 74 2f
> > [  429.681882][ T9870] RSP: 0018:ffffc90003690000 EFLAGS: 00010293
> > [  429.682637][ T9870] RAX: ffff888066a72000 RBX: ffff88806695b640 RCX: ffffffff82c82f80
> > [  429.683630][ T9870] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff88806695b640
> > [  429.684622][ T9870] RBP: ffffc90003690018 R08: 0000000000000000 R09: 0000000000000000
> > [  429.685611][ T9870] R10: ffffc90003693e70 R11: 0000000000000000 R12: ffffffff82c82f10
> > [  429.686601][ T9870] R13: 0000000000000007 R14: ffff88806695b710 R15: ffff88806695b710
> > [  429.687592][ T9870] FS:  00005555569fc940(0000) GS:ffff88807db80000(0000) knlGS:0000000000000000
> > [  429.688701][ T9870] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  429.689519][ T9870] CR2: ffffc9000368fff8 CR3: 00000000762bc002 CR4: 00000000007606e0
> > [  429.690511][ T9870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [  429.691507][ T9870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [  429.692502][ T9870] PKRU: 55555554
> > [  429.692941][ T9870] Kernel panic - not syncing: Fatal exception
> > [  429.694377][ T9870] Kernel Offset: disabled
> > [  429.694913][ T9870] Rebooting in 86400 seconds..
> >
> >
> In case of short circuit, bail out after warning on the unusual event to advoid
> spiraling down the stack without brakes.

Thanks for tracking this down, I was able to reproduce with sys-execprog
so will try and come up with some fix.

We really shouldn't ever get here, somehow we dropped the psock but left
the sk_prot hooked up. At this point just returning is going to leave
lots of psock/tls/sock state floating around.

I guess we need to find the case where sk_prot is not reset.

.John

> 
> Thanks
> Hillf
> ---
>  net/ipv4/tcp_bpf.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c
> index 3d1e154..5022cd3 100644
> --- a/net/ipv4/tcp_bpf.c
> +++ b/net/ipv4/tcp_bpf.c
> @@ -546,6 +546,8 @@ static void tcp_bpf_unhash(struct sock *sk)
>  	psock = sk_psock(sk);
>  	if (unlikely(!psock)) {
>  		rcu_read_unlock();
> +		if (WARN_ON(sk->sk_prot->unhash == tcp_bpf_unhash))
> +			return;
>  		if (sk->sk_prot->unhash)
>  			sk->sk_prot->unhash(sk);
>  		return;
> @@ -568,6 +570,8 @@ static void tcp_bpf_close(struct sock *sk, long timeout)
>  	if (unlikely(!psock)) {
>  		rcu_read_unlock();
>  		release_sock(sk);
> +		if (WARN_ON(sk->sk_prot->close == tcp_bpf_close))
> +			return;
>  		return sk->sk_prot->close(sk, timeout);
>  	}
> 
> --
>