Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 2 Aug 2018 14:25:37 -0700
From:   Roman Gushchin <guro@fb.com>
To:     Daniel Borkmann <daniel@iogearbox.net>
CC:     syzbot <syzbot+25554ab865a12b51c66f@syzkaller.appspotmail.com>,
        <ast@kernel.org>, <linux-kernel@vger.kernel.org>,
        <netdev@vger.kernel.org>, <syzkaller-bugs@googlegroups.com>
Subject: Re: KASAN: use-after-free Read in bpf_cgroup_storage_release
Message-ID: <20180802212534.GA17831@castle.DHCP.thefacebook.com>
References: <000000000000f3b1570572779079@google.com>
 <1b3dde38-549e-43ed-d3f6-18b451444d2f@iogearbox.net>
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <1b3dde38-549e-43ed-d3f6-18b451444d2f@iogearbox.net>
User-Agent: Mutt/1.9.2 (2017-12-15)
Received-SPF: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2018 21:25:43.1041 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 7e9e96c6-91c7-407c-103b-08d5f8be814b
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR15MB0162
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-08-02_05:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Aug 02, 2018 at 09:23:00PM +0200, Daniel Borkmann wrote:
> [ +Roman ]
> 
> On 08/02/2018 07:59 PM, syzbot wrote:
> > Hello,
> > 
> > syzbot found the following crash on:
> > 
> > HEAD commit:??? fc2a3b5dd618 Merge branch 'bpf-cgroup-local-storage'
> > git tree:?????? bpf-next
> > console output: https://urldefense.proofpoint.com/v2/url?u=https-3A__syzkaller.appspot.com_x_log.txt-3Fx-3D17a6a1c8400000&d=DwIDaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=EDSyTXUjQWi0Wr-pal_Z622tnk3UnxrZ318k9Tct4wQ&s=kNpA8CkCvXgMPDNNfqgkqDlGShzYMrSvH1AWBdBslJo&e=
> > kernel config:? https://urldefense.proofpoint.com/v2/url?u=https-3A__syzkaller.appspot.com_x_.config-3Fx-3D3bfcc1651962483&d=DwIDaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=EDSyTXUjQWi0Wr-pal_Z622tnk3UnxrZ318k9Tct4wQ&s=PYjD3Jqpwgzv3cBV64k8EAbXwdolWkrrrcanCfJU6KQ&e=
> > dashboard link: https://urldefense.proofpoint.com/v2/url?u=https-3A__syzkaller.appspot.com_bug-3Fextid-3D25554ab865a12b51c66f&d=DwIDaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=EDSyTXUjQWi0Wr-pal_Z622tnk3UnxrZ318k9Tct4wQ&s=FQHY4B4Ok0SJh_C-lkToT7mX_Ehj3NSHe6Fe1UiL--Y&e=
> > compiler:?????? gcc (GCC) 8.0.1 20180413 (experimental)
> > syzkaller repro:https://urldefense.proofpoint.com/v2/url?u=https-3A__syzkaller.appspot.com_x_repro.syz-3Fx-3D12c4b9b4400000&d=DwIDaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=EDSyTXUjQWi0Wr-pal_Z622tnk3UnxrZ318k9Tct4wQ&s=38j2eoBTEgFiU2FlYHK86YXiVNEjxV6n3ug4tlAZ3MQ&e=
> > C reproducer:?? https://urldefense.proofpoint.com/v2/url?u=https-3A__syzkaller.appspot.com_x_repro.c-3Fx-3D13e9d6f0400000&d=DwIDaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=EDSyTXUjQWi0Wr-pal_Z622tnk3UnxrZ318k9Tct4wQ&s=R75neIjoG9ODJgTDAUAfWqORwwOVX0k_cz7NsKyF6qw&e=
> > 
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+25554ab865a12b51c66f@syzkaller.appspotmail.com
> > 
> > ==================================================================
> > BUG: KASAN: use-after-free in debug_spin_lock_before kernel/locking/spinlock_debug.c:83 [inline]
> > BUG: KASAN: use-after-free in do_raw_spin_lock+0x1c0/0x200 kernel/locking/spinlock_debug.c:112
> > Read of size 4 at addr ffff8801c4723644 by task syz-executor865/9746
> > 
> > CPU: 0 PID: 9746 Comm: syz-executor865 Not tainted 4.18.0-rc5+ #68
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > Call Trace:
> > ?__dump_stack lib/dump_stack.c:77 [inline]
> > ?dump_stack+0x1c9/0x2b4 lib/dump_stack.c:113
> > ?print_address_description+0x6c/0x20b mm/kasan/report.c:256
> > ?kasan_report_error mm/kasan/report.c:354 [inline]
> > ?kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412
> > ?__asan_report_load4_noabort+0x14/0x20 mm/kasan/report.c:432
> > ?debug_spin_lock_before kernel/locking/spinlock_debug.c:83 [inline]
> > ?do_raw_spin_lock+0x1c0/0x200 kernel/locking/spinlock_debug.c:112
> > ?__raw_spin_lock_bh include/linux/spinlock_api_smp.h:136 [inline]
> > ?_raw_spin_lock_bh+0x39/0x40 kernel/locking/spinlock.c:168
> > ?spin_lock_bh include/linux/spinlock.h:315 [inline]
> > ?bpf_cgroup_storage_release+0x2c/0x110 kernel/bpf/local_storage.c:276
> > ?free_used_maps+0x81/0x200 kernel/bpf/syscall.c:961
> > ?bpf_prog_load+0x17ba/0x1c90 kernel/bpf/syscall.c:1414
> > ?__do_sys_bpf kernel/bpf/syscall.c:2338 [inline]
> > ?__se_sys_bpf kernel/bpf/syscall.c:2300 [inline]
> > ?__x64_sys_bpf+0x36c/0x510 kernel/bpf/syscall.c:2300
> > ?do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
> > ?entry_SYSCALL_64_after_hwframe+0x49/0xbe

So, it looks like we drop the last refcount on local storage map
from release_maps() and schedule bpf_map_free_deferred(), which
runs prior to bpf_cgroup_storage_release() in free_used_maps().

If so, here is the fix:

diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c
index cd0b115c2395..fc4e37f68f2a 100644
--- a/kernel/bpf/local_storage.c
+++ b/kernel/bpf/local_storage.c
@@ -277,6 +277,7 @@ void bpf_cgroup_storage_release(struct bpf_prog *prog, struct bpf_map *_map)
        if (map->prog == prog) {
                WARN_ON(prog->aux->cgroup_storage != _map);
                map->prog = NULL;
+               prog->aux->cgroup_storage = NULL;
        }
        spin_unlock_bh(&map->lock);
 }

--

I'll post an updated version (v7) in few minutes.

Thanks to syzbot team for the report!