Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754664AbZINH5E (ORCPT ); Mon, 14 Sep 2009 03:57:04 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752211AbZINH5D (ORCPT ); Mon, 14 Sep 2009 03:57:03 -0400 Received: from mail-fx0-f217.google.com ([209.85.220.217]:49961 "EHLO mail-fx0-f217.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751043AbZINH5B convert rfc822-to-8bit (ORCPT ); Mon, 14 Sep 2009 03:57:01 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=WYU35aCoM7mNBfDCco6eSbsniUkhUleVc3az1es4Ln00++j9B8IoMBWxeN3/Ff4vYQ RBWRmBUf47RW1u/kGoBEsa1bBrWkMyD8jdwECH/SLfnhSjdP8X9ok5trNRdNG8Fw4SgU yd3AIZe2lHkH9vfFTfwJnLPTgmDqR5j1o26sM= MIME-Version: 1.0 In-Reply-To: <20090914071631.GA24801@elte.hu> References: <20090912072450.GA6767@elte.hu> <1252808939.13780.30.camel@dhcp231-106.rdu.redhat.com> <20090914071631.GA24801@elte.hu> Date: Mon, 14 Sep 2009 10:57:03 +0300 X-Google-Sender-Auth: cd95e56c6028b498 Message-ID: <84144f020909140057j542a3ee8wc996ffc6f8fcbbd1@mail.gmail.com> Subject: Re: [origin tree SLAB corruption] BUG kmalloc-64: Poison overwritten, INFO: Allocated in bdi_alloc_work+0x2b/0x100 age=175 cpu=1 pid=3514 From: Pekka Enberg To: Ingo Molnar Cc: Eric Paris , Jens Axboe , James Morris , Thomas Liu , linux-kernel@vger.kernel.org, Linus Torvalds Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3935 Lines: 86 * Eric Paris wrote: >> On Sat, 2009-09-12 at 09:24 +0200, Ingo Molnar wrote: >> > James - i did not see a security pull request email from you in my >> > lkml folder so i created this new thread. -tip testing found the >> > easy crash below. It reverts cleanly so i went that easy route. >> > >> > At a really quick 10-seconds glance the crash happens because we >> > destroy the slab cache twice, if the sysctl is toggled twice? >> >> Something a lot worse than SELinux here. ?I added this exact code and >> got this warning. ?Something is wrong in the world of >> kmem_cache_destroy..... Btw, the kmem_cache_destroy() bug Eric found is not in Linu's tree yet. On Mon, Sep 14, 2009 at 10:16 AM, Ingo Molnar wrote: > -tip testing just triggered another type of SLAB problem (this time > not apparently related to the security subsystem): > > BUG kmalloc-64: Poison overwritten > ----------------------------------------------------------------------------- > > INFO: 0xf498f6a0-0xf498f6a7. First byte 0x90 instead of 0x6b > INFO: Allocated in bdi_alloc_work+0x2b/0x100 age=175 cpu=1 pid=3514 > INFO: Freed in bdi_work_free+0x45/0x60 age=9 cpu=1 pid=3509 > INFO: Slab 0xc3257d84 objects=36 used=11 fp=0xf498f690 flags=0x400000c3 > INFO: Object 0xf498f690 @offset=1680 fp=0xf498fe00 > > Bytes b4 0xf498f680: ?ab 0d 00 00 9c 27 ff ff 5a 5a 5a 5a 5a 5a 5a 5a ?....'??ZZZZZZZZ > ?Object 0xf498f690: ?6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > ?Object 0xf498f6a0: ?90 f3 98 f4 60 3c 11 c1 6b 6b 6b 6b 6b 6b 6b 6b .?.?`<.?kkkkkkkk This would be use-after-free in kmalloc-64 cache. Given the trace and the fact that bdi_work_alloc() got introduce recently, it seems more likely that fs/fs-writeback.c is to blame here. Jens, does the warning ring a bell to you? > ?Object 0xf498f6b0: ?6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > ?Object 0xf498f6c0: ?6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk? > ?Redzone 0xf498f6d0: ?bb bb bb bb ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ???? > ?Padding 0xf498f6f8: ?5a 5a 5a 5a 5a 5a 5a 5a ? ? ? ? ? ? ? ? ? ? ? ? ZZZZZZZZ > Pid: 3514, comm: sync Not tainted 2.6.31-tip-02343-gb432421-dirty #14071 > Call Trace: > ?[] print_trailer+0xf9/0x170 > ?[] check_bytes_and_report+0xf5/0x120 > ?[] check_object+0x1e9/0x230 > ?[] alloc_debug_processing+0xfd/0x1d0 > ?[] ? bdi_alloc_work+0x2b/0x100 > ?[] __slab_alloc+0x127/0x330 > ?[] ? bdi_alloc_work+0x2b/0x100 > ?[] ? bdi_alloc_work+0x2b/0x100 > ?[] kmem_cache_alloc+0x1b3/0x1d0 > ?[] ? bdi_alloc_work+0x2b/0x100 > ?[] ? bdi_alloc_work+0x2b/0x100 > ?[] ? bdi_writeback_all+0x34/0x190 > ?[] bdi_alloc_work+0x2b/0x100 > ?[] ? _spin_lock+0x72/0x90 > ?[] bdi_writeback_all+0x72/0x190 > ?[] ? mark_held_locks+0x6b/0xb0 > ?[] ? __mutex_unlock_slowpath+0xf5/0x160 > ?[] ? trace_hardirqs_on_caller+0x15c/0x1c0 > ?[] sync_inodes_sb+0x48/0x70 > ?[] __sync_filesystem+0x7b/0x90 > ?[] sync_filesystems+0xf3/0x140 > ?[] sys_sync+0x27/0x60 > ?[] sysenter_do_call+0x12/0x36 > FIX kmalloc-64: Restoring 0xf498f6a0-0xf498f6a7=0x6b > > Now, this might be an extended arm of the security related slab > troubles. (which seem to have been cured by the revert i posted > though.) > > This bug is not bisectable at all - it happened after 1000+ > successful random bootups. The mainline base for > 2.6.31-tip-02343-gb432421 is upstream commit 86d7101. > > Full bootlog and config attached. > > ? ? ? ?Ingo > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/