Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751466Ab0DUGOb (ORCPT ); Wed, 21 Apr 2010 02:14:31 -0400 Received: from smtp-out.google.com ([216.239.44.51]:21651 "EHLO smtp-out.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751173Ab0DUGOa convert rfc822-to-8bit (ORCPT ); Wed, 21 Apr 2010 02:14:30 -0400 DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=mime-version:in-reply-to:references:from:date:message-id: subject:to:cc:content-type:content-transfer-encoding:x-system-of-record; b=QJkUqZIEfsrI32RtnmjSUUe+yMDqShHKZSLHcGQ3sGcNT1dsk3V3MKvpN6UDeac8D ZsEPmGjlnqW5/MMAVHKBg== MIME-Version: 1.0 In-Reply-To: <20100420211749.GF3020@redhat.com> References: <20100420211749.GF3020@redhat.com> From: Nauman Rafique Date: Tue, 20 Apr 2010 23:14:04 -0700 Message-ID: Subject: Re: blkio controller BUG_ON() To: Vivek Goyal Cc: Divyesh Shah , linux kernel mailing list , Jens Axboe , Gui Jianfeng Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-System-Of-Record: true Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5703 Lines: 97 Hi Vivek, Divyesh is out on vacation for the week. We are running our kernel with group isolation turned on, so that would explain why we would not see this in our testing. I feel like moving queues around with pending requests would be quite problematic in general. Can we allocate new queue in the root group and let the old queue drain in the older cgroup? In fact, are there many use cases of having cgroups but having group isolation off? Thanks. -- Nauman On Tue, Apr 20, 2010 at 2:17 PM, Vivek Goyal wrote: > Hi Divyesh, > > While testing blkio controller, I ran into following BUG_ON(). I am doing > my testing on for-2.6.35 branch of blk tree. This can be easily reproduced > by putting two sequential readers in two cgroups. > > I think what is happening is that you are assuming that rq->cfqq->cfqg > will remain same at the time of request dispatch and completion. But > because of cfqq migration logic (move cfqq to root group if it is not > sequential workload), it is not necessary that request dispatched from > one group complete in same group. > > I guess one solution would be to stash a cfq_group pointer in rq > (rq->elevator_private3)? > > Thanks > Vivek > > > [ ? 65.163523] ------------[ cut here ]------------ > [ ? 65.164301] kernel BUG at block/blk-cgroup.c:117! > [ ? 65.164301] invalid opcode: 0000 [#1] SMP > [ ? 65.164301] last sysfs file: /sys/devices/pci0000:00/0000:00:05.0/0000:60:00.1/host9/rport-9:0-0/target9:0:0/9:0:0:2/block/sde/stat > [ ? 65.164301] CPU 1 > [ ? 65.164301] Modules linked in: dm_round_robin dm_multipath qla2xxx scsi_transport_fc dm_zero dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > [ ? 65.164301] > [ ? 65.164301] Pid: 4505, comm: fio Not tainted 2.6.34-rc4-blk-for-35 #34 0A98h/HP xw8600 Workstation > [ ? 65.164301] RIP: 0010:[] ?[] blkiocg_update_io_remove_stats+0x5b/0xaf > [ ? 65.164301] RSP: 0018:ffff8800ba5a79e8 ?EFLAGS: 00010046 > [ ? 65.164301] RAX: 0000000000000096 RBX: ffff8800bb268d60 RCX: 0000000000000000 > [ ? 65.164301] RDX: ffff8800bb268eb8 RSI: 0000000000000000 RDI: ffff8800bb268e00 > [ ? 65.164301] RBP: ffff8800ba5a7a08 R08: 0000000000000064 R09: 0000000000000001 > [ ? 65.164301] R10: 0000000000079640 R11: ffff8800a0bd5bf0 R12: ffff8800bab4af01 > [ ? 65.164301] R13: ffff8800bab4af00 R14: ffff8800bb1d8928 R15: 0000000000000000 > [ ? 65.164301] FS: ?00007f18f75056f0(0000) GS:ffff880001e40000(0000) knlGS:0000000000000000 > [ ? 65.164301] CS: ?0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ ? 65.164301] CR2: 000000000040e7f0 CR3: 00000000ba52b000 CR4: 00000000000006e0 > [ ? 65.164301] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ ? 65.164301] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ ? 65.164301] Process fio (pid: 4505, threadinfo ffff8800ba5a6000, task ffff8800ba45ae80) > [ ? 65.164301] Stack: > [ ? 65.164301] ?ffff8800ba5a7a08 ffff8800ba722540 ffff8800bab4af68 ffff8800bab4af68 > [ ? 65.164301] <0> ffff8800ba5a7a38 ffffffff8121d814 ffff8800ba722540 ffff8800bab4af68 > [ ? 65.164301] <0> ffff8800ba722540 ffff8800a08f6800 ffff8800ba5a7a68 ffffffff8121d8ca > [ ? 65.164301] Call Trace: > [ ? 65.164301] ?[] cfq_remove_request+0xe4/0x116 > [ ? 65.164301] ?[] cfq_dispatch_insert+0x84/0xe1 > [ ? 65.164301] ?[] cfq_dispatch_requests+0x767/0x8e8 > [ ? 65.164301] ?[] ? submit_bio+0xc3/0xcc > [ ? 65.164301] ?[] ? sync_page_killable+0x0/0x35 > [ ? 65.164301] ?[] blk_peek_request+0x191/0x1a7 > [ ? 65.164301] ?[] ? dm_get_live_table+0x44/0x4f [dm_mod] > [ ? 65.164301] ?[] dm_request_fn+0x38/0x14c [dm_mod] > [ ? 65.164301] ?[] ? sync_page_killable+0x0/0x35 > [ ? 65.164301] ?[] __generic_unplug_device+0x32/0x37 > [ ? 65.164301] ?[] generic_unplug_device+0x2e/0x3c > [ ? 65.164301] ?[] dm_unplug_all+0x42/0x5b [dm_mod] > [ ? 65.164301] ?[] blk_unplug+0x29/0x2d > [ ? 65.164301] ?[] blk_backing_dev_unplug+0x12/0x14 > [ ? 65.164301] ?[] block_sync_page+0x35/0x39 > [ ? 65.164301] ?[] sync_page+0x41/0x4a > [ ? 65.164301] ?[] sync_page_killable+0xe/0x35 > [ ? 65.164301] ?[] __wait_on_bit_lock+0x46/0x8f > [ ? 65.164301] ?[] __lock_page_killable+0x66/0x6d > [ ? 65.164301] ?[] ? wake_bit_function+0x0/0x33 > [ ? 65.164301] ?[] lock_page_killable+0x2c/0x2e > [ ? 65.164301] ?[] generic_file_aio_read+0x361/0x4f0 > [ ? 65.164301] ?[] do_sync_read+0xcb/0x108 > [ ? 65.164301] ?[] ? security_file_permission+0x16/0x18 > [ ? 65.164301] ?[] vfs_read+0xab/0x108 > [ ? 65.164301] ?[] sys_read+0x4a/0x6e > [ ? 65.164301] ?[] system_call_fastpath+0x16/0x1b > [ ? 65.164301] Code: 00 74 1c 48 8b 8b 60 01 00 00 48 85 c9 75 04 0f 0b eb fe 48 ff c9 48 89 8b 60 01 00 00 eb 1a 48 8b 8b 58 01 00 00 48 85 c9 75 04 <0f> 0b eb fe 48 ff c9 48 89 8b 58 01 00 00 45 84 e4 74 16 48 8b > [ ? 65.164301] RIP ?[] blkiocg_update_io_remove_stats+0x5b/0xaf > [ ? 65.164301] ?RSP > [ ? 65.164301] ---[ end trace 1b2b828753032e68 ]--- > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/