Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754365Ab0GWLNc (ORCPT ); Fri, 23 Jul 2010 07:13:32 -0400 Received: from bld-mail19.adl2.internode.on.net ([150.101.137.104]:51843 "EHLO mail.internode.on.net" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752427Ab0GWLN3 (ORCPT ); Fri, 23 Jul 2010 07:13:29 -0400 Date: Fri, 23 Jul 2010 21:13:10 +1000 From: Dave Chinner To: Nick Piggin Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Frank Mayhar , John Stultz Subject: Re: VFS scalability git tree Message-ID: <20100723111310.GI32635@dastard> References: <20100722190100.GA22269@amd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100722190100.GA22269@amd> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 15960 Lines: 229 On Fri, Jul 23, 2010 at 05:01:00AM +1000, Nick Piggin wrote: > I'm pleased to announce I have a git tree up of my vfs scalability work. > > git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git > http://git.kernel.org/?p=linux/kernel/git/npiggin/linux-npiggin.git > > Branch vfs-scale-working I've got a couple of patches needed to build XFS - they shrinker merge left some bad fragments - I'll post them in a minute. This email is for the longest ever lockdep warning I've seen that occurred on boot. Cheers, Dave. [ 6.368707] ====================================================== [ 6.369773] [ INFO: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected ] [ 6.370379] 2.6.35-rc5-dgc+ #58 [ 6.370882] ------------------------------------------------------ [ 6.371475] pmcd/2124 [HC0[0]:SC0[1]:HE1:SE0] is trying to acquire: [ 6.372062] (&sb->s_type->i_lock_key#6){+.+...}, at: [] socket_get_id+0x3c/0x60 [ 6.372268] [ 6.372268] and this task is already holding: [ 6.372268] (&(&hashinfo->ehash_locks[i])->rlock){+.-...}, at: [] established_get_first+0x60/0x120 [ 6.372268] which would create a new lock dependency: [ 6.372268] (&(&hashinfo->ehash_locks[i])->rlock){+.-...} -> (&sb->s_type->i_lock_key#6){+.+...} [ 6.372268] [ 6.372268] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 6.372268] (&(&hashinfo->ehash_locks[i])->rlock){+.-...} [ 6.372268] ... which became SOFTIRQ-irq-safe at: [ 6.372268] [] __lock_acquire+0x576/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [] tcp_v4_syn_recv_sock+0x1aa/0x2d0 [ 6.372268] [] tcp_check_req+0x202/0x440 [ 6.372268] [] tcp_v4_do_rcv+0x304/0x4f0 [ 6.372268] [] tcp_v4_rcv+0x684/0x7e0 [ 6.372268] [] ip_local_deliver+0xe2/0x1c0 [ 6.372268] [] ip_rcv+0x397/0x760 [ 6.372268] [] __netif_receive_skb+0x277/0x330 [ 6.372268] [] process_backlog+0xd4/0x1e0 [ 6.372268] [] net_rx_action+0x188/0x2b0 [ 6.372268] [] __do_softirq+0xd2/0x260 [ 6.372268] [] call_softirq+0x1c/0x50 [ 6.372268] [] local_bh_enable_ip+0xeb/0xf0 [ 6.372268] [] _raw_spin_unlock_bh+0x34/0x40 [ 6.372268] [] release_sock+0x14e/0x1a0 [ 6.372268] [] inet_stream_connect+0x75/0x320 [ 6.372268] [] sys_connect+0xa7/0xc0 [ 6.372268] [] system_call_fastpath+0x16/0x1b [ 6.372268] [ 6.372268] to a SOFTIRQ-irq-unsafe lock: [ 6.372268] (&sb->s_type->i_lock_key#6){+.+...} [ 6.372268] ... which became SOFTIRQ-irq-unsafe at: [ 6.372268] ... [] __lock_acquire+0x5c3/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] new_inode+0x52/0xd0 [ 6.372268] [] get_sb_pseudo+0xb0/0x180 [ 6.372268] [] sockfs_get_sb+0x21/0x30 [ 6.372268] [] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [] kern_mount_data+0x19/0x20 [ 6.372268] [] sock_init+0x4e/0x59 [ 6.372268] [] do_one_initcall+0x3c/0x1a0 [ 6.372268] [] kernel_init+0x17a/0x204 [ 6.372268] [] kernel_thread_helper+0x4/0x10 [ 6.372268] [ 6.372268] other info that might help us debug this: [ 6.372268] [ 6.372268] 3 locks held by pmcd/2124: [ 6.372268] #0: (&p->lock){+.+.+.}, at: [] seq_read+0x3e/0x430 [ 6.372268] #1: (&(&hashinfo->ehash_locks[i])->rlock){+.-...}, at: [] established_get_first+0x60/0x120 [ 6.372268] #2: (clock-AF_INET){++....}, at: [] sock_i_ino+0x2e/0x70 [ 6.372268] [ 6.372268] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 6.372268] -> (&(&hashinfo->ehash_locks[i])->rlock){+.-...} ops: 3 { [ 6.372268] HARDIRQ-ON-W at: [ 6.372268] [] __lock_acquire+0x597/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [] __inet_hash_connect+0x33a/0x3d0 [ 6.372268] [] inet_hash_connect+0x4f/0x60 [ 6.372268] [] tcp_v4_connect+0x272/0x4f0 [ 6.372268] [] inet_stream_connect+0x28e/0x320 [ 6.372268] [] sys_connect+0xa7/0xc0 [ 6.372268] [] system_call_fastpath+0x16/0x1b [ 6.372268] IN-SOFTIRQ-W at: [ 6.372268] [] __lock_acquire+0x576/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [] tcp_v4_syn_recv_sock+0x1aa/0x2d0 [ 6.372268] [] tcp_check_req+0x202/0x440 [ 6.372268] [] tcp_v4_do_rcv+0x304/0x4f0 [ 6.372268] [] tcp_v4_rcv+0x684/0x7e0 [ 6.372268] [] ip_local_deliver+0xe2/0x1c0 [ 6.372268] [] ip_rcv+0x397/0x760 [ 6.372268] [] __netif_receive_skb+0x277/0x330 [ 6.372268] [] process_backlog+0xd4/0x1e0 [ 6.372268] [] net_rx_action+0x188/0x2b0 [ 6.372268] [] __do_softirq+0xd2/0x260 [ 6.372268] [] call_softirq+0x1c/0x50 [ 6.372268] [] local_bh_enable_ip+0xeb/0xf0 [ 6.372268] [] _raw_spin_unlock_bh+0x34/0x40 [ 6.372268] [] release_sock+0x14e/0x1a0 [ 6.372268] [] inet_stream_connect+0x75/0x320 [ 6.372268] [] sys_connect+0xa7/0xc0 [ 6.372268] [] system_call_fastpath+0x16/0x1b [ 6.372268] INITIAL USE at: [ 6.372268] [] __lock_acquire+0x232/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] __inet_hash_nolisten+0xfa/0x180 [ 6.372268] [] __inet_hash_connect+0x33a/0x3d0 [ 6.372268] [] inet_hash_connect+0x4f/0x60 [ 6.372268] [] tcp_v4_connect+0x272/0x4f0 [ 6.372268] [] inet_stream_connect+0x28e/0x320 [ 6.372268] [] sys_connect+0xa7/0xc0 [ 6.372268] [] system_call_fastpath+0x16/0x1b [ 6.372268] } [ 6.372268] ... key at: [] __key.47027+0x0/0x8 [ 6.372268] ... acquired at: [ 6.372268] [] check_irq_usage+0x60/0xf0 [ 6.372268] [] __lock_acquire+0xc4f/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] socket_get_id+0x3c/0x60 [ 6.372268] [] sock_i_ino+0x43/0x70 [ 6.372268] [] tcp4_seq_show+0x1a9/0x520 [ 6.372268] [] seq_read+0x295/0x430 [ 6.372268] [] proc_reg_read+0x84/0xc0 [ 6.372268] [] vfs_read+0xb5/0x170 [ 6.372268] [] sys_read+0x54/0x90 [ 6.372268] [] system_call_fastpath+0x16/0x1b [ 6.372268] [ 6.372268] [ 6.372268] the dependencies between the lock to be acquired and SOFTIRQ-irq-unsafe lock: [ 6.372268] -> (&sb->s_type->i_lock_key#6){+.+...} ops: 1185 { [ 6.372268] HARDIRQ-ON-W at: [ 6.372268] [] __lock_acquire+0x597/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] new_inode+0x52/0xd0 [ 6.372268] [] get_sb_pseudo+0xb0/0x180 [ 6.372268] [] sockfs_get_sb+0x21/0x30 [ 6.372268] [] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [] kern_mount_data+0x19/0x20 [ 6.372268] [] sock_init+0x4e/0x59 [ 6.372268] [] do_one_initcall+0x3c/0x1a0 [ 6.372268] [] kernel_init+0x17a/0x204 [ 6.372268] [] kernel_thread_helper+0x4/0x10 [ 6.372268] SOFTIRQ-ON-W at: [ 6.372268] [] __lock_acquire+0x5c3/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] new_inode+0x52/0xd0 [ 6.372268] [] get_sb_pseudo+0xb0/0x180 [ 6.372268] [] sockfs_get_sb+0x21/0x30 [ 6.372268] [] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [] kern_mount_data+0x19/0x20 [ 6.372268] [] sock_init+0x4e/0x59 [ 6.372268] [] do_one_initcall+0x3c/0x1a0 [ 6.372268] [] kernel_init+0x17a/0x204 [ 6.372268] [] kernel_thread_helper+0x4/0x10 [ 6.372268] INITIAL USE at: [ 6.372268] [] __lock_acquire+0x232/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] new_inode+0x52/0xd0 [ 6.372268] [] get_sb_pseudo+0xb0/0x180 [ 6.372268] [] vfs_kern_mount+0x8a/0x1e0 [ 6.372268] [] kern_mount_data+0x19/0x20 [ 6.372268] [] sock_init+0x4e/0x59 [ 6.372268] [] do_one_initcall+0x3c/0x1a0 [ 6.372268] [] kernel_init+0x17a/0x204 [ 6.372268] [] kernel_thread_helper+0x4/0x10 [ 6.372268] } [ 6.372268] ... key at: [] sock_fs_type+0x58/0x80 [ 6.372268] ... acquired at: [ 6.372268] [] check_irq_usage+0x60/0xf0 [ 6.372268] [] __lock_acquire+0xc4f/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] socket_get_id+0x3c/0x60 [ 6.372268] [] sock_i_ino+0x43/0x70 [ 6.372268] [] tcp4_seq_show+0x1a9/0x520 [ 6.372268] [] seq_read+0x295/0x430 [ 6.372268] [] proc_reg_read+0x84/0xc0 [ 6.372268] [] vfs_read+0xb5/0x170 [ 6.372268] [] sys_read+0x54/0x90 [ 6.372268] [] system_call_fastpath+0x16/0x1b [ 6.372268] [ 6.372268] [ 6.372268] stack backtrace: [ 6.372268] Pid: 2124, comm: pmcd Not tainted 2.6.35-rc5-dgc+ #58 [ 6.372268] Call Trace: [ 6.372268] [] check_usage+0x499/0x4a0 [ 6.372268] [] ? check_usage+0x86/0x4a0 [ 6.372268] [] ? __bfs+0x129/0x260 [ 6.372268] [] check_irq_usage+0x60/0xf0 [ 6.372268] [] __lock_acquire+0xc4f/0x1450 [ 6.372268] [] lock_acquire+0xa6/0x160 [ 6.372268] [] ? socket_get_id+0x3c/0x60 [ 6.372268] [] _raw_spin_lock+0x36/0x70 [ 6.372268] [] ? socket_get_id+0x3c/0x60 [ 6.372268] [] socket_get_id+0x3c/0x60 [ 6.372268] [] sock_i_ino+0x43/0x70 [ 6.372268] [] tcp4_seq_show+0x1a9/0x520 [ 6.372268] [] ? established_get_first+0x60/0x120 [ 6.372268] [] ? _raw_spin_lock_bh+0x67/0x70 [ 6.372268] [] seq_read+0x295/0x430 [ 6.372268] [] ? seq_read+0x0/0x430 [ 6.372268] [] proc_reg_read+0x84/0xc0 [ 6.372268] [] vfs_read+0xb5/0x170 [ 6.372268] [] sys_read+0x54/0x90 [ 6.372268] [] system_call_fastpath+0x16/0x1b -- Dave Chinner david@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/