From: Trond Myklebust Subject: Re: 2.6.31-rc2 soft lockups; traces point at rpc_wake_up, nfs4_run_state_manager, bit_waitqueue Date: Sun, 05 Jul 2009 09:31:01 -0400 Message-ID: <1246800661.5937.1.camel@heimdal.trondhjem.org> References: <87eisv7g5p.fsf@bulky.wgtn.ondioline.org> <20090705081225.GA12783@elte.hu> Mime-Version: 1.0 Content-Type: text/plain Cc: Paul Collins , linux-nfs@vger.kernel.org, Andy Adamson , Benny Halevy , linux-kernel@vger.kernel.org To: Ingo Molnar Return-path: Received: from mx2.netapp.com ([216.240.18.37]:36488 "EHLO mx2.netapp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750773AbZGENbB (ORCPT ); Sun, 5 Jul 2009 09:31:01 -0400 In-Reply-To: <20090705081225.GA12783@elte.hu> Sender: linux-nfs-owner@vger.kernel.org List-ID: On Sun, 2009-07-05 at 10:12 +0200, Ingo Molnar wrote: > (added more Cc:s) > > potential suspects are: > > 1f84603: Merge branch 'devel-for-2.6.31' into for-2.6.31 > 3f09df7: NFS: Ensure we always hold the BKL when dereferencing inode->i_flock > 965b5d6: NFSv4: Handle more errors when recovering open file and locking state > 34dc1ad: nfs41: increment_{open,lock}_seqid > 78722e9: nfs41: only retry EXCHANGE_ID on recoverable errors > b4b8260: nfs41: get_clid_cred for EXCHANGE_ID > 90a1661: nfs41: add a get_clid_cred function to nfs4_state_recovery_ops > 591d71c: nfs41: establish sessions-based clientid > a7b7210: nfs41: introduce get_state_renewal_cred > 8e69514f: nfs41: support minorversion 1 for nfs4_check_lease > c3fad1b: nfs41: add session reset to state manager > 76db6d95: nfs41: add session setup to the state manager > c2e713d: nfs41: translate NFS4ERR_MINOR_VERS_MISMATCH to EPROTONOSUPPORT Or possibly either rpc.gssd or rpc.idmapd dying. Have you checked to see if they are up and running correctly? Trond > In case the bug is in fs/nfs/nfs4proc.c you could perhaps do a > pretty quick ~5 reboots bisection using: > > git bisect start fs/nfs/nfs4proc.c > > Ingo > > * Paul Collins wrote: > > > I just tried 2.6.31-rc2 but I had to give up after a few minutes due to > > a bunch of soft lockups. Quite a bunch of processes got stuck in D, > > including emacs starting up and xterms I was attempting to close. > > > > Jul 5 18:36:51 bulky kernel: [ 526.136006] BUG: soft lockup - CPU#0 stuck for 61s! [10.2.4.3-manage:3991] > > Jul 5 18:36:51 bulky kernel: [ 526.136006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:36:51 bulky kernel: re intel_agp > > Jul 5 18:36:51 bulky kernel: [ 526.136006] CPU 0: > > Jul 5 18:36:51 bulky kernel: [ 526.136006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:36:51 bulky kernel: re intel_agp > > Jul 5 18:36:51 bulky kernel: [ 526.136006] Pid: 3991, comm: 10.2.4.3-manage Not tainted 2.6.31-rc2 #1 7454CTO > > Jul 5 18:36:51 bulky kernel: [ 526.136006] RIP: 0010:[] [] rpc_wake_up+0x27/0x7a [sunrpc] > > Jul 5 18:36:51 bulky kernel: [ 526.136006] RSP: 0018:ffff880113573e80 EFLAGS: 00000246 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] RAX: ffff880127c2b1b0 RBX: ffff880127c2b1a8 RCX: ffffffffffffffcf > > Jul 5 18:36:51 bulky kernel: [ 526.136006] RDX: 0000000000002151 RSI: ffff880127c2b0f0 RDI: ffff880127c2b1a8 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] RBP: ffffffff810115ae R08: 0000000000000000 R09: 0000000000000000 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] R10: ffff880028026f18 R11: ffff880028026f18 R12: ffffffffffffffcf > > Jul 5 18:36:51 bulky kernel: [ 526.136006] R13: 0000000000000000 R14: ffff880028026f18 R15: ffff880028026f18 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] FS: 0000000000000000(0000) GS:ffff880028023000(0000) knlGS:0000000000000000 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jul 5 18:36:51 bulky kernel: [ 526.136006] CR2: 0000000001ff3930 CR3: 0000000001001000 CR4: 00000000000426e0 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] Call Trace: > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? nfs4_run_state_manager+0x232/0x2a1 [nfs] > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? nfs4_run_state_manager+0x0/0x2a1 [nfs] > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? kthread+0x84/0x8c > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? child_rip+0xa/0x20 > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? gss_unwrap_resp+0x0/0x1c4 [auth_rpcgss] > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? kthread+0x0/0x8c > > Jul 5 18:36:51 bulky kernel: [ 526.136006] [] ? child_rip+0x0/0x20 > > Jul 5 18:37:57 bulky kernel: [ 591.632007] BUG: soft lockup - CPU#0 stuck for 61s! [10.2.4.3-manage:3991] > > Jul 5 18:37:57 bulky kernel: [ 591.632008] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:37:57 bulky kernel: re intel_agp > > Jul 5 18:37:57 bulky kernel: [ 591.632008] CPU 0: > > Jul 5 18:37:57 bulky kernel: [ 591.632008] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:37:57 bulky kernel: re intel_agp > > Jul 5 18:37:57 bulky kernel: [ 591.632008] Pid: 3991, comm: 10.2.4.3-manage Not tainted 2.6.31-rc2 #1 7454CTO > > Jul 5 18:37:57 bulky kernel: [ 591.632008] RIP: 0010:[] [] nfs4_run_state_manager+0x243/0x2a1 [nfs] > > Jul 5 18:37:57 bulky kernel: [ 591.632008] RSP: 0018:ffff880113573eb0 EFLAGS: 00000202 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] RAX: ffff880127c2b0f0 RBX: ffff880127c2b000 RCX: 0000000000000010 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] RDX: 0000000000009b11 RSI: ffff880127c2b108 RDI: ffffffffa03dec0f > > Jul 5 18:37:57 bulky kernel: [ 591.632008] RBP: ffffffff810115ae R08: 0000000000000010 R09: 0000000000000000 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] R10: ffff880136de4000 R11: 0000000000000040 R12: ffff880127c2b000 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] R13: ffffffff8101140e R14: ffff880127c2b1a8 R15: ffffffff8101140e > > Jul 5 18:37:57 bulky kernel: [ 591.632008] FS: 0000000000000000(0000) GS:ffff880028023000(0000) knlGS:0000000000000000 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jul 5 18:37:57 bulky kernel: [ 591.632008] CR2: 0000000001ff3930 CR3: 0000000001001000 CR4: 00000000000426e0 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] Call Trace: > > Jul 5 18:37:57 bulky kernel: [ 591.632008] [] ? nfs4_run_state_manager+0x0/0x2a1 [nfs] > > Jul 5 18:37:57 bulky kernel: [ 591.632008] [] ? kthread+0x84/0x8c > > Jul 5 18:37:57 bulky kernel: [ 591.632008] [] ? child_rip+0xa/0x20 > > Jul 5 18:37:57 bulky kernel: [ 591.632008] [] ? gss_unwrap_resp+0x0/0x1c4 [auth_rpcgss] > > Jul 5 18:37:57 bulky kernel: [ 591.632008] [] ? kthread+0x0/0x8c > > Jul 5 18:37:57 bulky kernel: [ 591.632008] [] ? child_rip+0x0/0x20 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] BUG: soft lockup - CPU#0 stuck for 61s! [10.2.4.3-manage:3991] > > Jul 5 18:39:01 bulky kernel: [ 657.132006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:39:01 bulky kernel: re intel_agp > > Jul 5 18:39:01 bulky kernel: [ 657.132006] CPU 0: > > Jul 5 18:39:01 bulky kernel: [ 657.132006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:39:01 bulky kernel: re intel_agp > > Jul 5 18:39:01 bulky kernel: [ 657.132006] Pid: 3991, comm: 10.2.4.3-manage Not tainted 2.6.31-rc2 #1 7454CTO > > Jul 5 18:39:01 bulky kernel: [ 657.132006] RIP: 0010:[] [] nfs4_run_state_manager+0x217/0x2a1 [nfs] > > Jul 5 18:39:01 bulky kernel: [ 657.132006] RSP: 0018:ffff880113573eb0 EFLAGS: 00000246 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] RAX: 0000000000000000 RBX: ffff880127c2b000 RCX: ffff880113573e98 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] RDX: 000000000000cec9 RSI: ffff880127c2b108 RDI: ffffffffa03dec0f > > Jul 5 18:39:01 bulky kernel: [ 657.132006] RBP: ffffffff810115ae R08: ffff880113573e98 R09: 0000000000000000 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] R10: ffff880028026f18 R11: ffff880028026f18 R12: ffff880028026f18 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] R13: ffff880028026f18 R14: ffff880127c2b000 R15: ffffffff8101178e > > Jul 5 18:39:01 bulky kernel: [ 657.132006] FS: 0000000000000000(0000) GS:ffff880028023000(0000) knlGS:0000000000000000 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jul 5 18:39:01 bulky kernel: [ 657.132006] CR2: 0000000001ff3930 CR3: 0000000001001000 CR4: 00000000000426e0 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jul 5 18:39:01 bulky kernel: [ 657.132006] Call Trace: > > Jul 5 18:39:01 bulky kernel: [ 657.132542] [] ? nfs4_run_state_manager+0x0/0x2a1 [nfs] > > Jul 5 18:39:01 bulky kernel: [ 657.132542] [] ? kthread+0x84/0x8c > > Jul 5 18:39:01 bulky kernel: [ 657.132542] [] ? child_rip+0xa/0x20 > > Jul 5 18:39:01 bulky kernel: [ 657.132542] [] ? gss_unwrap_resp+0x0/0x1c4 [auth_rpcgss] > > Jul 5 18:39:01 bulky kernel: [ 657.132542] [] ? kthread+0x0/0x8c > > Jul 5 18:39:01 bulky kernel: [ 657.132542] [] ? child_rip+0x0/0x20 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] BUG: soft lockup - CPU#0 stuck for 61s! [10.2.4.3-manage:3991] > > Jul 5 18:40:07 bulky kernel: [ 722.632006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:40:07 bulky kernel: re intel_agp > > Jul 5 18:40:07 bulky kernel: [ 722.632006] CPU 0: > > Jul 5 18:40:07 bulky kernel: [ 722.632006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:40:07 bulky kernel: re intel_agp > > Jul 5 18:40:07 bulky kernel: [ 722.632006] Pid: 3991, comm: 10.2.4.3-manage Not tainted 2.6.31-rc2 #1 7454CTO > > Jul 5 18:40:07 bulky kernel: [ 722.632006] RIP: 0010:[] [] rpc_wake_up+0x5c/0x7a [sunrpc] > > Jul 5 18:40:07 bulky kernel: [ 722.632006] RSP: 0018:ffff880113573e80 EFLAGS: 00000246 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] RAX: ffff880127c2b1b0 RBX: ffff880127c2b1a8 RCX: ffffffffffffff46 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] RDX: 0000000000004a07 RSI: ffff880127c2b108 RDI: ffff880127c2b1a8 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] RBP: ffffffff810115ae R08: 0000000000000000 R09: 0000000000000000 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] R10: ffff880028026f18 R11: ffff880028026f18 R12: ffff880127c2b1a8 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] R13: ffffffff8101140e R14: 0000000000000000 R15: ffffffff810115ae > > Jul 5 18:40:07 bulky kernel: [ 722.632006] FS: 0000000000000000(0000) GS:ffff880028023000(0000) knlGS:0000000000000000 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jul 5 18:40:07 bulky kernel: [ 722.632006] CR2: 0000000001ff3930 CR3: 0000000001001000 CR4: 00000000000426e0 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] Call Trace: > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? nfs4_run_state_manager+0x232/0x2a1 [nfs] > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? nfs4_run_state_manager+0x0/0x2a1 [nfs] > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? kthread+0x84/0x8c > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? child_rip+0xa/0x20 > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? gss_unwrap_resp+0x0/0x1c4 [auth_rpcgss] > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? kthread+0x0/0x8c > > Jul 5 18:40:07 bulky kernel: [ 722.632006] [] ? child_rip+0x0/0x20 > > Jul 5 18:41:12 bulky kernel: [ 788.128005] BUG: soft lockup - CPU#0 stuck for 61s! [10.2.4.3-manage:3991] > > Jul 5 18:41:12 bulky kernel: [ 788.128006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:41:12 bulky kernel: re intel_agp > > Jul 5 18:41:12 bulky kernel: [ 788.128006] CPU 0: > > Jul 5 18:41:12 bulky kernel: [ 788.128006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:41:12 bulky kernel: re intel_agp > > Jul 5 18:41:12 bulky kernel: [ 788.128006] Pid: 3991, comm: 10.2.4.3-manage Not tainted 2.6.31-rc2 #1 7454CTO > > Jul 5 18:41:12 bulky kernel: [ 788.128006] RIP: 0010:[] [] nfs4_run_state_manager+0x243/0x2a1 [nfs] > > Jul 5 18:41:12 bulky kernel: [ 788.128006] RSP: 0018:ffff880113573eb0 EFLAGS: 00000202 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] RAX: ffff880127c2b0f0 RBX: ffff880127c2b000 RCX: ffffffffffffff10 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] RDX: 000000000000ed8a RSI: ffff880127c2b108 RDI: ffffffffa03dec0f > > Jul 5 18:41:12 bulky kernel: [ 788.128006] RBP: ffffffff810115ae R08: ffffffffffffff10 R09: 0000000000000000 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] R10: ffff880028026f18 R11: ffff880028026f18 R12: 0000000000000000 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] R13: 0000000000000002 R14: 0000000000000000 R15: 1eba3d9900ac3c00 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] FS: 0000000000000000(0000) GS:ffff880028023000(0000) knlGS:0000000000000000 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jul 5 18:41:12 bulky kernel: [ 788.128006] CR2: 0000000001ff3930 CR3: 0000000001001000 CR4: 00000000000426e0 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jul 5 18:41:12 bulky kernel: [ 788.128006] Call Trace: > > Jul 5 18:41:12 bulky kernel: [ 788.128538] [] ? nfs4_run_state_manager+0x0/0x2a1 [nfs] > > Jul 5 18:41:12 bulky kernel: [ 788.128538] [] ? kthread+0x84/0x8c > > Jul 5 18:41:12 bulky kernel: [ 788.128538] [] ? child_rip+0xa/0x20 > > Jul 5 18:41:12 bulky kernel: [ 788.128538] [] ? gss_unwrap_resp+0x0/0x1c4 [auth_rpcgss] > > Jul 5 18:41:12 bulky kernel: [ 788.128538] [] ? kthread+0x0/0x8c > > Jul 5 18:41:12 bulky kernel: [ 788.128538] [] ? child_rip+0x0/0x20 > > Jul 5 18:42:05 bulky kernel: [ 840.396622] INFO: task apt-get:4011 blocked for more than 120 seconds. > > Jul 5 18:42:05 bulky kernel: [ 840.396629] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > Jul 5 18:42:05 bulky kernel: [ 840.396635] apt-get D ffff8800280547c0 0 4011 3824 0x00000000 > > Jul 5 18:42:05 bulky kernel: [ 840.396646] ffff88013a4b8800 0000000000000082 ffff88013a4b8800 ffff880028055100 > > Jul 5 18:42:05 bulky kernel: [ 840.396656] ffffffff81016fe3 00000000000147c0 000000000000f7f0 ffff88011353a800 > > Jul 5 18:42:05 bulky kernel: [ 840.396666] ffff88011353aaf8 0000000181045646 0000000000000001 0000000000000046 > > Jul 5 18:42:05 bulky kernel: [ 840.396675] Call Trace: > > Jul 5 18:42:05 bulky kernel: [ 840.396691] [] ? sched_clock+0x5/0x8 > > Jul 5 18:42:05 bulky kernel: [ 840.396704] [] ? schedule_timeout+0x1e/0xb8 > > Jul 5 18:42:05 bulky kernel: [ 840.396712] [] ? wait_for_common+0xd7/0x148 > > Jul 5 18:42:05 bulky kernel: [ 840.396722] [] ? default_wake_function+0x0/0x9 > > Jul 5 18:42:05 bulky kernel: [ 840.396731] [] ? flush_cpu_workqueue+0x6c/0x75 > > Jul 5 18:42:05 bulky kernel: [ 840.396739] [] ? wq_barrier_func+0x0/0x9 > > Jul 5 18:42:05 bulky kernel: [ 840.396746] [] ? flush_workqueue+0x33/0x55 > > Jul 5 18:42:05 bulky kernel: [ 840.396755] [] ? tty_ldisc_release+0x3f/0x7e > > Jul 5 18:42:05 bulky kernel: [ 840.396765] [] ? tty_release_dev+0x45d/0x48e > > Jul 5 18:42:05 bulky kernel: [ 840.396776] [] ? vfs_ioctl+0x21/0x6c > > Jul 5 18:42:05 bulky kernel: [ 840.396783] [] ? tty_release+0x11/0x1a > > Jul 5 18:42:05 bulky kernel: [ 840.396792] [] ? __fput+0xe8/0x190 > > Jul 5 18:42:05 bulky kernel: [ 840.396799] [] ? filp_close+0x5b/0x62 > > Jul 5 18:42:05 bulky kernel: [ 840.396807] [] ? sys_close+0x94/0xcd > > Jul 5 18:42:05 bulky kernel: [ 840.396817] [] ? system_call_fastpath+0x16/0x1b > > Jul 5 18:42:05 bulky kernel: [ 840.396824] INFO: task mcelog:4218 blocked for more than 120 seconds. > > Jul 5 18:42:05 bulky kernel: [ 840.396829] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > Jul 5 18:42:05 bulky kernel: [ 840.396833] mcelog D ffff8800280547c0 0 4218 4217 0x00000000 > > Jul 5 18:42:05 bulky kernel: [ 840.396843] ffff88013a4710c0 0000000000000082 0000000100000041 0000000000000286 > > Jul 5 18:42:05 bulky kernel: [ 840.396852] ffff880113532c00 00000000000147c0 000000000000f7f0 ffff8801105a4080 > > Jul 5 18:42:05 bulky kernel: [ 840.396862] ffff8801105a4378 0000000100010808 000000024a504ac1 0000000000000000 > > Jul 5 18:42:05 bulky kernel: [ 840.396871] Call Trace: > > Jul 5 18:42:05 bulky kernel: [ 840.396880] [] ? schedule_timeout+0x1e/0xb8 > > Jul 5 18:42:05 bulky kernel: [ 840.396889] [] ? rcu_implicit_dynticks_qs+0x6c/0x91 > > Jul 5 18:42:05 bulky kernel: [ 840.396897] [] ? rcu_process_dyntick+0xd2/0xf2 > > Jul 5 18:42:05 bulky kernel: [ 840.396904] [] ? rcu_implicit_dynticks_qs+0x0/0x91 > > Jul 5 18:42:05 bulky kernel: [ 840.396913] [] ? wait_for_common+0xd7/0x148 > > Jul 5 18:42:05 bulky kernel: [ 840.396920] [] ? default_wake_function+0x0/0x9 > > Jul 5 18:42:05 bulky kernel: [ 840.396929] [] ? synchronize_rcu+0x45/0x4b > > Jul 5 18:42:05 bulky kernel: [ 840.396937] [] ? wakeme_after_rcu+0x0/0x9 > > Jul 5 18:42:05 bulky kernel: [ 840.396946] [] ? mce_read+0x12f/0x1d6 > > Jul 5 18:42:05 bulky kernel: [ 840.396954] [] ? vfs_read+0xa6/0xff > > Jul 5 18:42:05 bulky kernel: [ 840.396961] [] ? sys_read+0x45/0x6e > > Jul 5 18:42:05 bulky kernel: [ 840.396970] [] ? system_call_fastpath+0x16/0x1b > > Jul 5 18:42:18 bulky kernel: [ 853.628005] BUG: soft lockup - CPU#0 stuck for 61s! [10.2.4.3-manage:3991] > > Jul 5 18:42:18 bulky kernel: [ 853.628006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:42:18 bulky kernel: re intel_agp > > Jul 5 18:42:18 bulky kernel: [ 853.628006] CPU 0: > > Jul 5 18:42:18 bulky kernel: [ 853.628006] Modules linked in: des_generic hidp hid tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc bnep sco rfcomm l2cap kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_stats cpufreq_userspace cpufreq_conservative rpcsec_gss_krb5 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext2 loop btusb bluetooth snd_hda_codec_conexant arc4 ecb snd_hda_intel snd_hda_codec iwlagn iwlcore snd_hwdep snd_pcm snd_seq snd_timer mac80211 thinkpad_acpi snd_seq_device led_class wmi psmouse cfg80211 serio_raw i2c_i801 snd evdev soundcore nvram rfkill snd_page_alloc ac battery button processor ext3 jbd mbcache sha256_generic aes_x86_64 aes_gener ic cbc dm_crypt dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif uhci_hcd ahci libata scsi_mod ehci_hcd e1000e thermal fan i915 i2c_algo_bit cfbcopyarea video thermal_sy! s output cfbimgblt cfbfillrect drm i2c_co > > Jul 5 18:42:18 bulky kernel: re intel_agp > > Jul 5 18:42:18 bulky kernel: [ 853.628006] Pid: 3991, comm: 10.2.4.3-manage Not tainted 2.6.31-rc2 #1 7454CTO > > Jul 5 18:42:18 bulky kernel: [ 853.628006] RIP: 0010:[] [] bit_waitqueue+0x95/0xa0 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] RSP: 0018:ffff880113573e60 EFLAGS: 00000212 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] RAX: 0000000000000b70 RBX: 0000000000000000 RCX: 0000000000000036 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] RDX: ffff88000000dc00 RSI: 0000000000000000 RDI: 0000000000000000 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] RBP: ffffffff810115ae R08: 0000000000000000 R09: 0000000000000000 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] R10: ffff880028026f18 R11: ffff880028026f18 R12: 0000000000000000 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] R13: ffffffff810115ae R14: 0000000000000082 R15: ffff880127c2b100 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] FS: 0000000000000000(0000) GS:ffff880028023000(0000) knlGS:0000000000000000 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jul 5 18:42:18 bulky kernel: [ 853.628006] CR2: 0000000001ff3930 CR3: 0000000001001000 CR4: 00000000000426e0 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] Call Trace: > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? wake_up_bit+0x11/0x22 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? nfs4_clear_state_manager_bit+0x21/0x2a [nfs] > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? nfs4_run_state_manager+0x232/0x2a1 [nfs] > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? nfs4_run_state_manager+0x0/0x2a1 [nfs] > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? kthread+0x84/0x8c > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? child_rip+0xa/0x20 > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? gss_unwrap_resp+0x0/0x1c4 [auth_rpcgss] > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? kthread+0x0/0x8c > > Jul 5 18:42:18 bulky kernel: [ 853.628006] [] ? child_rip+0x0/0x20 > > > > > > -- > > Paul Collins > > Wellington, New Zealand > > > > Dag vijandelijk luchtschip de huismeester is dood > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com