Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S262805AbTFJODG (ORCPT ); Tue, 10 Jun 2003 10:03:06 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S262878AbTFJODF (ORCPT ); Tue, 10 Jun 2003 10:03:05 -0400 Received: from franka.aracnet.com ([216.99.193.44]:54495 "EHLO franka.aracnet.com") by vger.kernel.org with ESMTP id S262805AbTFJOBS (ORCPT ); Tue, 10 Jun 2003 10:01:18 -0400 Date: Tue, 10 Jun 2003 07:14:08 -0700 From: "Martin J. Bligh" To: Andrew Morton cc: linux-kernel@vger.kernel.org Subject: Re: 2.5.70 (virgin) hangs running SDET Message-ID: <5930000.1055254447@[10.10.2.4]> In-Reply-To: <20030609140834.11ad0d63.akpm@digeo.com> References: <60380000.1055188542@flay> <20030609140834.11ad0d63.akpm@digeo.com> X-Mailer: Mulberry/2.2.1 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 62969 Lines: 1771 > The ps instance which is spinning on kernel_flag in proc_root_lookup is > what's holding things up. > > Or is it spinning in proc_lookup() or proc_pid_lookup()? I have a vague > feeling that I've seen these traces miss out the innermost stack slot... > > Suggest: > > a) Use CONFIG_SPINLINE, get a new sysrq-T trace > > b) Enable spinlock debugging > > c) Try disabling the sched_balance_exec() code. Moving that locking around as discussed last night didn't fix it. It did last 2.5 cycles of repeated SDET this time instead of 0.5, but that might just be coincidence, as the hang looks the same. Got a better backtrace out of her this time though .... Want me to back out the patch I was pointing fingers at? Or I can carry on with the path you suggested above if you think it'll help ... Thanks, M. ps R 00000060 4291798672 21055 20137 21069 (NOTLB) Call Trace: [] proc_pid_lookup+0x16e/0x1d0 [] proc_root_lookup+0x85/0x98 [] real_lookup+0x54/0xc4 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] __user_walk+0x28/0x40 [] vfs_stat+0x19/0x48 [] sys_stat64+0x13/0x30 [] syscall_call+0x7/0xb --------------------------------------------- The whole thing (minus the boring parts): bash S EEE4597C 4291181460 295 293 297 (NOTLB) Call Trace: [] session_of_pgrp+0x27/0x84 [] tiocspgrp+0x6f/0x94 [] sys_wait4+0x20c/0x244 [] sys_rt_sigprocmask+0x128/0x140 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb stressdet S EEE44D3C 4291110652 297 295 16086 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sdetbench S EEDDF39C 4291452472 16086 297 19967 (NOTLB) Call Trace: [] do_sigaction+0x184/0x25c [] do_sigaction+0x1ec/0x25c [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb pdflush S EBA11FE4 4240644464 19965 1 292 (L-TLB) Call Trace: [] __pdflush+0x9e/0x1dc [] pdflush+0x0/0x14 [] pdflush+0xb/0x14 [] kernel_thread_helper+0x5/0xc sh S ECCBB35C 4252419920 19967 16086 19968 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb spec_auto S EC885A5C 4250181712 19968 19967 19980 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb make S EC8847FC 4271321776 19980 19968 19981 19982 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] sys_rt_sigprocmask+0x82/0x140 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb benchrun S EC88543C 36947572 19981 19980 19987 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb tee S EB43992C 4260444592 19982 19968 19980 (NOTLB) Call Trace: [] do_IRQ+0xcf/0x114 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb benchrun S EC884E1C 4269534432 19987 19981 19992 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb make S EAE6419C 22804752 19992 19987 20105 19993 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] sys_rt_sigprocmask+0x82/0x140 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb tee S EB43924C 4291174704 19993 19987 19992 (NOTLB) Call Trace: [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb sh S EC8841DC 7002320 20105 19992 20110 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb time S EAE647BC 6673136 20110 20105 20111 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb time S EB4D273C 49304432 20111 20110 20112 (NOTLB) Call Trace: [] do_sigaction+0x184/0x25c [] do_sigaction+0x1ec/0x25c [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb run.sdet S EB75997C 19620144 20112 20111 20117 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb run S EB75935C 17762208 20117 20112 20119 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb driver S EB758D3C 21826976 20119 20117 20120 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb driver.exec S EB75871C 4272960400 20120 20119 20121 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EE76F99C 4254974224 20121 20120 21638 20122 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EE76F37C 4230375216 20122 20120 21684 20123 20121 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EE76ED5C 4252224848 20123 20120 22609 20125 20122 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EE76E11C 4288371088 20125 20120 22643 20127 20123 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S E9FAF39C 41101072 20127 20120 21662 20128 20125 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S E9FAED7C 17714484 20128 20120 21261 20129 20127 (NOTLB) Call Trace: [] wake_up_forked_process+0x114/0x120 [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S E9FAE75C 11875200 20129 20120 22440 20130 20128 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S E9FAE13C 24607088 20130 20120 21132 20131 20129 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EB5A99DC 20111744 20131 20120 22675 20132 20130 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EB5A93BC 4273489648 20132 20120 21183 20133 20131 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EB5A8D9C 1055968 20133 20120 21634 20134 20132 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EB5A877C 43340592 20134 20120 22160 20137 20133 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EB8C415C 4288125264 20137 20120 21055 20139 20134 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S ECDF875C 4293235536 20139 20120 21197 20140 20137 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S ECDF813C 4251761008 20140 20120 21444 20141 20139 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S ECDF99BC 4269826288 20141 20120 21112 20143 20140 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EA2EC6FC 46306224 20143 20120 21380 20145 20141 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EA2ED95C 15974736 20145 20120 20598 20149 20143 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EC840DBC 4254018880 20149 20120 22393 20151 20145 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EC84017C 4275960112 20151 20120 22236 20152 20149 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb sh S EB971A1C 4266975584 20152 20120 22370 20151 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb make S EB970DDC 32836816 20598 20145 21096 (NOTLB) Call Trace: [] sys_wait4+0x20c/0x244 [] sys_rt_sigprocmask+0x82/0x140 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb ps R 00000060 4291798672 21055 20137 21069 (NOTLB) Call Trace: [] proc_pid_lookup+0x16e/0x1d0 [] proc_root_lookup+0x85/0x98 [] real_lookup+0x54/0xc4 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] __user_walk+0x28/0x40 [] vfs_stat+0x19/0x48 [] sys_stat64+0x13/0x30 [] syscall_call+0x7/0xb wc S EE59B7CC 4294354672 21069 20137 21055 (NOTLB) Call Trace: [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb cc S 00000000 60306288 21096 20598 21161 (NOTLB) Call Trace: [] apic_timer_interrupt+0x1a/0x20 [] release_task+0x31/0x184 [] wait_task_zombie+0x142/0x164 [] sys_wait4+0x12a/0x244 [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] syscall_call+0x7/0xb ps D 00000246 18039088 21112 20141 21116 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.readdir+0x5/0x18 [] sys_getdents64+0x67/0xb0 [] filldir64+0x0/0x10c [] syscall_call+0x7/0xb wc S EE37E68C 4261032896 21116 20141 21112 (NOTLB) Call Trace: [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb as W EB8C5320 4279334592 21161 21096 (L-TLB) Call Trace: [] do_exit+0x365/0x378 [] sys_exit+0xe/0x10 [] syscall_call+0x7/0xb sh R EC289DD0 5369232 21132 20130 (NOTLB) Call Trace: [] __block_commit_write+0x6c/0x94 [] wait_for_completion+0x8a/0xbc [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] set_cpus_allowed+0xfa/0x108 [] sched_migrate_task+0x21/0x34 [] sched_balance_exec+0x2d/0x34 [] do_execve+0x1d/0x20c [] __alloc_pages+0x82/0x2b4 [] invalidate_vcache+0x19/0x88 [] do_lookup+0x18/0x8c [] dput+0x19/0x148 [] cp_new_stat64+0xe7/0x100 [] do_sigaction+0x184/0x25c [] do_sigaction+0x1ec/0x25c [] sys_rt_sigaction+0xae/0xd4 [] getname+0x5d/0x9c [] sys_execve+0x2f/0x6c [] syscall_call+0x7/0xb ps D 00000246 13887184 21183 20132 21186 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.namei+0x5/0x172 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] open_namei+0x8b/0x3f0 [] filp_open+0x3b/0x5c [] sys_open+0x37/0x78 [] syscall_call+0x7/0xb wc S EC55590C 19002304 21186 20132 21183 (NOTLB) Call Trace: [] do_page_fault+0x133/0x478 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb sh R EB6E9DD0 4263313624 21197 20139 (NOTLB) Call Trace: [] wait_for_completion+0x8a/0xbc [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] set_cpus_allowed+0xfa/0x108 [] sched_migrate_task+0x21/0x34 [] sched_balance_exec+0x2d/0x34 [] do_execve+0x1d/0x20c [] __alloc_pages+0x82/0x2b4 [] invalidate_vcache+0x19/0x88 [] do_wp_page+0x396/0x3ac [] handle_mm_fault+0x14c/0x1a0 [] do_page_fault+0x133/0x478 [] do_page_fault+0x0/0x478 [] dput+0x19/0x148 [] do_sigaction+0x184/0x25c [] do_sigaction+0x1ec/0x25c [] sys_rt_sigaction+0xae/0xd4 [] getname+0x5d/0x9c [] sys_execve+0x2f/0x6c [] syscall_call+0x7/0xb ps D 00000246 33562900 21261 20128 21262 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.namei+0x5/0x172 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] open_namei+0x8b/0x3f0 [] filp_open+0x3b/0x5c [] sys_open+0x37/0x78 [] syscall_call+0x7/0xb wc S EF5A196C 18028944 21262 20128 21261 (NOTLB) Call Trace: [] do_page_fault+0x133/0x478 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb ps D 00000246 6288304 21380 20143 21382 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.namei+0x5/0x172 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] open_namei+0x8b/0x3f0 [] filp_open+0x3b/0x5c [] sys_open+0x37/0x78 [] syscall_call+0x7/0xb wc S EC163D8C 4269816048 21382 20143 21380 (NOTLB) Call Trace: [] do_page_fault+0x133/0x478 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb ps D 00000246 4270581392 21444 20140 21445 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.namei+0x5/0x172 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] open_namei+0x8b/0x3f0 [] filp_open+0x3b/0x5c [] sys_open+0x37/0x78 [] syscall_call+0x7/0xb wc S EC555D2C 4294529748 21445 20140 21444 (NOTLB) Call Trace: [] do_page_fault+0x133/0x478 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb ps D 00000246 9704452 21634 20133 21637 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.namei+0x5/0x172 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] open_namei+0x8b/0x3f0 [] filp_open+0x3b/0x5c [] sys_open+0x37/0x78 [] syscall_call+0x7/0xb wc S EE37E26C 4291931332 21637 20133 21634 (NOTLB) Call Trace: [] do_page_fault+0x133/0x478 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb ps D 00000246 4286429648 21638 20121 21650 (NOTLB) Call Trace: [] __down+0x7d/0xe0 [] default_wake_function+0x0/0x20 [] __down_failed+0x8/0xc [] .text.lock.namei+0x5/0x172 [] do_lookup+0x48/0x8c [] link_path_walk+0x557/0x7fc [] path_lookup+0x156/0x15c [] open_namei+0x8b/0x3f0 [] filp_open+0x3b/0x5c [] sys_open+0x37/0x78 [] syscall_call+0x7/0xb wc S ED952D8C 18189552 21650 20121 21638 (NOTLB) Call Trace: [] do_page_fault+0x133/0x478 [] pipe_wait+0x70/0x94 [] autoremove_wake_function+0x0/0x40 [] autoremove_wake_function+0x0/0x40 [] pipe_read+0x19d/0x208 [] do_brk+0x190/0x1c0 [] vfs_read+0x9c/0xcc [] sys_read+0x31/0x4c [] syscall_call+0x7/0xb ps D 00000246 4271391008 21662 20127 21668 (NOTLB) Call Trace: [