On Thu, Jun 2, 2011 at 7:01 AM, Tejun Heo <[email protected]> wrote:
> Maybe I misunderstood the problem but wasn't the problem about not
> being able to wait for the exit of a leader thread and detach it? ?We
> have reliable (sans exec but that's a different story) exit
> notification with EVENT_EXIT which even reports the exit_code, so I
> don't see what the problem is. ?What am I missing?
The problem is that right now it seems that if tracer doesn't catch
EVENT_EXIT and detach tracee when it sees it, really weird things
happen.
Case 1: tracer traces thread leader. An untraced thread execs.
tracer sees EVENT_EXIT, PTRACE_CONTs tracee, and...
never sees WIFEXITED, waitpid just blocks forever!
Case 2: discovered when I started experiments with current kernel
behavior. No execve involved. I just run two tracees, make
leader exit, then make the other tracee signal itself with fatal signal
(SIGUSR1).
Tracer sees leader's exit, but never sees other tracee's signal!
Please see attached program.
The output on my F15 machine:
4816: thread leader
4816: status:0003057f WIFSTOPPED sig:5 (TRAP) event:CLONE eventdata:0x12d1
4817: status:0000137f WIFSTOPPED sig:19 (STOP) event:none eventdata:0x0
EXITING <=== leader will exit now. tracer sees it:
4816: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0x7700
DYING <=== other thread SIGUSR1's itself. tracer *doesn't see anything*
alarm timed out <=== tracer dies on alarm(1).
Now, if I send a non-fatal signal first, by uncommenting these lines:
// usleep(100*1000);
// VERBOSE("WINCH\n");
// raise(SIGWINCH);
Output:
4834: thread leader
4834: status:0003057f WIFSTOPPED sig:5 (TRAP) event:CLONE eventdata:0x12e3
4835: status:0000137f WIFSTOPPED sig:19 (STOP) event:none eventdata:0x0
EXITING <=== leader will exit now. tracer sees it:
4834: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0x7700
WINCH <=== other thread WINCH's itself. tracer *doesn't see anything*
DYING <=== other thread SIGUSR1's itself. tracer sees it:
4835: status:00000a7f WIFSTOPPED sig:10 (USR1) event:none eventdata:0x0
tracer sees other thread about to die from signal 10:
4835: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0xa
tracer sees other thread die from signal 10:
4835: status:0000000a WIFSIGNALED sig:10 (USR1)
tracer sees leader die from signal 10:
4834: status:0000000a WIFSIGNALED sig:10 (USR1)
tracer is informed "no more tracees":
waitpid returned -1
This is just plain broken, right?
(1) It should have worked in the first case too.
(2) Where is SIGWINCH notification? Why ptrace didn't report it?
Re (2): if I disable leader exit - these lines:
VERBOSE("EXITING\n");
syscall(__NR_exit, 0x77);
then I do see SIGWINCH notification:
4891: thread leader
4891: status:0003057f WIFSTOPPED sig:5 (TRAP) event:CLONE eventdata:0x131c
4892: status:0000137f WIFSTOPPED sig:19 (STOP) event:none eventdata:0x0
WINCH
4891: status:00001c7f WIFSTOPPED sig:28 ((null)) event:none eventdata:0x131c
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DYING
4891: status:00000a7f WIFSTOPPED sig:10 (USR1) event:none eventdata:0x131c
4892: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0xa
4891: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0xa
4892: status:0000000a WIFSIGNALED sig:10 (USR1)
4891: status:0000000a WIFSIGNALED sig:10 (USR1)
waitpid returned -1
Looks like leader's exit throws a wrench into ptrace machinery.
Now I understand why strace DETACHs on exit! :)
--
vda
On 06/02, Denys Vlasenko wrote:
>
> The problem is that right now it seems that if tracer doesn't catch
> EVENT_EXIT and detach tracee when it sees it, really weird things
> happen.
The test-case is wrong afaics...
Perhaps this should be considered as a bug in glibc, I dunno.
> thread1(void *unused)
> {
> // usleep(100*1000);
> // VERBOSE("WINCH\n");
> // raise(SIGWINCH);
>
> usleep(100*1000);
> VERBOSE("DYING\n");
> raise(SIGUSR1);
This doesn't send a signal. This does tgkill(tgid, 0, SIGUSR1) which
fails correctly with -EINVAL.
> static int
> thread_leader(void *unused)
> {
> /* malloc gives sufficiently aligned buffer.
> * long buf[] does not! (on ia64).
> */
> clone2(thread1, malloc(16 * 1024), 16 * 1024, 0
Probably because of this clone2.
Could you test with pthread_create? Or s/raise/tkill/ ?
Oleg.
On 06/02, Oleg Nesterov wrote:
>
> On 06/02, Denys Vlasenko wrote:
> >
> > The problem is that right now it seems that if tracer doesn't catch
> > EVENT_EXIT and detach tracee when it sees it, really weird things
> > happen.
>
> The test-case is wrong afaics...
>
> Perhaps this should be considered as a bug in glibc, I dunno.
>
> > thread1(void *unused)
> > {
> > // usleep(100*1000);
> > // VERBOSE("WINCH\n");
> > // raise(SIGWINCH);
> >
> > usleep(100*1000);
> > VERBOSE("DYING\n");
> > raise(SIGUSR1);
>
> This doesn't send a signal. This does tgkill(tgid, 0, SIGUSR1) which
> fails correctly with -EINVAL.
>
> > static int
> > thread_leader(void *unused)
> > {
> > /* malloc gives sufficiently aligned buffer.
> > * long buf[] does not! (on ia64).
> > */
> > clone2(thread1, malloc(16 * 1024), 16 * 1024, 0
>
> Probably because of this clone2.
>
> Could you test with pthread_create? Or s/raise/tkill/ ?
Btw, did you use -lpthread?
Oleg.
On Thursday 02 June 2011 18:39, Oleg Nesterov wrote:
> On 06/02, Oleg Nesterov wrote:
> >
> > On 06/02, Denys Vlasenko wrote:
> > >
> > > The problem is that right now it seems that if tracer doesn't catch
> > > EVENT_EXIT and detach tracee when it sees it, really weird things
> > > happen.
> >
> > The test-case is wrong afaics...
> >
> > Perhaps this should be considered as a bug in glibc, I dunno.
> >
> > > thread1(void *unused)
> > > {
> > > // usleep(100*1000);
> > > // VERBOSE("WINCH\n");
> > > // raise(SIGWINCH);
> > >
> > > usleep(100*1000);
> > > VERBOSE("DYING\n");
> > > raise(SIGUSR1);
> >
> > This doesn't send a signal. This does tgkill(tgid, 0, SIGUSR1) which
> > fails correctly with -EINVAL.
Yes. After I fixed this, it works as expected. See attached.
The output is:
# ./thread_leader_exit
2876: thread leader
2876: status:0003057f WIFSTOPPED sig:5 (TRAP) event:CLONE eventdata:0xb3d
2877: status:0000137f WIFSTOPPED sig:19 (STOP) event:none eventdata:0x0
Leader exits
2876: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0x7700
Sending WINCH to self (pid,tid:2876,2877)
2877: status:00001c7f WIFSTOPPED sig:28 (WINCH) event:none eventdata:0x0
res:0
Sending USR1 to self (pid,tid:2876,2877)
2877: status:00000a7f WIFSTOPPED sig:10 (USR1) event:none eventdata:0x0
2877: status:0006057f WIFSTOPPED sig:5 (TRAP) event:EXIT eventdata:0xa
2877: status:0000000a WIFSIGNALED sig:10 (USR1)
2876: status:0000000a WIFSIGNALED sig:10 (USR1)
> > > static int
> > > thread_leader(void *unused)
> > > {
> > > /* malloc gives sufficiently aligned buffer.
> > > * long buf[] does not! (on ia64).
> > > */
> > > clone2(thread1, malloc(16 * 1024), 16 * 1024, 0
> >
> > Probably because of this clone2.
This seems to be not a problem (it is defined to clone()).
--
vda
On 06/03, Denys Vlasenko wrote:
>
> On Thursday 02 June 2011 18:39, Oleg Nesterov wrote:
> > > > raise(SIGUSR1);
> > >
> > > This doesn't send a signal. This does tgkill(tgid, 0, SIGUSR1) which
> > > fails correctly with -EINVAL.
>
> Yes. After I fixed this, it works as expected. See attached.
> The output is:
Great.
> > > > thread_leader(void *unused)
> > > > {
> > > > /* malloc gives sufficiently aligned buffer.
> > > > * long buf[] does not! (on ia64).
> > > > */
> > > > clone2(thread1, malloc(16 * 1024), 16 * 1024, 0
> > >
> > > Probably because of this clone2.
>
> This seems to be not a problem (it is defined to clone()).
Doesn't matter.
Unlike pthread_create() which uses CLONE_SETTLS, this doesn't setup
the tls area, and I assume you used -lpthread. In this case it is clear
why raise() doesn't work, pt-raise.c thinks that THREAD_GETMEM(tid)
should always work.
Oleg.
On Friday 03 June 2011 17:29, Oleg Nesterov wrote:
> > > > > thread_leader(void *unused)
> > > > > {
> > > > > /* malloc gives sufficiently aligned buffer.
> > > > > * long buf[] does not! (on ia64).
> > > > > */
> > > > > clone2(thread1, malloc(16 * 1024), 16 * 1024, 0
> > > >
> > > > Probably because of this clone2.
> >
> > This seems to be not a problem (it is defined to clone()).
>
> Doesn't matter.
>
> Unlike pthread_create() which uses CLONE_SETTLS, this doesn't setup
> the tls area, and I assume you used -lpthread. In this case it is clear
> why raise() doesn't work, pt-raise.c thinks that THREAD_GETMEM(tid)
> should always work.
I don't link against pthread.
--
vda
On 06/03, Denys Vlasenko wrote:
>
> On Friday 03 June 2011 17:29, Oleg Nesterov wrote:
> > > > > > thread_leader(void *unused)
> > > > > > {
> > > > > > /* malloc gives sufficiently aligned buffer.
> > > > > > * long buf[] does not! (on ia64).
> > > > > > */
> > > > > > clone2(thread1, malloc(16 * 1024), 16 * 1024, 0
> > > > >
> > > > > Probably because of this clone2.
> > >
> > > This seems to be not a problem (it is defined to clone()).
> >
> > Doesn't matter.
> >
> > Unlike pthread_create() which uses CLONE_SETTLS, this doesn't setup
> > the tls area, and I assume you used -lpthread. In this case it is clear
> > why raise() doesn't work, pt-raise.c thinks that THREAD_GETMEM(tid)
> > should always work.
>
> I don't link against pthread.
Hmm. OK, I was wrong, I thought that the !pt version in raise.c should
work because it does
selftid = THREAD_GETMEM(tid);
if (!selftid) {
selftid = sys_gettid();
THREAD_GETMEM(tid) = selftid;
}
and thus uses the correct tid. But it doesn't work because it uses the
wrong _pid_ by the same reason (tls). It rechecks THREAD_GETMEM(tid)
but not THREAD_GETMEM(pid), then it does
if (!pid)
pid = selftid;
and tgkill() correctly fails again.
Heh,
int tfunc(void *unused)
{
raise(SIGKILL);
printf("WTF? SIGKILL doesn't work\n");
printf("thread: tgid = %d\n", getpid());
exit(0);
}
char stack[32 * 1024];
int main(void)
{
printf("main: tgid = %ld\n", syscall(__NR_getpid));
clone(tfunc, stack + sizeof(stack)/2,
CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD,
NULL);
pause();
assert(0);
return 0;
}
prints
main: tgid = 5959
WTF? SIGKILL doesn't work
thread: tgid = 5960
on my machine. Note that if the main thread uses getpid() (which caches
the returned value in THREAD_GETMEM) instead of syscall, everything works.
And if you remove raise() from tfunc(), the thread prints the correct tgid.
This is because raise() fills THREAD_GETMEM(tid) which is used (why???) by
really_getpid() before sys_getpid().
Funny that...
On your machine you can have the different results, my glibc is rather
old. Anyway, I think we can conclude that there is no kernel bug involved.
I am not brave enough to contact glibc developers, may be you can ;)
Oleg.