2009-01-29 21:16:03

by Randy Dunlap

[permalink] [raw]
Subject: boot hang: async vs. kexec

I (try to) do daily build/boot testing. The newly built kernel
is booted via kexec. This was working until sometime between
2.6.28 and 2.6.29-rc1, so I bisected it.*

git bisect blames this commit:

96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634 is first bad commit
commit 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634
Author: Dave Kleikamp <[email protected]>
Date: Thu Jan 8 09:46:31 2009 -0600

async: Don't call async_synchronize_full_special() while holding sb_lock

sync_filesystems() shouldn't be calling async_synchronize_full_special
while holding a spinlock. The second while loop in that function is the
right place for this anyway.


The new/kexec-loaded kernel hangs during initcalls. The last one that
I can see (via netconsole, might miss a few of the very last lines) is:

calling net_ns_init+0x0/0x14d @ 1
net_namespace: 1008 bytes
initcall net_ns_init+0x0/0x14d returned 0 after 0 usecs



Any ideas/suggestions?
Thanks.



*caveat: This was all done with the "don't use gcc 4.1.[01]
because it miscompiles __weak" patch reverted. Could that
be an issue/problem here? (I'm using gcc 4.1.1.)

--
~Randy


2009-01-29 22:28:23

by Dave Kleikamp

[permalink] [raw]
Subject: Re: boot hang: async vs. kexec

On Thu, 2009-01-29 at 13:15 -0800, Randy Dunlap wrote:
> I (try to) do daily build/boot testing. The newly built kernel
> is booted via kexec. This was working until sometime between
> 2.6.28 and 2.6.29-rc1, so I bisected it.*
>
> git bisect blames this commit:
>
> 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634 is first bad commit
> commit 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634
> Author: Dave Kleikamp <[email protected]>
> Date: Thu Jan 8 09:46:31 2009 -0600
>
> async: Don't call async_synchronize_full_special() while holding sb_lock
>
> sync_filesystems() shouldn't be calling async_synchronize_full_special
> while holding a spinlock. The second while loop in that function is the
> right place for this anyway.
>
>
> The new/kexec-loaded kernel hangs during initcalls. The last one that
> I can see (via netconsole, might miss a few of the very last lines) is:
>
> calling net_ns_init+0x0/0x14d @ 1
> net_namespace: 1008 bytes
> initcall net_ns_init+0x0/0x14d returned 0 after 0 usecs
>
>
>
> Any ideas/suggestions?

I'm not sure about any limitations of git bisect, but it seems unlikely
to me that sync_filesystems() would be getting called this early. Are
any filesystems even mounted at this point?

Does reverting that commit fix the problem? (I would be surprised, but
stranger things have happened.)

> Thanks.
>
>
>
> *caveat: This was all done with the "don't use gcc 4.1.[01]
> because it miscompiles __weak" patch reverted. Could that
> be an issue/problem here? (I'm using gcc 4.1.1.)

I have no idea.

Shaggy
--
David Kleikamp
IBM Linux Technology Center

2009-01-29 23:34:59

by Randy Dunlap

[permalink] [raw]
Subject: Re: boot hang: async vs. kexec

Dave Kleikamp wrote:
> On Thu, 2009-01-29 at 13:15 -0800, Randy Dunlap wrote:
>> I (try to) do daily build/boot testing. The newly built kernel
>> is booted via kexec. This was working until sometime between
>> 2.6.28 and 2.6.29-rc1, so I bisected it.*
>>
>> git bisect blames this commit:
>>
>> 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634 is first bad commit
>> commit 96777fe7b042e5a5d0fe5fb861fcd6cd80ef9634
>> Author: Dave Kleikamp <[email protected]>
>> Date: Thu Jan 8 09:46:31 2009 -0600
>>
>> async: Don't call async_synchronize_full_special() while holding sb_lock
>>
>> sync_filesystems() shouldn't be calling async_synchronize_full_special
>> while holding a spinlock. The second while loop in that function is the
>> right place for this anyway.
>>
>>
>> The new/kexec-loaded kernel hangs during initcalls. The last one that
>> I can see (via netconsole, might miss a few of the very last lines) is:
>>
>> calling net_ns_init+0x0/0x14d @ 1
>> net_namespace: 1008 bytes
>> initcall net_ns_init+0x0/0x14d returned 0 after 0 usecs
>>
>>
>>
>> Any ideas/suggestions?
>
> I'm not sure about any limitations of git bisect, but it seems unlikely
> to me that sync_filesystems() would be getting called this early. Are
> any filesystems even mounted at this point?

I don't think so.

> Does reverting that commit fix the problem? (I would be surprised, but
> stranger things have happened.)

I was also skeptical, and reverting it made no difference.


>> Thanks.
>>
>>
>>
>> *caveat: This was all done with the "don't use gcc 4.1.[01]
>> because it miscompiles __weak" patch reverted. Could that
>> be an issue/problem here? (I'm using gcc 4.1.1.)
>
> I have no idea.

I am now using gcc 4.1.2 and seeing the same boot hang problem.

--
~Randy