LinuxLists.cc - Drives missing at boot

Subject: Re: Drives missing at boot

On 07/03/2010 06:42 PM, Mark Knecht wrote:
> On Sat, Jul 3, 2010 at 9:13 AM, Tejun Heo <[email protected]> wrote:
>> Hello,
>>
>> On 07/03/2010 06:06 PM, Mark Knecht wrote:
>>>> Can you please *attach* full logs of a successful boot and several
>>>> failing boots?
>>>
>>> Certainly? Which logs? dmesg or something else?
>>
>> dmesg output preferably with printk timestamp enabled.

Can you please apply the attached patch, reproduce the problem and
post the kernel log?

Thanks.

--
tejun

Attachments:

resume-dbg.patch (1.03 kB)

2010-07-05 16:48:37

On Mon, Jul 5, 2010 at 11:33 PM, Tejun Heo <[email protected]> wrote:
> Hello,
>
> On 07/05/2010 06:56 PM, Mark Knecht wrote:
>>>>> dmesg output preferably with printk timestamp enabled.
>>>
>>> Can you please apply the attached patch, reproduce the problem and
>>> post the kernel log?
>>
>> I'm sorry. What am I patching? I'm not a kernel developer - not even a
>> programmer - so I'll need some help with this. What's the command I
<SNIP>
>
> Hmm...
>
> $ cd /usr/src/linux && patch -p1 < PATCH_FILE
>
> should do it. You know how to build and install the compiled kernel,
> right?
>
> Thanks.
>
> --
> tejun
>

OK - thanks. The patch seemed to install correctly. I then did

make clean
make && make modules_install

and then a Gentoo command:

modules-rebuild -X rebuild

to pick up any package modules that need to be rebuild when I use a
new kernel. (X drivers, vmware, etc.)

The kernel boots fine:

mark@c2stable ~ $ uname -a
Linux c2stable 2.6.34-gentoo-r1 #4 SMP PREEMPT Tue Jul 6 10:35:15 PDT
2010 x86_64 Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz GenuineIntel
GNU/Linux
mark@c2stable ~ $

I don't know what I'm looking for in the dmesg files but I do see one
message about a SATA Link being down.

Files attached.

Cheers,
Mark

Attachments:

dmesg.IDEAllDrives (60.89 kB)
dmesg.IDEDriveMissing (58.82 kB)
Download all attachments

2010-07-07 05:50:21

by Tejun Heo

[permalink] [raw]

Subject: Re: Drives missing at boot

Hello,

On 07/06/2010 08:13 PM, Mark Knecht wrote:
> OK - thanks. The patch seemed to install correctly. I then did
>
> make clean
> make && make modules_install
>
> and then a Gentoo command:
>
> modules-rebuild -X rebuild
>
> to pick up any package modules that need to be rebuild when I use a
> new kernel. (X drivers, vmware, etc.)
>
> The kernel boots fine:
>
> mark@c2stable ~ $ uname -a
> Linux c2stable 2.6.34-gentoo-r1 #4 SMP PREEMPT Tue Jul 6 10:35:15 PDT
> 2010 x86_64 Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz GenuineIntel
> GNU/Linux
> mark@c2stable ~ $
>
> I don't know what I'm looking for in the dmesg files but I do see one
> message about a SATA Link being down.

Umm... the patched module isn't being loaded. If patched, it should
be reporting whether it's hard or soft resetting and some other debug
messages. One common mistake is not updating initrd with patched
modules. Is "modules-rebuild -X rebuild" the command for initrd
update?

Thanks.

--
tejun

2010-07-07 15:34:33

by Mark Knecht

[permalink] [raw]

Subject: Re: Drives missing at boot

On Tue, Jul 6, 2010 at 10:50 PM, Tejun Heo <[email protected]> wrote:
> Hello,
>
> On 07/06/2010 08:13 PM, Mark Knecht wrote:
>> OK - thanks. The patch seemed to install correctly. I then did
>>
>> make clean
>> make && make modules_install
>>
>> and then a Gentoo command:
>>
>> modules-rebuild -X rebuild
>>
>> to pick up any package modules that need to be rebuild when I use a
>> new kernel. (X drivers, vmware, etc.)
>>
>> The kernel boots fine:
>>
>> mark@c2stable ~ $ uname -a
>> Linux c2stable 2.6.34-gentoo-r1 #4 SMP PREEMPT Tue Jul 6 10:35:15 PDT
>> 2010 x86_64 Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz GenuineIntel
>> GNU/Linux
>> mark@c2stable ~ $
>>
>> I don't know what I'm looking for in the dmesg files but I do see one
>> message about a SATA Link being down.
>
> Umm... the patched module isn't being loaded. If patched, it should
> be reporting whether it's hard or soft resetting and some other debug
> messages. One common mistake is not updating initrd with patched
> modules. Is "modules-rebuild -X rebuild" the command for initrd
> update?
>
> Thanks.
>
> --
> tejun
>
I don't use an initrd.

I don't know what happened with the patch but clearly it wasn't in
there. I wasn't confident so I used the --dry-run option. Maybe I
forgot to remove it or something. Sorry. It's in now.

OK - I don't know if this was you intention but since adding this
patch I've not had a single drive missing failure. I've cold booted
about 8 times and warm booted at least 20 times. Every one has come up
fine. I've even gone so far as to turn off the UPS and sit for 5
minutes before cold booting. Still nothing fails right now.

I've had this sort of statistical thing happen before where it hasn't
failed for days, maybe even weeks, but then it starts failing and
fails every time for awhile. Over the past few days working with you
I've never had to reboot more than twice to get you a file. Now I've
tried 30 times this morning and I've come up with nothing.

I will continue to watch the machine and send you the failing dmesg
file whenever I finally get it. For now I can only attach the passing
file showing the patch is now included.

modules-rebuild is just a little Gentoo script that rebuilds a list of
modules that I've previously set up. Each time I build a new kernel I
rebuild some modules as well as mesa:

xorg-drivers
xorg-input-evdev
xorg-input-keyboard
xorg-input-mouse
xf86-video-ati
xf86-video-fbdev
xf86-video-vmware
vmware-modules
mesa

Clearly I shouldn't need both evdev as well as keyboard/mouse but
there's been problems with hald so I've been flipping back and forth a
bit. mesa is just superstition but I _think_ it's helped a couple of
times.

Thanks!

- Mark

Attachments:

dmesg.IDE_AllDrives (62.07 kB)

2010-07-07 15:48:41

On Wed, Jul 7, 2010 at 8:48 AM, Tejun Heo <[email protected]> wrote:
> Hello,
>
> On 07/07/2010 05:34 PM, Mark Knecht wrote:
>> OK - I don't know if this was you intention but since adding this
>> patch I've not had a single drive missing failure. I've cold booted
>> about 8 times and warm booted at least 20 times. Every one has come up
>> fine. I've even gone so far as to turn off the UPS and sit for 5
>> minutes before cold booting. Still nothing fails right now.
>>
>> I've had this sort of statistical thing happen before where it hasn't
>> failed for days, maybe even weeks, but then it starts failing and
>> fails every time for awhile. Over the past few days working with you
>> I've never had to reboot more than twice to get you a file. Now I've
>> tried 30 times this morning and I've come up with nothing.
>>
>> I will continue to watch the machine and send you the failing dmesg
>> file whenever I finally get it. For now I can only attach the passing
>> file showing the patch is now included.
>
> It seems like SIDPR is a bit more unreliable than the current code can
> handle and the added delay and read could have affected the result.
> Eh... weird. Can you please apply the attached patch instead? The
> only difference is it will print out two SControl values instead of
> one. ie. "XXX SControl after resume = AAA BBB, tries=T". Can you
> please try to boot multiple times and see if AAA and BBB differ
> anytime? If that happens, please attach the boot log. Also, if you
> see one with T > 1, please attach that one too.
>
> Thanks.
>
> --
> tejun
>

4 warm reboots. All 4 said 300 300. However the 4th one only showed an
extra attempt at running the patch code with and also showed Tries =
2. I'm attaching boot #1 and boot #4 for now. I've saved them all if
you need or just want them.

Please note that in all 4 cases all drives were found. Nothing is
missing in any test yet after adding these either of these patches.

I've not tried cold boots yet. That's next.

Cheers,
Mark

Attachments:

dmesg.Patch-R1.Test1 (62.13 kB)
dmesg.Patch-R1.Test4 (62.18 kB)
Download all attachments

2010-07-07 17:26:25

by Tejun Heo

[permalink] [raw]

Subject: Re: Drives missing at boot

Hello,

On 07/07/2010 07:06 PM, Mark Knecht wrote:
> 4 warm reboots. All 4 said 300 300. However the 4th one only showed an
> extra attempt at running the patch code with and also showed Tries =
> 2. I'm attaching boot #1 and boot #4 for now. I've saved them all if
> you need or just want them.
>
> Please note that in all 4 cases all drives were found. Nothing is
> missing in any test yet after adding these either of these patches.
>
> I've not tried cold boots yet. That's next.

Hmm... just in case you're being lucky, please keep an eye on it over
several days and report the result. I think all that's necessary is
slight modification to the resume logic but let's watch a bit first.

Thanks.

--
tejun

2010-07-07 17:32:27

by Mark Knecht

[permalink] [raw]

Subject: Re: Drives missing at boot

On Wed, Jul 7, 2010 at 10:26 AM, Tejun Heo <[email protected]> wrote:
> Hello,
>
> On 07/07/2010 07:06 PM, Mark Knecht wrote:
>> 4 warm reboots. All 4 said 300 300. However the 4th one only showed an
>> extra attempt at running the patch code with and also showed Tries =
>> 2. I'm attaching boot #1 and boot #4 for now. I've saved them all if
>> you need or just want them.
>>
>> Please note that in all 4 cases all drives were found. Nothing is
>> missing in any test yet after adding these either of these patches.
>>
>> I've not tried cold boots yet. That's next.
>
> Hmm... just in case you're being lucky, please keep an eye on it over
> several days and report the result. I think all that's necessary is
> slight modification to the resume logic but let's watch a bit first.
>
> Thanks.
>
> --
> tejun
>
I've tried two cold boots so far. One of them had that same extra
Tries = 2 at the same place. I'll do a couple more just to see what
happens.

I'm happy to watch it as long as it takes and will certainly save any
results if and when a drive isn't found. This problem has run hot and
cold for a few months. Sometimes I go a week with every boot is good.
This last two weeks was finally bad enough to get me to report it.

I also want to investigate actually getting the BIOS AHCI setting
working, but I think I'll leave that alone for afew days so as to not
upset this experiment. Performance on the machine isn't critical right
now, assuming AHCI helps.

I'll get back to you when I've got something new to report.

Cheers,
Mark

2010-07-19 19:31:50

by Mark Knecht

[permalink] [raw]

Subject: Re: Drives missing at boot

2010-07-19 21:01:57

2010-07-20 16:16:34

On Tue, Aug 3, 2010 at 11:55 AM, Randy Dunlap <[email protected]> wrote:
> On Tue, 3 Aug 2010 11:47:25 -0700 Mark Knecht wrote:
>
>> On Tue, Aug 3, 2010 at 11:41 AM, Randy Dunlap <[email protected]> wrote:
>> > On Mon, 2 Aug 2010 15:07:11 -0700 Mark Knecht wrote:
>> >
>> >> On Thu, Jul 22, 2010 at 5:39 AM, Tejun Heo <[email protected]> wrote:
>> >>
>> >> Hi Tejun,
>> >> I'm finally home and trying to get back to this. I'm really a bad
>> >> programmer so I don't know what I've done wrong but it seems patch
>> >> isn't happy with me.
>> >>
>> >> c2stable linux # patch --dry-run -p1 <../ata_piix-sidpr-lock.patch
>> >> patching file drivers/ata/ata_piix.c
>> >> patch: **** malformed patch at line 13:
>> >
>> > Whenever the patch file was saved on this system, line 13 of it was
>> > split (probably by an email client). Whenever I see this, I just
>> > join (merge) that line and the next one and try again... sometimes
>> > several lines are malformed and have to be fixed like this.
>> >
>>
>> Randy,
>> Could very well be what happened. I added line 13 (the printk) by hand
>>
>> <SNIP - ORIGINAL PATCH FILE>
>> struct piix_host_priv {
>> const int *map;
>> u32 saved_iocfg;
>> + spinlock_t sidpr_lock; /* FIXME: remove once locking in EH is fixed */
>> void __iomem *sidpr;
>> };
>>
>> <SNIP - MY CHANGE BY HAND>
>> struct piix_host_priv {
>> const int *map;
>> u32 saved_iocfg;
>> + spinlock_t sidpr_lock; /* FIXME: remove once locking in EH is fixed */
>> + printk("MWK - ata_sidpr patch applied!\n");
>> void __iomem *sidpr;
>> };
>>
>> Maybe I should have just put it on the same line as the previous
>> spinlock command?
>>
>> I'll play with it and see if I can get it working.
>
>
> Ah, so you added a line to a patch file. That means that the patch
> block header must be changed from something like this:
>
> @@ -964,12 +968,15 @@
>
> to like this:
>
> @@ -964,12 +968,16 @@
>
> Any text following the second "@@" is just a comment so it does not matter.
>
> The ,12 ,15 ,16 are all line counts. The ",12" is the number of lines
> in the before version of the patch. The ",15" or ",16" is the number of lines
> in the after version of the patch, so you would need to increase it by 1 if
> you added one line. Or you can just put the printk on the same line as another
> part of the patch and it won't matter. :)
>
>
> ---
> ~Randy
> *** Remember to use Documentation/SubmitChecklist when testing your code ***
>

Thanks Randy. This helped.

OK - so I'm going to hang fire on this problem right now. As part of
this process I've updated to a 2.6.35-gentoo kernel and I'm not seeing
the problem at all so far. I've booted about 25 times over the last 3
days - warm and cold boots mixed - and so far no failures. While I've
had times in the past where earlier kernels didn't exhibit the problem
for days I don't remember a time where I got this many good boots in a
row.

As my problem didn't seem to be exactly the same as Paul's, and since
Tejun expressed surprise that his fix for Paul's issue didn't fix
mine, maybe my problem is actually something different and (hopefully)
already fixed.

I'll continue to use the machine and watch for any failures. If I get
one I'll reengage the list and go into debug mode.

Cheers,
Mark