Hello,
Pulling today's upstream 6.3+ tree, my system now hangs loading initrd. Nothing useful is seen on
serial console after that. This was working late last week. Other older kernels boot OK, so
I think is is related to the kernel.
I have not yet started bisect, curious if this is already know problem.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
> Hello,
>
> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd. Nothing useful is seen on
> serial console after that. This was working late last week. Other older kernels boot OK, so
> I think is is related to the kernel.
What is your setup (especially kernel config)?
Is v6.3 works for you? I haven't seen this regression on my computer
using v6.3.
>
> I have not yet started bisect, curious if this is already know problem.
Certainly you should try bisection.
Thanks.
--
An old man doll... just what I always wanted! - Clara
On 03.05.23 04:31, Bagas Sanjaya wrote:
> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>
>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
Just to understand this properly: you mean after the boot loader said it
loaded the initramfs, not when the kernel starts using it?
>> Nothing useful is seen on
>> serial console after that. This was working late last week. Other older kernels boot OK, so
>> I think is is related to the kernel.
>
> What is your setup (especially kernel config)?
>
> Is v6.3 works for you? I haven't seen this regression on my computer
> using v6.3.
Ben, you just to be sure might want to check if a kernel image that
currently works still boots if you regenerate its initramfs: maybe the
initramfs generator or something it uses changed recently and is causing
your problem. I've seen a case or two where that was the case.
>> I have not yet started bisect, curious if this is already know problem.
> Certainly you should try bisection.
Yeah.
BTW: What kind of platform is this?
Ciao, Thorsten
On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 03.05.23 04:31, Bagas Sanjaya wrote:
>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>
>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>
> Just to understand this properly: you mean after the boot loader said it
> loaded the initramfs, not when the kernel starts using it?
I am bisecting...it appears to be .config related. If I skip enabling things
during bisect, then I do not see the problem. But I copied my original buggy .config
into an otherwise good kernel commit, and now it fails. I'm manually bisecting
the .config settings....
Thanks,
Ben
>
>>> Nothing useful is seen on
>>> serial console after that. This was working late last week. Other older kernels boot OK, so
>>> I think is is related to the kernel.
>>
>> What is your setup (especially kernel config)?
>>
>> Is v6.3 works for you? I haven't seen this regression on my computer
>> using v6.3.
>
> Ben, you just to be sure might want to check if a kernel image that
> currently works still boots if you regenerate its initramfs: maybe the
> initramfs generator or something it uses changed recently and is causing
> your problem. I've seen a case or two where that was the case.
>
>>> I have not yet started bisect, curious if this is already know problem.
>> Certainly you should try bisection.
>
> Yeah.
>
> BTW: What kind of platform is this?
>
> Ciao, Thorsten
>
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 5/4/2023 5:19 AM, Ben Greear wrote:
> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>
>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>
>> Just to understand this properly: you mean after the boot loader said it
>> loaded the initramfs, not when the kernel starts using it?
>
> I am bisecting...it appears to be .config related. If I skip enabling
> things
> during bisect, then I do not see the problem. But I copied my original
> buggy .config
> into an otherwise good kernel commit, and now it fails. I'm manually
> bisecting
> the .config settings....
>
> Thanks,
> Ben
>
Hello Ben,
(CCing Jens here)
I was facing same issue too. Surprising I did not hit the same on the
guest VM.
After bisecting, found that Reverting
[9f4107b07b17b5ee68af680150f91227bea2df6f] block: store
bdev->bd_disk->fops->submit_bio state in bdev
Helped me to boot back the system. Can you please confirm
Thanks
- Raghu
>>
>>>> Nothing useful is seen on
>>>> serial console after that. This was working late last week. Other
>>>> older kernels boot OK, so
>>>> I think is is related to the kernel.
>>>
>>> What is your setup (especially kernel config)?
>>>
>>> Is v6.3 works for you? I haven't seen this regression on my computer
>>> using v6.3.
>>
>> Ben, you just to be sure might want to check if a kernel image that
>> currently works still boots if you regenerate its initramfs: maybe the
>> initramfs generator or something it uses changed recently and is causing
>> your problem. I've seen a case or two where that was the case.
>>
>>>> I have not yet started bisect, curious if this is already know problem.
>>> Certainly you should try bisection.
>>
>> Yeah.
>>
>> BTW: What kind of platform is this?
>>
>> Ciao, Thorsten
>>
>
On 5/4/23 10:38 AM, Raghavendra K T wrote:
> On 5/4/2023 5:19 AM, Ben Greear wrote:
>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>
>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>
>>> Just to understand this properly: you mean after the boot loader said it
>>> loaded the initramfs, not when the kernel starts using it?
>>
>> I am bisecting...it appears to be .config related. If I skip enabling things
>> during bisect, then I do not see the problem. But I copied my original buggy .config
>> into an otherwise good kernel commit, and now it fails. I'm manually bisecting
>> the .config settings....
>>
>> Thanks,
>> Ben
>>
>
> Hello Ben,
>
> (CCing Jens here)
>
> I was facing same issue too. Surprising I did not hit the same on the
> guest VM.
>
> After bisecting, found that Reverting
> [9f4107b07b17b5ee68af680150f91227bea2df6f] block: store bdev->bd_disk->fops->submit_bio state in bdev
>
> Helped me to boot back the system. Can you please confirm
Can you try and pull:
git://git.kernel.dk/linux for-6.4/block
into current -git and see if that works for you? There's a fix for that
commit in there.
--
Jens Axboe
On 5/3/23 16:49, Ben Greear wrote:
> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>
>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>
>> Just to understand this properly: you mean after the boot loader said it
>> loaded the initramfs, not when the kernel starts using it?
>
> I am bisecting...it appears to be .config related. If I skip enabling things
> during bisect, then I do not see the problem. But I copied my original buggy .config
> into an otherwise good kernel commit, and now it fails. I'm manually bisecting
> the .config settings....
I had enabled this option: CONFIG_HYPERV_VTL_MODE
And that makes it not boot. Not sure that is expected or not, but I will simply un-select it
and continue on, as I do not actually need that feature.
Thanks,
Ben
From: Ben Greear <[email protected]> Sent: Thursday, May 4, 2023 10:23 AM
>
> On 5/3/23 16:49, Ben Greear wrote:
> > On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
> >> On 03.05.23 04:31, Bagas Sanjaya wrote:
> >>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
> >>>>
> >>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
> >>
> >> Just to understand this properly: you mean after the boot loader said it
> >> loaded the initramfs, not when the kernel starts using it?
> >
> > I am bisecting...it appears to be .config related. If I skip enabling things
> > during bisect, then I do not see the problem. But I copied my original buggy .config
> > into an otherwise good kernel commit, and now it fails. I'm manually bisecting
> > the .config settings....
>
> I had enabled this option: CONFIG_HYPERV_VTL_MODE
>
> And that makes it not boot. Not sure that is expected or not, but
> I will simply un-select it and continue on, as I do not actually need that feature.
>
What you observed is expected when CONFIG_HYPERV_VTL_MODE is selected,
which is why it defaults to N. The comment for that option in the Kconfig file says:
Select this option to build a Linux kernel to run at a VTL other than
the normal VTL0, which currently is only VTL2. This option
initializes the x86 platform for VTL2, and adds the ability to boot
secondary CPUs directly into 64-bit context as required for VTLs other
than 0. A kernel built with this option must run at VTL2, and will
not run as a normal guest.
Note the last sentence.
Michael
On 5/4/2023 10:14 PM, Jens Axboe wrote:
> On 5/4/23 10:38 AM, Raghavendra K T wrote:
>> On 5/4/2023 5:19 AM, Ben Greear wrote:
>>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>>
>>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>>
>>>> Just to understand this properly: you mean after the boot loader said it
>>>> loaded the initramfs, not when the kernel starts using it?
>>>
>>> I am bisecting...it appears to be .config related. If I skip enabling things
>>> during bisect, then I do not see the problem. But I copied my original buggy .config
>>> into an otherwise good kernel commit, and now it fails. I'm manually bisecting
>>> the .config settings....
>>>
>>> Thanks,
>>> Ben
>>>
>>
>> Hello Ben,
>>
>> (CCing Jens here)
>>
>> I was facing same issue too. Surprising I did not hit the same on the
>> guest VM.
>>
>> After bisecting, found that Reverting
>> [9f4107b07b17b5ee68af680150f91227bea2df6f] block: store bdev->bd_disk->fops->submit_bio state in bdev
>>
>> Helped me to boot back the system. Can you please confirm
>
> Can you try and pull:
>
> git://git.kernel.dk/linux for-6.4/block
>
> into current -git and see if that works for you? There's a fix for that
> commit in there.
>
Jens, pulling for-6.4/block helped in my case.
Thank you :)
On 5/4/23 12:48 PM, Raghavendra K T wrote:
> On 5/4/2023 10:14 PM, Jens Axboe wrote:
>> On 5/4/23 10:38 AM, Raghavendra K T wrote:
>>> On 5/4/2023 5:19 AM, Ben Greear wrote:
>>>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>>>
>>>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>>>
>>>>> Just to understand this properly: you mean after the boot loader said it
>>>>> loaded the initramfs, not when the kernel starts using it?
>>>>
>>>> I am bisecting...it appears to be .config related. If I skip enabling things
>>>> during bisect, then I do not see the problem. But I copied my original buggy .config
>>>> into an otherwise good kernel commit, and now it fails. I'm manually bisecting
>>>> the .config settings....
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>
>>> Hello Ben,
>>>
>>> (CCing Jens here)
>>>
>>> I was facing same issue too. Surprising I did not hit the same on the
>>> guest VM.
>>>
>>> After bisecting, found that Reverting
>>> [9f4107b07b17b5ee68af680150f91227bea2df6f] block: store bdev->bd_disk->fops->submit_bio state in bdev
>>>
>>> Helped me to boot back the system. Can you please confirm
>>
>> Can you try and pull:
>>
>> git://git.kernel.dk/linux for-6.4/block
>>
>> into current -git and see if that works for you? There's a fix for that
>> commit in there.
>>
>
> Jens, pulling for-6.4/block helped in my case.
Great! It will be heading to Linus's tree before -rc1.
--
Jens Axboe
On 5/4/23 10:59, Michael Kelley (LINUX) wrote:
> From: Ben Greear <[email protected]> Sent: Thursday, May 4, 2023 10:23 AM
>>
>> On 5/3/23 16:49, Ben Greear wrote:
>>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
>>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
>>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
>>>>>>
>>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
>>>>
>>>> Just to understand this properly: you mean after the boot loader said it
>>>> loaded the initramfs, not when the kernel starts using it?
>>>
>>> I am bisecting...it appears to be .config related. If I skip enabling things
>>> during bisect, then I do not see the problem. But I copied my original buggy .config
>>> into an otherwise good kernel commit, and now it fails. I'm manually bisecting
>>> the .config settings....
>>
>> I had enabled this option: CONFIG_HYPERV_VTL_MODE
>>
>> And that makes it not boot. Not sure that is expected or not, but
>> I will simply un-select it and continue on, as I do not actually need that feature.
>>
>
> What you observed is expected when CONFIG_HYPERV_VTL_MODE is selected,
> which is why it defaults to N. The comment for that option in the Kconfig file says:
>
> Select this option to build a Linux kernel to run at a VTL other than
> the normal VTL0, which currently is only VTL2. This option
> initializes the x86 platform for VTL2, and adds the ability to boot
> secondary CPUs directly into 64-bit context as required for VTLs other
> than 0. A kernel built with this option must run at VTL2, and will
> not run as a normal guest.
I didn't read it close enough, but that is my fault.
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
From: Ben Greear <[email protected]> Sent: Thursday, May 4, 2023 4:46 PM
>
> On 5/4/23 10:59, Michael Kelley (LINUX) wrote:
> > From: Ben Greear <[email protected]> Sent: Thursday, May 4, 2023 10:23 AM
> >>
> >> On 5/3/23 16:49, Ben Greear wrote:
> >>> On 5/2/23 21:46, Linux regression tracking (Thorsten Leemhuis) wrote:
> >>>> On 03.05.23 04:31, Bagas Sanjaya wrote:
> >>>>> On Tue, May 02, 2023 at 02:03:33PM -0700, Ben Greear wrote:
> >>>>>>
> >>>>>> Pulling today's upstream 6.3+ tree, my system now hangs loading initrd.
> >>>>
> >>>> Just to understand this properly: you mean after the boot loader said it
> >>>> loaded the initramfs, not when the kernel starts using it?
> >>>
> >>> I am bisecting...it appears to be .config related.? If I skip enabling things
> >>> during bisect, then I do not see the problem.? But I copied my original buggy .config
> >>> into an otherwise good kernel commit, and now it fails.? I'm manually bisecting
> >>> the .config settings....
> >>
> >> I had enabled this option: CONFIG_HYPERV_VTL_MODE
> >>
> >> And that makes it not boot. Not sure that is expected or not, but
> >> I will simply un-select it and continue on, as I do not actually need that feature.
> >>
> >
> > What you observed is expected when CONFIG_HYPERV_VTL_MODE is selected,
> > which is why it defaults to N. The comment for that option in the Kconfig file says:
> >
> > Select this option to build a Linux kernel to run at a VTL other than
> > the normal VTL0, which currently is only VTL2. This option
> > initializes the x86 platform for VTL2, and adds the ability to boot
> > secondary CPUs directly into 64-bit context as required for VTLs other
> > than 0. A kernel built with this option must run at VTL2, and will
> > not run as a normal guest.
>
> I didn't read it close enough, but that is my fault.
>
No problem. :-) Just FYI, in the future if you are come across issues
specifically related to running Linux as a guest on Hyper-V, feel free
to copy the [email protected] mailing list. That may
elicit faster help from someone with Hyper-V expertise and potentially
save you some trouble.
Michael