On Mon, Oct 26, 2015 at 4:40 AM, Eddie Huang <[email protected]> wrote:
> Hi Kevin,
>
> On Mon, 2015-10-26 at 17:06 +0900, Kevin Hilman wrote:
>> On Mon, Oct 26, 2015 at 4:15 PM, Yingjoe Chen <[email protected]> wrote:
>> > On Mon, 2015-10-26 at 09:56 +0900, Kevin Hilman wrote:
>> >> Hello,
>> >>
>> >> On Sat, Oct 3, 2015 at 12:19 AM, Yingjoe Chen <[email protected]> wrote:
>> >> > Add arch timer node to enable arch-timer support. MT8135 firmware
>> >> > doesn't correctly setup arch-timer frequency and CNTVOFF, add
>> >> > properties to workaround this.
>> >> >
>> >> > This also set cpu enable-method to enable SMP.
>> >> >
>> >> > Signed-off-by: Yingjoe Chen <[email protected]>
>> >>
>> >> kernelci.org started detecting new boot failures for the mt8135-evb in
>> >> the arm-soc tree[1], and the boot failures were bisected down to this
>> >> patch, which landed upstream in the form of commit d186a394bb98 (ARM:
>> >> dts: mt8135: enable basic SMP bringup for mt8135)
>> >>
>> >> Maybe this new SMP support requires updating the firmware on the board
>> >> as well? If so, the changelog should've been a bit more explicit
>> >> about firmware dependencies.
>> >
>> > Kevin,
>> >
>> > Thanks for testing.
>> >
>> > No, it doesn't need new firmware. Will test to see what's going wrong.
>>
>> Thanks for following up. I'll be glad to test any patches if you have
>> anything you'd like me to test.
>
> You need "clockevents/drivers/mtk: Fix spurious interrupt leading to
> crash" (https://lkml.org/lkml/2015/8/24/803)
>
> Daniel Lezcano already applied this patch in his tree for 4.4.
> (https://git.linaro.org/people/daniel.lezcano/linux.git)
The mt8135-evp board is still failing to boot in linux-next[1] (and
has been since next-20151022.) It appears the dependencies are not
quite correct in linux-next either.
Kevin
[1]
http://kernelci.org/boot/mt8135-evbp1/job/next/kernel/next-20151109/defconfig/multi_v7_defconfig/lab/lab-khilman/?_id=564037fe59b514b2f6091909
On Mon, 2015-11-09 at 09:59 -0800, Kevin Hilman wrote:
> On Mon, Oct 26, 2015 at 4:40 AM, Eddie Huang <[email protected]> wrote:
> > Hi Kevin,
> >
> > On Mon, 2015-10-26 at 17:06 +0900, Kevin Hilman wrote:
> >> On Mon, Oct 26, 2015 at 4:15 PM, Yingjoe Chen <[email protected]> wrote:
> >> > On Mon, 2015-10-26 at 09:56 +0900, Kevin Hilman wrote:
> >> >> Hello,
> >> >>
> >> >> On Sat, Oct 3, 2015 at 12:19 AM, Yingjoe Chen <[email protected]> wrote:
> >> >> > Add arch timer node to enable arch-timer support. MT8135 firmware
> >> >> > doesn't correctly setup arch-timer frequency and CNTVOFF, add
> >> >> > properties to workaround this.
> >> >> >
> >> >> > This also set cpu enable-method to enable SMP.
> >> >> >
> >> >> > Signed-off-by: Yingjoe Chen <[email protected]>
> >> >>
> >> >> kernelci.org started detecting new boot failures for the mt8135-evb in
> >> >> the arm-soc tree[1], and the boot failures were bisected down to this
> >> >> patch, which landed upstream in the form of commit d186a394bb98 (ARM:
> >> >> dts: mt8135: enable basic SMP bringup for mt8135)
> >> >>
> >> >> Maybe this new SMP support requires updating the firmware on the board
> >> >> as well? If so, the changelog should've been a bit more explicit
> >> >> about firmware dependencies.
> >> >
> >> > Kevin,
> >> >
> >> > Thanks for testing.
> >> >
> >> > No, it doesn't need new firmware. Will test to see what's going wrong.
> >>
> >> Thanks for following up. I'll be glad to test any patches if you have
> >> anything you'd like me to test.
> >
> > You need "clockevents/drivers/mtk: Fix spurious interrupt leading to
> > crash" (https://lkml.org/lkml/2015/8/24/803)
> >
> > Daniel Lezcano already applied this patch in his tree for 4.4.
> > (https://git.linaro.org/people/daniel.lezcano/linux.git)
>
> The mt8135-evp board is still failing to boot in linux-next[1] (and
> has been since next-20151022.) It appears the dependencies are not
> quite correct in linux-next either.
>
> Kevin
>
> [1]
> http://kernelci.org/boot/mt8135-evbp1/job/next/kernel/next-20151109/defconfig/multi_v7_defconfig/lab/lab-khilman/?_id=564037fe59b514b2f6091909
Hi Kevin,
I check the log [0], it seems first time mt8135-evbp1 boot to kernel
shell successfully, then boot again. In the second time, mt8135 stay in
fastboot mode, waiting host send boot image, then timeout.
I download zImage and dtb in [1], and kernel run to shell successfully
on my platform.
I don't know whether this issue caused by test environment or not.
[0]
http://storage.kernelci.org/next/next-20151109/arm-multi_v7_defconfig/lab-khilman/boot-mt8135-evbp1.txt
[1]
http://kernelci.org/boot/mt8135-evbp1/job/next/kernel/next-20151109/defconfig/multi_v7_defconfig/lab/lab-khilman/?_id=564037fe59b514b2f6091909#bisect-content
Hi Eddie,
[...]
> I check the log [0],
Thanks for checking into this boot failure.
> it seems first time mt8135-evbp1 boot to kernel
> shell successfully, then boot again. In the second time, mt8135 stay in
> fastboot mode, waiting host send boot image, then timeout.
Actually, it never gets to a shell the first time. If you look closely,
the target reboots as soon as userspace starts. Look for the PYBOOT
line which says "finished booting, starting userspace"
Later on, pyboot thinks it finds a root shell due to finding '#'
characters, but clearly it never got to a shell.
> I download zImage and dtb in [1], and kernel run to shell successfully
> on my platform.
Are you can you try using a ramdisk as well? You can use the pre-built
one here:
http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
Please check my boot logs to see how I'm generating the boot.img file
(search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
that the kernel image size with a ramdisk is breaking some of the
assumptions in the fastboot mode. I've seen problems like this on other
platforms due to hard-coded sizes/addresses in the boot firmware.
> I don't know whether this issue caused by test environment or not.
I do not think it is the test environment, since as you can see from
[1], it's booting just fine with mainline kernel, and was working fine
until next-20151102.
Kevin
[1] http://kernelci.org/boot/mt8135-evbp1/job/next/kernel/next-20151110/defconfig/multi_v7_defconfig/lab/lab-khilman/?_id=5642652759b5149878091908
Hi Kevin,
On Tue, 2015-11-10 at 17:16 -0800, Kevin Hilman wrote:
> Hi Eddie,
>
> [...]
>
> > I check the log [0],
>
> Thanks for checking into this boot failure.
>
> > it seems first time mt8135-evbp1 boot to kernel
> > shell successfully, then boot again. In the second time, mt8135 stay in
> > fastboot mode, waiting host send boot image, then timeout.
>
> Actually, it never gets to a shell the first time. If you look closely,
> the target reboots as soon as userspace starts. Look for the PYBOOT
> line which says "finished booting, starting userspace"
>
> Later on, pyboot thinks it finds a root shell due to finding '#'
> characters, but clearly it never got to a shell.
>
> > I download zImage and dtb in [1], and kernel run to shell successfully
> > on my platform.
>
> Are you can you try using a ramdisk as well? You can use the pre-built
> one here:
> http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
>
Yes, I tried this ramdisk, and I can reproduce fail issue.
> Please check my boot logs to see how I'm generating the boot.img file
> (search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
> that the kernel image size with a ramdisk is breaking some of the
> assumptions in the fastboot mode. I've seen problems like this on other
> platforms due to hard-coded sizes/addresses in the boot firmware.
>
MT8135 allocate 10MB for BOOT partition, but the test boot.img is 11MB,
thus cause user space fail. I will prepare new firmware that extend BOOT
partition to 16MB. and put new firmware on Howard's github. I will mail
to you when I am ready..
Eddie
Thanks
Hi Eddie,
Eddie Huang <[email protected]> writes:
> On Tue, 2015-11-10 at 17:16 -0800, Kevin Hilman wrote:
>> Hi Eddie,
>>
>> [...]
>>
>> > I check the log [0],
>>
>> Thanks for checking into this boot failure.
>>
>> > it seems first time mt8135-evbp1 boot to kernel
>> > shell successfully, then boot again. In the second time, mt8135 stay in
>> > fastboot mode, waiting host send boot image, then timeout.
>>
>> Actually, it never gets to a shell the first time. If you look closely,
>> the target reboots as soon as userspace starts. Look for the PYBOOT
>> line which says "finished booting, starting userspace"
>>
>> Later on, pyboot thinks it finds a root shell due to finding '#'
>> characters, but clearly it never got to a shell.
>>
>> > I download zImage and dtb in [1], and kernel run to shell successfully
>> > on my platform.
>>
>> Are you can you try using a ramdisk as well? You can use the pre-built
>> one here:
>> http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
>>
>
> Yes, I tried this ramdisk, and I can reproduce fail issue.
>
OK, good. Thanks for looking into it.
>> Please check my boot logs to see how I'm generating the boot.img file
>> (search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
>> that the kernel image size with a ramdisk is breaking some of the
>> assumptions in the fastboot mode. I've seen problems like this on other
>> platforms due to hard-coded sizes/addresses in the boot firmware.
>>
>
> MT8135 allocate 10MB for BOOT partition, but the test boot.img is 11MB,
> thus cause user space fail.
Aha, I was right! ;)
> I will prepare new firmware that extend BOOT
> partition to 16MB. and put new firmware on Howard's github. I will mail
> to you when I am ready..
Great, thanks for working on this.
Any chance of making it even bigger? We're working on running some more
automated testing from a ramdisk, and those ramdisks will be easily
30-50 Mb with modules included and a rootfs with extra tests.
Kevin
Hi Eddie,
Kevin Hilman <[email protected]> writes:
> Eddie Huang <[email protected]> writes:
>
>> On Tue, 2015-11-10 at 17:16 -0800, Kevin Hilman wrote:
>>> Hi Eddie,
>>>
>>> [...]
>>>
>>> > I check the log [0],
>>>
>>> Thanks for checking into this boot failure.
>>>
>>> > it seems first time mt8135-evbp1 boot to kernel
>>> > shell successfully, then boot again. In the second time, mt8135 stay in
>>> > fastboot mode, waiting host send boot image, then timeout.
>>>
>>> Actually, it never gets to a shell the first time. If you look closely,
>>> the target reboots as soon as userspace starts. Look for the PYBOOT
>>> line which says "finished booting, starting userspace"
>>>
>>> Later on, pyboot thinks it finds a root shell due to finding '#'
>>> characters, but clearly it never got to a shell.
>>>
>>> > I download zImage and dtb in [1], and kernel run to shell successfully
>>> > on my platform.
>>>
>>> Are you can you try using a ramdisk as well? You can use the pre-built
>>> one here:
>>> http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
>>>
>>
>> Yes, I tried this ramdisk, and I can reproduce fail issue.
>>
>
> OK, good. Thanks for looking into it.
>
>>> Please check my boot logs to see how I'm generating the boot.img file
>>> (search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
>>> that the kernel image size with a ramdisk is breaking some of the
>>> assumptions in the fastboot mode. I've seen problems like this on other
>>> platforms due to hard-coded sizes/addresses in the boot firmware.
>>>
>>
>> MT8135 allocate 10MB for BOOT partition, but the test boot.img is 11MB,
>> thus cause user space fail.
>
> Aha, I was right! ;)
Also notice in kernelci.org that the mt8173 board has also been failing
to boot in mainline[1]. I wonder if this same limitation exists in the
mt8173 boot firmware?
Kevin
[1] http://kernelci.org/boot/mt8173-evb/job/mainline/kernel/v4.3-11553-g8d3de01cfa37/defconfig/defconfig/lab/lab-khilman/?_id=5643bc3959b5145c9e0918f4
On Wed, 2015-11-11 at 20:54 -0800, Kevin Hilman wrote:
> Hi Eddie,
>
> Kevin Hilman <[email protected]> writes:
>
> > Eddie Huang <[email protected]> writes:
> >
> >> On Tue, 2015-11-10 at 17:16 -0800, Kevin Hilman wrote:
> >>> Hi Eddie,
> >>>
> >>> [...]
> >>>
> >>> > I check the log [0],
> >>>
> >>> Thanks for checking into this boot failure.
> >>>
> >>> > it seems first time mt8135-evbp1 boot to kernel
> >>> > shell successfully, then boot again. In the second time, mt8135 stay in
> >>> > fastboot mode, waiting host send boot image, then timeout.
> >>>
> >>> Actually, it never gets to a shell the first time. If you look closely,
> >>> the target reboots as soon as userspace starts. Look for the PYBOOT
> >>> line which says "finished booting, starting userspace"
> >>>
> >>> Later on, pyboot thinks it finds a root shell due to finding '#'
> >>> characters, but clearly it never got to a shell.
> >>>
> >>> > I download zImage and dtb in [1], and kernel run to shell successfully
> >>> > on my platform.
> >>>
> >>> Are you can you try using a ramdisk as well? You can use the pre-built
> >>> one here:
> >>> http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
> >>>
> >>
> >> Yes, I tried this ramdisk, and I can reproduce fail issue.
> >>
> >
> > OK, good. Thanks for looking into it.
> >
> >>> Please check my boot logs to see how I'm generating the boot.img file
> >>> (search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
> >>> that the kernel image size with a ramdisk is breaking some of the
> >>> assumptions in the fastboot mode. I've seen problems like this on other
> >>> platforms due to hard-coded sizes/addresses in the boot firmware.
> >>>
> >>
> >> MT8135 allocate 10MB for BOOT partition, but the test boot.img is 11MB,
> >> thus cause user space fail.
> >
> > Aha, I was right! ;)
>
> Also notice in kernelci.org that the mt8173 board has also been failing
> to boot in mainline[1]. I wonder if this same limitation exists in the
> mt8173 boot firmware?
>
MT8173 is another case, the failure is due to following commit:
67e56c5 arm64: dts: mt8173: Add subsystem clock controller device nodes
It is because this commit register MM subsystem clock, but kernel don't
use MM clock yet, then CCF disable it. (My internal platform kernel
command include clk_ignore_unused parameter, so don't have this
problem).I will do further checking and release solution later. Thanks
your testing.
Eddie
Thanks
Eddie Huang <[email protected]> writes:
> On Wed, 2015-11-11 at 20:54 -0800, Kevin Hilman wrote:
>> Hi Eddie,
>>
>> Kevin Hilman <[email protected]> writes:
>>
>> > Eddie Huang <[email protected]> writes:
>> >
>> >> On Tue, 2015-11-10 at 17:16 -0800, Kevin Hilman wrote:
>> >>> Hi Eddie,
>> >>>
>> >>> [...]
>> >>>
>> >>> > I check the log [0],
>> >>>
>> >>> Thanks for checking into this boot failure.
>> >>>
>> >>> > it seems first time mt8135-evbp1 boot to kernel
>> >>> > shell successfully, then boot again. In the second time, mt8135 stay in
>> >>> > fastboot mode, waiting host send boot image, then timeout.
>> >>>
>> >>> Actually, it never gets to a shell the first time. If you look closely,
>> >>> the target reboots as soon as userspace starts. Look for the PYBOOT
>> >>> line which says "finished booting, starting userspace"
>> >>>
>> >>> Later on, pyboot thinks it finds a root shell due to finding '#'
>> >>> characters, but clearly it never got to a shell.
>> >>>
>> >>> > I download zImage and dtb in [1], and kernel run to shell successfully
>> >>> > on my platform.
>> >>>
>> >>> Are you can you try using a ramdisk as well? You can use the pre-built
>> >>> one here:
>> >>> http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
>> >>>
>> >>
>> >> Yes, I tried this ramdisk, and I can reproduce fail issue.
>> >>
>> >
>> > OK, good. Thanks for looking into it.
>> >
>> >>> Please check my boot logs to see how I'm generating the boot.img file
>> >>> (search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
>> >>> that the kernel image size with a ramdisk is breaking some of the
>> >>> assumptions in the fastboot mode. I've seen problems like this on other
>> >>> platforms due to hard-coded sizes/addresses in the boot firmware.
>> >>>
>> >>
>> >> MT8135 allocate 10MB for BOOT partition, but the test boot.img is 11MB,
>> >> thus cause user space fail.
>> >
>> > Aha, I was right! ;)
>>
>> Also notice in kernelci.org that the mt8173 board has also been failing
>> to boot in mainline[1]. I wonder if this same limitation exists in the
>> mt8173 boot firmware?
>>
>
> MT8173 is another case, the failure is due to following commit:
> 67e56c5 arm64: dts: mt8173: Add subsystem clock controller device nodes
>
> It is because this commit register MM subsystem clock, but kernel don't
> use MM clock yet, then CCF disable it. (My internal platform kernel
> command include clk_ignore_unused parameter, so don't have this
> problem).I will do further checking and release solution later. Thanks
> your testing.
OK, thanks for looking into it.
However, since the merge window is very close to closing, unless you can
git a fix out soon (and one that doesn't introduce other bugs),
probablly the right solution is to just revert the above patch so things
are fixed for mainline ASAP.
Kevin
On Thu, 2015-11-12 at 15:56 -0800, Kevin Hilman wrote:
> Eddie Huang <[email protected]> writes:
>
> > On Wed, 2015-11-11 at 20:54 -0800, Kevin Hilman wrote:
> >> Hi Eddie,
> >>
> >> Kevin Hilman <[email protected]> writes:
> >>
> >> > Eddie Huang <[email protected]> writes:
> >> >
> >> >> On Tue, 2015-11-10 at 17:16 -0800, Kevin Hilman wrote:
> >> >>> Hi Eddie,
> >> >>>
> >> >>> [...]
> >> >>>
> >> >>> > I check the log [0],
> >> >>>
> >> >>> Thanks for checking into this boot failure.
> >> >>>
> >> >>> > it seems first time mt8135-evbp1 boot to kernel
> >> >>> > shell successfully, then boot again. In the second time, mt8135 stay in
> >> >>> > fastboot mode, waiting host send boot image, then timeout.
> >> >>>
> >> >>> Actually, it never gets to a shell the first time. If you look closely,
> >> >>> the target reboots as soon as userspace starts. Look for the PYBOOT
> >> >>> line which says "finished booting, starting userspace"
> >> >>>
> >> >>> Later on, pyboot thinks it finds a root shell due to finding '#'
> >> >>> characters, but clearly it never got to a shell.
> >> >>>
> >> >>> > I download zImage and dtb in [1], and kernel run to shell successfully
> >> >>> > on my platform.
> >> >>>
> >> >>> Are you can you try using a ramdisk as well? You can use the pre-built
> >> >>> one here:
> >> >>> http://storage.kernelci.org/images/rootfs/buildroot/armel/rootfs.cpio.gz
> >> >>>
> >> >>
> >> >> Yes, I tried this ramdisk, and I can reproduce fail issue.
> >> >>
> >> >
> >> > OK, good. Thanks for looking into it.
> >> >
> >> >>> Please check my boot logs to see how I'm generating the boot.img file
> >> >>> (search for mkbootimg) with a kernel/dtb/ramdisk. It may be possible
> >> >>> that the kernel image size with a ramdisk is breaking some of the
> >> >>> assumptions in the fastboot mode. I've seen problems like this on other
> >> >>> platforms due to hard-coded sizes/addresses in the boot firmware.
> >> >>>
> >> >>
> >> >> MT8135 allocate 10MB for BOOT partition, but the test boot.img is 11MB,
> >> >> thus cause user space fail.
> >> >
> >> > Aha, I was right! ;)
> >>
> >> Also notice in kernelci.org that the mt8173 board has also been failing
> >> to boot in mainline[1]. I wonder if this same limitation exists in the
> >> mt8173 boot firmware?
> >>
> >
> > MT8173 is another case, the failure is due to following commit:
> > 67e56c5 arm64: dts: mt8173: Add subsystem clock controller device nodes
> >
> > It is because this commit register MM subsystem clock, but kernel don't
> > use MM clock yet, then CCF disable it. (My internal platform kernel
> > command include clk_ignore_unused parameter, so don't have this
> > problem).I will do further checking and release solution later. Thanks
> > your testing.
>
> OK, thanks for looking into it.
>
> However, since the merge window is very close to closing, unless you can
> git a fix out soon (and one that doesn't introduce other bugs),
> probablly the right solution is to just revert the above patch so things
> are fixed for mainline ASAP.
>
I send one patch to fix this problem. Hope this patch can apply to 4.4.
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-November/384800.html
Eddie
Thanks