From time to time, my apq8096-db820c fails to boot to userspace, reset back to
fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
cron job with a 1 min reboot entry on the board while leaving a "while 1; do
fastboot boot boot.img; done" on the host pc.
The issue is present in mainline up to 5.2-rc4, using defconfig and:
CONFIG_SCSI_UFS_QCOM=y
CONFIG_PHY_QCOM_QMP=y
CONFIG_PHY_QCOM_UFS=y
but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
mainline), where it's even easier to reproduce (e.g. takes way less reboots to
trigger it).
These are the last lines printed out:
...
[ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
[ 7.538596] phy phy-34000.phy.0: phy init failed --> -110
[ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
[ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
using dummy regulator
Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...
Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Any idea what is going on? Am i doing something wrong?
--
bye,
p.
On Mon, Jun 10, 2019 at 03:44:01PM +0200, Paolo Pisati wrote:
> From time to time, my apq8096-db820c fails to boot to userspace, reset back to
> fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
> cron job with a 1 min reboot entry on the board while leaving a "while 1; do
> fastboot boot boot.img; done" on the host pc.
>
> The issue is present in mainline up to 5.2-rc4, using defconfig and:
>
> CONFIG_SCSI_UFS_QCOM=y
> CONFIG_PHY_QCOM_QMP=y
> CONFIG_PHY_QCOM_UFS=y
>
> but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
> mainline), where it's even easier to reproduce (e.g. takes way less reboots to
> trigger it).
Hello Paolo,
I have a guess of what is going on.
db820c has 3 PCIe controllers,
that shares a singe QMP block (that has clocks, regulators, and resets).
The QMP block has 3 PCIe PHYs, that have their own clocks and resets.
>
> These are the last lines printed out:
> ...
> [ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
^^ here the phy_init() called from pcie-qcom.c
which ends up to a call to qcom_qmp_phy_enable()
which has this code:
ret = qcom_qmp_phy_com_init(qphy);
if (ret)
return ret;
qcom_qmp_phy_com_init() has this code:
if (qmp->init_count++) {
mutex_unlock(&qmp->phy_mutex);
return 0;
}
qcom_qmp_phy_com_init() later fails,
since the common block init time out, so the qmp driver
disables clocks, asserts reset, and disables regulators
> [ 7.538596] phy phy-34000.phy.0: phy init failed --> -110
> [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
^^ here the first PCIe controller instance fails to probe
> [ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
> using dummy regulator
^^ here the second PCIe controller is probed.
it will call phy_init()
which will again call qcom_qmp_phy_enable() which will call
qcom_qmp_phy_com_init()
where this code:
if (qmp->init_count++) {
mutex_unlock(&qmp->phy_mutex);
return 0;
}
now will return 0,
so clocks will never be enabled, resets never deasserted, regulators
never enabled.
since qcom_qmp_phy_com_init() returns success in this case,
qcom_qmp_phy_enable() will try to continue with the init,
and writes to disabled hardware is usually not a good idea.
I think the proper fix for this is:
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
index cd91b4179b10..22352e3b0ec5 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
ret = qcom_qmp_phy_com_init(qphy);
if (ret)
- return ret;
+ goto err_lane_rst;
if (cfg->has_lane_rst) {
ret = reset_control_deassert(qphy->lane_rst);
Kind regards,
Niklas
>
> Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic
> S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
> S - IMAGE_VARIANT_STRING=M8996LAB
> S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
> S - Boot Interface: UFS
> S - Secure Boot: Off
> ...
>
> Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
>
> Any idea what is going on? Am i doing something wrong?
> --
> bye,
> p.
On Tue, Jun 11, 2019 at 07:12:25PM +0200, Niklas Cassel wrote:
> On Mon, Jun 10, 2019 at 03:44:01PM +0200, Paolo Pisati wrote:
> > From time to time, my apq8096-db820c fails to boot to userspace, reset back to
> > fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
> > cron job with a 1 min reboot entry on the board while leaving a "while 1; do
> > fastboot boot boot.img; done" on the host pc.
> >
> > The issue is present in mainline up to 5.2-rc4, using defconfig and:
> >
> > CONFIG_SCSI_UFS_QCOM=y
> > CONFIG_PHY_QCOM_QMP=y
> > CONFIG_PHY_QCOM_UFS=y
> >
> > but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
> > mainline), where it's even easier to reproduce (e.g. takes way less reboots to
> > trigger it).
>
> Hello Paolo,
>
> I have a guess of what is going on.
> db820c has 3 PCIe controllers,
> that shares a singe QMP block (that has clocks, regulators, and resets).
> The QMP block has 3 PCIe PHYs, that have their own clocks and resets.
>
> >
> > These are the last lines printed out:
> > ...
> > [ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> > [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> > [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> > [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
>
> ^^ here the phy_init() called from pcie-qcom.c
> which ends up to a call to qcom_qmp_phy_enable()
>
> which has this code:
>
> ret = qcom_qmp_phy_com_init(qphy);
> if (ret)
> return ret;
>
> qcom_qmp_phy_com_init() has this code:
>
> if (qmp->init_count++) {
> mutex_unlock(&qmp->phy_mutex);
> return 0;
> }
>
> qcom_qmp_phy_com_init() later fails,
> since the common block init time out, so the qmp driver
> disables clocks, asserts reset, and disables regulators
>
>
> > [ 7.538596] phy phy-34000.phy.0: phy init failed --> -110
> > [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
>
> ^^ here the first PCIe controller instance fails to probe
>
> > [ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
> > using dummy regulator
>
> ^^ here the second PCIe controller is probed.
>
> it will call phy_init()
>
> which will again call qcom_qmp_phy_enable() which will call
> qcom_qmp_phy_com_init()
>
> where this code:
>
> if (qmp->init_count++) {
> mutex_unlock(&qmp->phy_mutex);
> return 0;
> }
>
> now will return 0,
>
> so clocks will never be enabled, resets never deasserted, regulators
> never enabled.
>
> since qcom_qmp_phy_com_init() returns success in this case,
> qcom_qmp_phy_enable() will try to continue with the init,
> and writes to disabled hardware is usually not a good idea.
>
> I think the proper fix for this is:
>
> diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
> index cd91b4179b10..22352e3b0ec5 100644
> --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
>
> ret = qcom_qmp_phy_com_init(qphy);
> if (ret)
> - return ret;
> + goto err_lane_rst;
>
> if (cfg->has_lane_rst) {
> ret = reset_control_deassert(qphy->lane_rst);
>
>
>
> Kind regards,
> Niklas
>
> >
> > Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic
> > S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
> > S - IMAGE_VARIANT_STRING=M8996LAB
> > S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
> > S - Boot Interface: UFS
> > S - Secure Boot: Off
> > ...
> >
> > Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
> >
> > Any idea what is going on? Am i doing something wrong?
> > --
> > bye,
> > p.
Adding Vivek.
On Wed, Jun 12, 2019 at 03:17:35PM +0200, Niklas Cassel wrote:
> > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > index cd91b4179b10..22352e3b0ec5 100644
> > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
> >
> > ret = qcom_qmp_phy_com_init(qphy);
> > if (ret)
> > - return ret;
> > + goto err_lane_rst;
> >
> > if (cfg->has_lane_rst) {
> > ret = reset_control_deassert(qphy->lane_rst);
Hi Niklas,
unfortunately, it didn't help - i added a printk, to highlight when it failed:
--- a/drivers/phy/qualcomm/phy-qcom-qmp.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
@@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy)
}
ret = qcom_qmp_phy_com_init(qphy);
- if (ret)
- return ret;
+ if (ret) {
+ dev_err(qmp->dev, "qphy initialization failed\n");
+ goto err_lane_rst;
+ }
if (cfg->has_lane_rst) {
ret = reset_control_deassert(qphy->lane_rst);
After several reboots i was able to trigger the phy init failure again:
...
[ 2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[ 2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[ 2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[ 2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out
[ 2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed
...
these are the last lines printed, before rebooting in fastboot and
locking up there (as before[*]).
So, as far as i understand there are two distinct problems:
1) sometimes, qcom-qmp-phy fails to initialize
2) and when that happens, the failure is fatal and it led to a reboot & lockup
in fastboot
1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
--
bye,
p.
On Wed, Jun 12, 2019 at 04:09:11PM +0200, Paolo Pisati wrote:
> On Wed, Jun 12, 2019 at 03:17:35PM +0200, Niklas Cassel wrote:
> > > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > > index cd91b4179b10..22352e3b0ec5 100644
> > > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> > > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
> > >
> > > ret = qcom_qmp_phy_com_init(qphy);
> > > if (ret)
> > > - return ret;
> > > + goto err_lane_rst;
> > >
> > > if (cfg->has_lane_rst) {
> > > ret = reset_control_deassert(qphy->lane_rst);
>
> Hi Niklas,
> unfortunately, it didn't help - i added a printk, to highlight when it failed:
>
> --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> @@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy)
> }
>
> ret = qcom_qmp_phy_com_init(qphy);
> - if (ret)
> - return ret;
> + if (ret) {
> + dev_err(qmp->dev, "qphy initialization failed\n");
> + goto err_lane_rst;
> + }
>
> if (cfg->has_lane_rst) {
> ret = reset_control_deassert(qphy->lane_rst);
>
> After several reboots i was able to trigger the phy init failure again:
>
> ...
> [ 2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> [ 2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> [ 2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> [ 2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out
> [ 2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed
> ...
I still think that the above patch is correct,
even though it didn't fix your problem.
If you try to disable the two other PCIe controllers:
diff --git a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
index 943f69912074..95900fe99f89 100644
--- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
+++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
@@ -455,12 +455,12 @@
};
pcie@608000 {
- status = "okay";
+ status = "disabled";
perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>;
};
pcie@610000 {
- status = "okay";
+ status = "disabled";
perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>;
};
};
Can you still reproduce the reboot?
Kind regards,
Niklas
>
> these are the last lines printed, before rebooting in fastboot and
> locking up there (as before[*]).
>
> So, as far as i understand there are two distinct problems:
>
> 1) sometimes, qcom-qmp-phy fails to initialize
>
> 2) and when that happens, the failure is fatal and it led to a reboot & lockup
> in fastboot
>
> 1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
> --
> bye,
> p.
On 10/06/2019 15:44, Paolo Pisati wrote:
> From time to time, my apq8096-db820c fails to boot to userspace, reset back to
> fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
> cron job with a 1 min reboot entry on the board while leaving a "while 1; do
> fastboot boot boot.img; done" on the host pc.
>
> The issue is present in mainline up to 5.2-rc4, using defconfig and:
>
> CONFIG_SCSI_UFS_QCOM=y
> CONFIG_PHY_QCOM_QMP=y
> CONFIG_PHY_QCOM_UFS=y
>
> but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
> mainline), where it's even easier to reproduce (e.g. takes way less reboots to
> trigger it).
>
> These are the last lines printed out:
> ...
> [ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
> [ 7.538596] phy phy-34000.phy.0: phy init failed --> -110
> [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
> [ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
> using dummy regulator
>
> Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic
> S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
> S - IMAGE_VARIANT_STRING=M8996LAB
> S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
> S - Boot Interface: UFS
> S - Secure Boot: Off
> ...
>
> Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
>
> Any idea what is going on? Am i doing something wrong?
Hmmm... this might be related to:
https://patchwork.kernel.org/patch/10976217/
Regards.
On Wed, Jun 12, 2019 at 06:20:48PM +0200, Niklas Cassel wrote:
>
> Can you still reproduce the reboot?
--- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
+++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
@@ -447,12 +447,12 @@
};
pcie@608000 {
- status = "okay";
+ status = "disabled";
perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>;
};
pcie@610000 {
- status = "okay";
+ status = "disabled";
perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>;
};
};
disabling the two other pci controllers (and leaving your previous patch
applied), made the issue more frequent: i was never able to boot to user space
three times in a row without experiencing a crash, previously it was more
sporadic.
On the other hand, now i can reproduce the issue more easily this way.
--
bye,
p.