We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this wait fw_activate let
card in inconsistent state recoverable only by second flash/activate
Reproducer:
[root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
reload_actions_performed:
fw_activate
[root@host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0
dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
[ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
[ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
[ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
[ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
[ 55.647348] ice 0000:ca:00.0: PTP init successful
[ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
[ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
After patch:
[root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
reload_actions_performed:
fw_activate
[root@host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0
Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
Signed-off-by: Petr Oros <[email protected]>
---
drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index d768925785ca79..90ea2203cdc763 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
+#define ICE_EMP_RESET_SLEEP 5000
if (reset_type == ICE_RESET_EMPR) {
/* If an EMP reset has occurred, any previously pending flash
* update will have completed. We no longer know whether or
* not the NVM update EMP reset is restricted.
*/
pf->fw_emp_reset_disabled = false;
+
+ msleep(ICE_EMP_RESET_SLEEP);
}
err = ice_init_all_ctrlq(hw);
--
2.35.1
On 4/12/2022 3:27 AM, Petr Oros wrote:
> We need to wait for EMP reset after firmware flash.
> Code was extracted from OOT driver and without this wait fw_activate let
> card in inconsistent state recoverable only by second flash/activate
>
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
> Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
>
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
Ahh.. good find. I checked a bunch of places, but didn't check here for
differences. :(
For what its worth, I checked the source history of the out-of-tree
driver this came from. It appears to be a workaround added for fixing a
similar issue.
I haven't been able to dig up the full details yet. It appeares to be a
collision with firmware finalizing recovery after the EMP reset.
Still trying to dig for any more information I can find.
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
> Signed-off-by: Petr Oros <[email protected]>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
> index d768925785ca79..90ea2203cdc763 100644
> --- a/drivers/net/ethernet/intel/ice/ice_main.c
> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
>
> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>
> +#define ICE_EMP_RESET_SLEEP 5000
> if (reset_type == ICE_RESET_EMPR) {
> /* If an EMP reset has occurred, any previously pending flash
> * update will have completed. We no longer know whether or
> * not the NVM update EMP reset is restricted.
> */
> pf->fw_emp_reset_disabled = false;
> +
> + msleep(ICE_EMP_RESET_SLEEP);
> }
>
> err = ice_init_all_ctrlq(hw);
[Cc: +Jakob, [email protected]]
Dear Petr,
Thank you for your patch for the regression.
#regzbot ^introduced 399e27dbbd9e94
Am 12.04.22 um 12:27 schrieb Petr Oros:
Please mention the time in the commit message summary:
ice: Wait 5 s for EMP reset after firmware flash
> We need to wait for EMP reset after firmware flash.
> Code was extracted from OOT driver and without this wait fw_activate let
Which OOT driver exactly?
> card in inconsistent state recoverable only by second flash/activate
Please reflow for 75 characters per line, and add a dot/period to the
end of sentences.
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
So, what is the error? `Get PHY capabilities failed status = -5`?
What firmware version did the network card have before and after the update?
> Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
s/don't/doesn’t/
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
Did you try anything less than five seconds? i40e uses one second, cf.
commit 9b13bd53134c (i40e: Increase delay to 1 s after global EMP reset).
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
> Signed-off-by: Petr Oros <[email protected]>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
> index d768925785ca79..90ea2203cdc763 100644
> --- a/drivers/net/ethernet/intel/ice/ice_main.c
> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
>
> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>
> +#define ICE_EMP_RESET_SLEEP 5000
Please append the unit to the macro name.
> if (reset_type == ICE_RESET_EMPR) {
> /* If an EMP reset has occurred, any previously pending flash
> * update will have completed. We no longer know whether or
> * not the NVM update EMP reset is restricted.
> */
> pf->fw_emp_reset_disabled = false;
> +
> + msleep(ICE_EMP_RESET_SLEEP);
> }
>
> err = ice_init_all_ctrlq(hw);
Kind regards,
Paul
From: Petr Oros <[email protected]>
Date: Tue, 12 Apr 2022 12:27:53 +0200
> We need to wait for EMP reset after firmware flash.
> Code was extracted from OOT driver and without this wait fw_activate let
> card in inconsistent state recoverable only by second flash/activate
>
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
> Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
>
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
> Signed-off-by: Petr Oros <[email protected]>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
> index d768925785ca79..90ea2203cdc763 100644
> --- a/drivers/net/ethernet/intel/ice/ice_main.c
> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
>
> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>
> +#define ICE_EMP_RESET_SLEEP 5000
Ooof, 5 sec is a lot! Is there any way to poll the device readiness?
Does it really need the whole 5 sec?
> if (reset_type == ICE_RESET_EMPR) {
> /* If an EMP reset has occurred, any previously pending flash
> * update will have completed. We no longer know whether or
> * not the NVM update EMP reset is restricted.
> */
> pf->fw_emp_reset_disabled = false;
> +
> + msleep(ICE_EMP_RESET_SLEEP);
> }
>
> err = ice_init_all_ctrlq(hw);
> --
> 2.35.1
Thanks,
Al
We need to wait 5 s for EMP reset after firmware flash. Code was extracted
from OOT driver (ice v1.8.3 downloaded from sourceforge). Without this
wait, fw_activate let card in inconsistent state and recoverable only
by second flash/activate. Flash was tested on these fw's:
From -> To
3.00 -> 3.10/3.20
3.10 -> 3.00/3.20
3.20 -> 3.00/3.10
Reproducer:
[root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
reload_actions_performed:
fw_activate
[root@host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0
dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
[ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
[ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
[ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
[ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
[ 55.647348] ice 0000:ca:00.0: PTP init successful
[ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
[ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
Reboot doesn’t help, only second flash/activate with OOT or patched
driver put card back in consistent state.
After patch:
[root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
reload_actions_performed:
fw_activate
[root@host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0
v2 changes:
- fixed format issues
- added info about fw and OOT driver versions
- added time in the commit message summary
- appended the unit to the macro name
Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
Signed-off-by: Petr Oros <[email protected]>
---
drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index d768925785ca79..38825ed2ecd1de 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
+#define ICE_EMP_RESET_SLEEP_MS 5000
if (reset_type == ICE_RESET_EMPR) {
/* If an EMP reset has occurred, any previously pending flash
* update will have completed. We no longer know whether or
* not the NVM update EMP reset is restricted.
*/
pf->fw_emp_reset_disabled = false;
+
+ msleep(ICE_EMP_RESET_SLEEP_MS);
}
err = ice_init_all_ctrlq(hw);
--
2.35.1
Jacob Keller píše v Út 12. 04. 2022 v 09:58 -0700:
>
>
> On 4/12/2022 3:27 AM, Petr Oros wrote:
> > We need to wait for EMP reset after firmware flash.
> > Code was extracted from OOT driver and without this wait
> > fw_activate let
> > card in inconsistent state recoverable only by second
> > flash/activate
> >
> > Reproducer:
> > [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> > E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bi
> > n
> > Preparing to flash
> > [fw.mgmt] Erasing
> > [fw.mgmt] Erasing done
> > [fw.mgmt] Flashing 100%
> > [fw.mgmt] Flashing done 100%
> > [fw.undi] Erasing
> > [fw.undi] Erasing done
> > [fw.undi] Flashing 100%
> > [fw.undi] Flashing done 100%
> > [fw.netlist] Erasing
> > [fw.netlist] Erasing done
> > [fw.netlist] Flashing 100%
> > [fw.netlist] Flashing done 100%
> > Activate new firmware by devlink reload
> > [root@host ~]# devlink dev reload pci/0000:ca:00.0 action
> > fw_activate
> > reload_actions_performed:
> > fw_activate
> > [root@host ~]# ip link show ens7f0
> > 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
> > state DOWN mode DEFAULT group default qlen 1000
> > link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> > altname enp202s0f0
> >
> > dmesg after flash:
> > [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> > [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status
> > = -5, continuing anyway
> > [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully
> > loaded: ICE OS Default Package version 1.3.28.0
> > [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> > [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> > [ 55.647348] ice 0000:ca:00.0: PTP init successful
> > [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware,
> > max number of TCs supported on this port are 8
> > [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in
> > SW mode.
> > [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the
> > hardware
> > [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe
> > bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0
> > (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
> > Reboot don't help, only second flash/activate with OOT or patched
> > driver put card back in consistent state
> >
> > After patch:
> > [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> > E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bi
> > n
> > Preparing to flash
> > [fw.mgmt] Erasing
> > [fw.mgmt] Erasing done
> > [fw.mgmt] Flashing 100%
> > [fw.mgmt] Flashing done 100%
> > [fw.undi] Erasing
> > [fw.undi] Erasing done
> > [fw.undi] Flashing 100%
> > [fw.undi] Flashing done 100%
> > [fw.netlist] Erasing
> > [fw.netlist] Erasing done
> > [fw.netlist] Flashing 100%
> > [fw.netlist] Flashing done 100%
> > Activate new firmware by devlink reload
> > [root@host ~]# devlink dev reload pci/0000:ca:00.0 action
> > fw_activate
> > reload_actions_performed:
> > fw_activate
> > [root@host ~]# ip link show ens7f0
> > 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> > state UP mode DEFAULT group default qlen 1000
> > link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> > altname enp202s0f0
> >
>
> Ahh.. good find. I checked a bunch of places, but didn't check here
> for
> differences. :(
>
> For what its worth, I checked the source history of the out-of-tree
> driver this came from. It appears to be a workaround added for fixing
> a
> similar issue.
>
> I haven't been able to dig up the full details yet. It appeares to be
> a
> collision with firmware finalizing recovery after the EMP reset.
>
> Still trying to dig for any more information I can find.
Interesting time frame could be around this commit:
08771bce330036 ("ice: Continue probe on link/PHY errors")
Petr
>
> > Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation
> > via devlink reload")
> > Signed-off-by: Petr Oros <[email protected]>
> > ---
> > drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/net/ethernet/intel/ice/ice_main.c
> > b/drivers/net/ethernet/intel/ice/ice_main.c
> > index d768925785ca79..90ea2203cdc763 100644
> > --- a/drivers/net/ethernet/intel/ice/ice_main.c
> > +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> > @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf,
> > enum ice_reset_req reset_type)
> >
> > dev_dbg(dev, "rebuilding PF after reset_type=%d\n",
> > reset_type);
> >
> > +#define ICE_EMP_RESET_SLEEP 5000
> > if (reset_type == ICE_RESET_EMPR) {
> > /* If an EMP reset has occurred, any previously
> > pending flash
> > * update will have completed. We no longer know
> > whether or
> > * not the NVM update EMP reset is restricted.
> > */
> > pf->fw_emp_reset_disabled = false;
> > +
> > + msleep(ICE_EMP_RESET_SLEEP);
> > }
> >
> > err = ice_init_all_ctrlq(hw);
>
> -----Original Message-----
> From: Intel-wired-lan <[email protected]> On Behalf Of
> Petr Oros
> Sent: Wednesday, April 13, 2022 9:08 PM
> To: [email protected]
> Cc: ivecera <[email protected]>; [email protected]; intel-wired-
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]
> Subject: [Intel-wired-lan] [PATCH v2] ice: wait 5 s for EMP reset after
> firmware flash
>
> We need to wait 5 s for EMP reset after firmware flash. Code was extracted
> from OOT driver (ice v1.8.3 downloaded from sourceforge). Without this
> wait, fw_activate let card in inconsistent state and recoverable only by
> second flash/activate. Flash was tested on these fw's:
> From -> To
> 3.00 -> 3.10/3.20
> 3.10 -> 3.00/3.20
> 3.20 -> 3.00/3.10
>
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_80
> 00AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
> state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5,
> continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE
> OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number
> of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited
> by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0
> GT/s PCIe x16 link)
> Reboot doesn’t help, only second flash/activate with OOT or patched driver
> put card back in consistent state.
>
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_80
> 00AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> v2 changes:
> - fixed format issues
> - added info about fw and OOT driver versions
> - added time in the commit message summary
> - appended the unit to the macro name
>
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via
> devlink reload")
> Signed-off-by: Petr Oros <[email protected]>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
Tested-by: Gurucharan <[email protected]> (A Contingent worker at Intel)
TWIMC: this mail is primarily send for documentation purposes and for
regzbot, my Linux kernel regression tracking bot. These mails usually
contain '#forregzbot' in the subject, to make them easy to spot and filter.
#regzbot fixed-by: b537752e6cbf0e
On 12.04.22 15:28, Paul Menzel wrote:
> [Cc: +Jakob, [email protected]]
>
>
> Dear Petr,
>
>
> Thank you for your patch for the regression.
>
> #regzbot ^introduced 399e27dbbd9e94
>
>
> Am 12.04.22 um 12:27 schrieb Petr Oros:
>
> Please mention the time in the commit message summary:
>
> ice: Wait 5 s for EMP reset after firmware flash
>
>> We need to wait for EMP reset after firmware flash.
>> Code was extracted from OOT driver and without this wait fw_activate let
>
> Which OOT driver exactly?
>
>> card in inconsistent state recoverable only by second flash/activate
>
> Please reflow for 75 characters per line, and add a dot/period to the
> end of sentences.
>
>> Reproducer:
>> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
>> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
>> Preparing to flash
>> [fw.mgmt] Erasing
>> [fw.mgmt] Erasing done
>> [fw.mgmt] Flashing 100%
>> [fw.mgmt] Flashing done 100%
>> [fw.undi] Erasing
>> [fw.undi] Erasing done
>> [fw.undi] Flashing 100%
>> [fw.undi] Flashing done 100%
>> [fw.netlist] Erasing
>> [fw.netlist] Erasing done
>> [fw.netlist] Flashing 100%
>> [fw.netlist] Flashing done 100%
>> Activate new firmware by devlink reload
>> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
>> reload_actions_performed:
>> fw_activate
>> [root@host ~]# ip link show ens7f0
>> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
>> state DOWN mode DEFAULT group default qlen 1000
>> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
>> altname enp202s0f0
>>
>> dmesg after flash:
>> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
>> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status =
>> -5, continuing anyway
>> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully
>> loaded: ICE OS Default Package version 1.3.28.0
>> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
>> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
>> [ 55.647348] ice 0000:ca:00.0: PTP init successful
>> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max
>> number of TCs supported on this port are 8
>> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW
>> mode.
>> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
>> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe
>> bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable
>> of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
>
> So, what is the error? `Get PHY capabilities failed status = -5`?
>
> What firmware version did the network card have before and after the
> update?
>
>> Reboot don't help, only second flash/activate with OOT or patched
>> driver put card back in consistent state
>
> s/don't/doesn’t/
>
>> After patch:
>> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
>> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
>> Preparing to flash
>> [fw.mgmt] Erasing
>> [fw.mgmt] Erasing done
>> [fw.mgmt] Flashing 100%
>> [fw.mgmt] Flashing done 100%
>> [fw.undi] Erasing
>> [fw.undi] Erasing done
>> [fw.undi] Flashing 100%
>> [fw.undi] Flashing done 100%
>> [fw.netlist] Erasing
>> [fw.netlist] Erasing done
>> [fw.netlist] Flashing 100%
>> [fw.netlist] Flashing done 100%
>> Activate new firmware by devlink reload
>> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
>> reload_actions_performed:
>> fw_activate
>> [root@host ~]# ip link show ens7f0
>> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
>> UP mode DEFAULT group default qlen 1000
>> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
>> altname enp202s0f0
>
> Did you try anything less than five seconds? i40e uses one second, cf.
> commit 9b13bd53134c (i40e: Increase delay to 1 s after global EMP reset).
>
>> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via
>> devlink reload")
>> Signed-off-by: Petr Oros <[email protected]>
>> ---
>> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c
>> b/drivers/net/ethernet/intel/ice/ice_main.c
>> index d768925785ca79..90ea2203cdc763 100644
>> --- a/drivers/net/ethernet/intel/ice/ice_main.c
>> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
>> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf,
>> enum ice_reset_req reset_type)
>> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>> +#define ICE_EMP_RESET_SLEEP 5000
>
> Please append the unit to the macro name.
>
>> if (reset_type == ICE_RESET_EMPR) {
>> /* If an EMP reset has occurred, any previously pending flash
>> * update will have completed. We no longer know whether or
>> * not the NVM update EMP reset is restricted.
>> */
>> pf->fw_emp_reset_disabled = false;
>> +
>> + msleep(ICE_EMP_RESET_SLEEP);
>> }
>> err = ice_init_all_ctrlq(hw);
>
>
> Kind regards,
>
> Paul
>
>