Greeting's
IPR adapter reset triggers EEH errors and recovered 6 times and than
task hung on today's mainline kernel on powerpc box with loads of hung
task call traces
The problem is not seen with 5.14-rc7 (good) and is last recreated on
5.15-rc1 (bad) kernel
Console logs
EEH: Recovering PHB#134-PE#800000
EEH: PE location: N/A, PHB location: N/A
EEH: Frozen PHB#134-PE#800000 detected
EEH: Call Trace:
EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
EEH: [c008000000501c08] eeh_readl+0x70/0xa0 [ipr]
EEH: [c00800000050e76c] ipr_reset_enable_ioa+0x474/0x5a0 [ipr]
EEH: [c0080000005049a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
EEH: [c008000000508f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
EEH: [c00000000018151c] kthread+0x18c/0x1a0
EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
EEH: PHB#134-PE#800000 has failed 6 times in the last hour and has been
permanently disabled.
EEH: Unable to recover from failure from PHB#134-PE#800000.
Please try reseating or replacing it
EEH: of node=0134:80:00.0
EEH: PCI device/vendor: 034a1014
EEH: PCI cmd/status register: 00100546
EEH: PCI-E capabilities and status follow:
EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
EEH: PCI-E 10: 10830000 00000000 00000000 00000000
EEH: PCI-E 20: 00000000
EEH: PCI-E AER capability register set follows:
EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
EEH: PCI-E AER 30: 00000000 00000000
EEH: Beginning: 'error_detected(permanent failure)'
PCI 0134:80:00.0#800000: EEH: not actionable (1,1,1)
[FAILED] Failed to start udev Wait for Complete Device Initialization.
See 'systemctl status systemd-udev-settle.service' for details.
Starting Device-Mapper Multipath Device Controller...
[ OK ] Started Device-Mapper Multipath Device Controller.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
Starting Create Volatile Files and Directories...
[ OK ] Started Create Volatile Files and Directories.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
EEH: Finished:'error_detected(permanent failure)'
device-mapper: multipath service-time: version 0.3.0 loaded
INFO: task eehd:137 blocked for more than 122 seconds.
Not tainted 5.15.0-rc1-autotest #3
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:eehd state:D stack: 0 pid: 137 ppid: 2
flags:0x00000800
Call Trace:
[c00000000d167650] [c0000000001c329c] __wake_up_common+0xcc/0x290
(unreliable)
[c00000000d167840] [c00000000001e6c8] __switch_to+0x278/0x490
[c00000000d1678a0] [c000000000c8b9bc] __schedule+0x31c/0xa20
[c00000000d167960] [c000000000c8c128] schedule+0x68/0x120
[c00000000d167990] [c00800000050a3c4] __ipr_remove+0x10c/0x4b0 [ipr]
[c00000000d167a60] [c00800000050a828] ipr_remove+0xc0/0x110 [ipr]
[c00000000d167ae0] [c000000000732bd4] pci_device_remove+0x64/0x120
[c00000000d167b20] [c000000000811318]
device_release_driver_internal+0x168/0x2c0
[c00000000d167b60] [c000000000724618] pci_stop_bus_device+0xa8/0x100
[c00000000d167ba0] [c000000000724810]
pci_stop_and_remove_bus_device+0x20/0x40
[c00000000d167bd0] [c000000000075e60] pci_hp_remove_devices+0x90/0x130
[c00000000d167c60] [c000000000053cd8] eeh_handle_normal_event+0x7b8/0xa80
[c00000000d167d40] [c000000000054de8] eeh_event_handler+0x108/0x190
[c00000000d167da0] [c00000000018151c] kthread+0x18c/0x1a0
[c00000000d167e10] [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
INFO: task systemd-udevd:473 blocked for more than 122 seconds.
Not tainted 5.15.0-rc1-autotest #3
--
Regard's
Abdul Haleem
IBM Linux Technology Center