Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 816
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_get_inode_loc: unable to read inode block - inode=2, block=316
Oct 16 21:58:54 inif kernel: ------------[ cut here ]------------
Oct 16 21:58:54 inif kernel: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
Oct 16 21:58:54 inif kernel: Modules linked in: crc32c crypto_hash libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi firmware_class scsi_transport_fc tcp_highspeed w83627hf hwmon_vid eeprom adm1021 hwmon binfmt_misc dm_mirror dm_log dm_mod pci_slot battery ac ehci_hcd uhci_hcd sg e1000 pcspkr i2c_i801 i2c_core ide_cd_mod cdrom e7xxx_edac button usbcore [last unloaded: qla2xxx]
Oct 16 21:58:54 inif kernel: Pid: 19064, comm: pam_console_app Not tainted 2.6.27-ini #3
Oct 16 21:58:54 inif kernel: [<b0116065>] warn_on_slowpath+0x4d/0x66
Oct 16 21:58:54 inif kernel: [<b01270bd>] enqueue_hrtimer+0x72/0xea
Oct 16 21:58:54 inif kernel: [<b0112adb>] hrtick_start_fair+0x8e/0xd9
Oct 16 21:58:54 inif kernel: [<b0112dc3>] pick_next_task_fair+0x8c/0xa2
Oct 16 21:58:54 inif kernel: [<b03098fa>] schedule+0x173/0x3e9
Oct 16 21:58:54 inif kernel: [<b017a205>] mark_buffer_dirty+0x51/0x66
Oct 16 21:58:54 inif kernel: [<b01a5abc>] ext3_commit_super+0x3a/0x4e
Oct 16 21:58:54 inif kernel: [<b01a6603>] ext3_handle_error+0x62/0xaa
Oct 16 21:58:54 inif kernel: [<b0116d25>] printk+0x1b/0x1f
Oct 16 21:58:54 inif kernel: [<b01a66fd>] ext3_error+0x55/0x5a
Oct 16 21:58:54 inif kernel: [<b019e90f>] __ext3_get_inode_loc+0x253/0x2b9
Oct 16 21:58:54 inif kernel: [<b019e99f>] ext3_reserve_inode_write+0x19/0x5f
Oct 16 21:58:54 inif kernel: [<b0167e7c>] filldir64+0x0/0xc5
Oct 16 21:58:54 inif kernel: [<b019e9ff>] ext3_mark_inode_dirty+0x1a/0x30
Oct 16 21:58:54 inif kernel: [<b01ae8ec>] journal_start+0x80/0xa9
Oct 16 21:58:54 inif kernel: [<b01a16d6>] ext3_dirty_inode+0x65/0x6e
Oct 16 21:58:54 inif kernel: [<b0176b31>] __mark_inode_dirty+0x24/0x133
Oct 16 21:58:54 inif kernel: [<b0167e7c>] filldir64+0x0/0xc5
Oct 16 21:58:54 inif kernel: [<b016eb53>] touch_atime+0xa1/0xee
Oct 16 21:58:54 inif kernel: [<b0167e7c>] filldir64+0x0/0xc5
Oct 16 21:58:54 inif kernel: [<b0168077>] vfs_readdir+0x7c/0x7e
Oct 16 21:58:54 inif kernel: [<b01680dc>] sys_getdents64+0x63/0xa5
Oct 16 21:58:54 inif kernel: [<b0102cb9>] sysenter_do_call+0x12/0x21
Oct 16 21:58:54 inif kernel: =======================
Oct 16 21:58:54 inif kernel: ---[ end trace ddb1531dc775930c ]---
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf) in ext3_reserve_inode_write: IO failure
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:58:54 inif last message repeated 157 times
Oct 16 21:59:05 inif kernel: journal_bmap: journal block not found at offset 859 on sdf
Oct 16 21:59:05 inif kernel: Aborting journal on device sdf.
Oct 16 21:59:05 inif kernel: __ratelimit: 158 callbacks suppressed
Oct 16 21:59:05 inif kernel: Buffer I/O error on device sdf, logical block 822
Oct 16 21:59:05 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:59:05 inif kernel: journal commit I/O error
Oct 16 21:59:05 inif kernel: sd 12:0:0:0: [sdb] Synchronizing SCSI cache
Oct 16 21:59:05 inif kernel: sd 12:0:0:2: [sdd] Synchronizing SCSI cache
Oct 16 21:59:05 inif kernel: sd 12:0:0:3: [sde] Synchronizing SCSI cache
Oct 16 21:59:12 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0
Oct 16 21:59:12 inif kernel: Buffer I/O error on device sdf, logical block 0
Oct 16 21:59:12 inif kernel: lost page write due to I/O error on sdf
Oct 16 21:59:12 inif kernel: ext3_abort called.
Oct 16 21:59:12 inif kernel: EXT3-fs error (device sdf): ext3_journal_start_sb: Detected aborted journal
Oct 16 21:59:12 inif kernel: Remounting filesystem read-only
Oct 16 21:59:44 inif kernel: Buffer I/O error on device sdf, logical block 822
Oct 16 21:59:44 inif kernel: lost page write due to I/O error on sdf
On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
> Hi,
>
> During recent debugging session of my SCSI target SCST
> (http://scst.sf.net) I noticed many
>
> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>
> messages in kernel log on the initiator. I attached the full log of
> several of them.
>
> My target was buggy and I was working on fixing it, but I suppose Linux
> should handle such failures more gracefully. In all the cases the target
> had one type of failure: it "ate" a SCSI command and never returned
> result of it.
Right. This is one of the warnings I see in my fault-injection testing.
It is fixed by my patch to clean up and improve the page and buffer
error handling in the vm/fs.
Nick Piggin wrote:
> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
>> Hi,
>>
>> During recent debugging session of my SCSI target SCST
>> (http://scst.sf.net) I noticed many
>>
>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>>
>> messages in kernel log on the initiator. I attached the full log of
>> several of them.
>>
>> My target was buggy and I was working on fixing it, but I suppose Linux
>> should handle such failures more gracefully. In all the cases the target
>> had one type of failure: it "ate" a SCSI command and never returned
>> result of it.
>
> Right. This is one of the warnings I see in my fault-injection testing.
> It is fixed by my patch to clean up and improve the page and buffer
> error handling in the vm/fs.
Can you specify which patch you referring? Is it in 2.6.27?
On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
> Nick Piggin wrote:
> > On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
> >> Hi,
> >>
> >> During recent debugging session of my SCSI target SCST
> >> (http://scst.sf.net) I noticed many
> >>
> >> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
> >>
> >> messages in kernel log on the initiator. I attached the full log of
> >> several of them.
> >>
> >> My target was buggy and I was working on fixing it, but I suppose Linux
> >> should handle such failures more gracefully. In all the cases the target
> >> had one type of failure: it "ate" a SCSI command and never returned
> >> result of it.
> >
> > Right. This is one of the warnings I see in my fault-injection testing.
> > It is fixed by my patch to clean up and improve the page and buffer
> > error handling in the vm/fs.
>
> Can you specify which patch you referring? Is it in 2.6.27?
It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
Nick Piggin wrote:
> On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
>> Nick Piggin wrote:
>>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
>>>> Hi,
>>>>
>>>> During recent debugging session of my SCSI target SCST
>>>> (http://scst.sf.net) I noticed many
>>>>
>>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>>>>
>>>> messages in kernel log on the initiator. I attached the full log of
>>>> several of them.
>>>>
>>>> My target was buggy and I was working on fixing it, but I suppose Linux
>>>> should handle such failures more gracefully. In all the cases the target
>>>> had one type of failure: it "ate" a SCSI command and never returned
>>>> result of it.
>>> Right. This is one of the warnings I see in my fault-injection testing.
>>> It is fixed by my patch to clean up and improve the page and buffer
>>> error handling in the vm/fs.
>> Can you specify which patch you referring? Is it in 2.6.27?
>
> It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
I see. I'm looking forward to see it in 2.6.28 or .29. This is really a
needed work.
BTW, have you even seen in your fault-injection testing that after
receiving a failure from a SCSI device during heavy load ext3 file
system mounted on it gets corrupted and journal replay on remount
doesn't repair it, only manual e2fsck helps? I've many times seen that,
including cases when the target was remaining up and fully functional.
See, e.g., "MOANING MODE ON" part in
http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked
that case since then, although I see such corruptions quite often. But
in all them I can't so clearly say that it isn't a target's failure.
Vlad
Nov 1 16:27:35 inif kernel: kjournald starting. Commit interval 5 seconds
Nov 1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal
Nov 1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352
Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808
Nov 1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0
Nov 1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432
Nov 1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792
Nov 1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720
Nov 1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752
Nov 1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288
Nov 1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536
Nov 1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784
Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776
Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0
Nov 1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544
Nov 1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480
Nov 1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840
Nov 1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0
Nov 1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128
Nov 1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064
Nov 1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504
Nov 1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400
Nov 1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376
Nov 1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192
Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872
Nov 1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760
Nov 1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864
Nov 1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
On Saturday 01 November 2008 05:11, Vladislav Bolkhovitin wrote:
> Nick Piggin wrote:
> > On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
> >> Nick Piggin wrote:
> >>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
> >>>> Hi,
> >>>>
> >>>> During recent debugging session of my SCSI target SCST
> >>>> (http://scst.sf.net) I noticed many
> >>>>
> >>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
> >>>>
> >>>> messages in kernel log on the initiator. I attached the full log of
> >>>> several of them.
> >>>>
> >>>> My target was buggy and I was working on fixing it, but I suppose
> >>>> Linux should handle such failures more gracefully. In all the cases
> >>>> the target had one type of failure: it "ate" a SCSI command and never
> >>>> returned result of it.
> >>>
> >>> Right. This is one of the warnings I see in my fault-injection testing.
> >>> It is fixed by my patch to clean up and improve the page and buffer
> >>> error handling in the vm/fs.
> >>
> >> Can you specify which patch you referring? Is it in 2.6.27?
> >
> > It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
>
> I see. I'm looking forward to see it in 2.6.28 or .29. This is really a
> needed work.
Hopefully. Unfortunately it doesn't exactly make the filesystems
themselves more robust against failure. That needs to be done on
a case by case basis.
> BTW, have you even seen in your fault-injection testing that after
> receiving a failure from a SCSI device during heavy load ext3 file
> system mounted on it gets corrupted and journal replay on remount
> doesn't repair it, only manual e2fsck helps? I've many times seen that,
> including cases when the target was remaining up and fully functional.
> See, e.g., "MOANING MODE ON" part in
> http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked
> that case since then, although I see such corruptions quite often. But
> in all them I can't so clearly say that it isn't a target's failure.
I haven't seen that, but I'm not exactly testing for filesystem
robustness to errors, but more of core vm/fs layer robustness, so
I'm mainly trying to inject errors into data portion of inodes.
I think the ext3 development list should be interested in your
report.
Thanks,
Nick