Hi again!
An addition: Processes doing such I/O seem to be unkillable, and I also cannot change the queue parameters while this problem occurs (the process trying to write (e.g.: to queue/scheduler) is also blocked. The process status of the process doing I/O looks like this:
# cat /proc/2762/status
Name: randio
State: D (disk sleep)
Tgid: 2762
Pid: 2762
PPid: 53250
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 0
Groups: 0 105
Threads: 1
SigQ: 5/38340
SigPnd: 0000000000000000
ShdPnd: 0000000000004102
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
Cpus_allowed: ffffffff,ffffffff
Cpus_allowed_list: 0-63
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 5
nonvoluntary_ctxt_switches: 1
Best regards,
Ulrich
>>> Ulrich Windl schrieb am 07.12.2016 um 13:19 in Nachricht <5847FE66.7E4 : 161 :
60728>:
> Hi again!
>
> Maybe someone can confirm this:
> If you have a device (e.g. multipath map) that limits max_sectors_kb to
> maybe 64, and then define an LVM LV using that multipath map as PV, the LV
> still has a larger max_sectors_kb. If you send a big request (read in my
> case), the kernel will complain:
>
> kernel: [173116.098798] blk_rq_check_limits: over max size limit.
>
> Note that this message does not give any clue to the device being involved,
> nor the size of the IO attempted, nor the limit of the IO size.
>
> My expectation would be that the higher layer reports back an I/O error, and
> the user process receives an I/O error, or, alternatively, the big request is
> split into acceptable chunks before passing it to the lower layers.
>
> However none of the above happens; instead the request seems to block the
> request queue, because later TUR-checks also fail:
> kernel: [173116.105701] device-mapper: multipath: Failing path 66:384.
> kernel: [173116.105714] device-mapper: multipath: Failing path 66:352.
> multipathd: 66:384: mark as failed
> multipathd: NAP_S11: remaining active paths: 1
> multipathd: 66:352: mark as failed
> multipathd: NAP_S11: Entering recovery mode: max_retries=6
> multipathd: NAP_S11: remaining active paths: 0
>
> (somewhat later)
> multipathd: NAP_S11: sdkh - tur checker reports path is up
> multipathd: 66:336: reinstated
> multipathd: NAP_S11: Recovered to normal mode
> kernel: [173117.286712] device-mapper: multipath: Could not failover device
> 66:368: Handler scsi_dh_alua error 8.
> (I don't know the implications of this)
>
> Of course this error does not appear as long as all devices use the same
> maximum request size, but tests have shown that different SAN disk systems
> prefer different request sizes (as they split large requests internally to
> handle them in chunks anyway).
>
> Last seen with this kernel (SLES11 SP4 on x86_64): Linux version
> 3.0.101-88-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch
> revision 152973] (SUSE Linux) ) #1 SMP Fri Nov 4 22:07:35 UTC 2016 (b45f205)
>
> Regards,
> Ulrich
>
> >>> Ulrich Windl schrieb am 23.08.2016 um 17:03 in Nachricht <57BC65CD.D1A : 161
> :
> 60728>:
> > Hello!
> >
> > While performance-testing a 3PARdata StorServ 8400 with SLES11SP4, I
> noticed
> > that I/Os dropped, until everything stood still more or less. Looking into
> > the syslog I found that multipath's TUR-checker considered the paths (FC,
> > BTW) as dead. Amazingly I did not have this problem when I did read-only
> > tests.
> >
> > The start looks like this:
> > Aug 23 14:44:58 h10 multipathd: 8:32: mark as failed
> > Aug 23 14:44:58 h10 multipathd: FirstTest-32: remaining active paths: 3
> > Aug 23 14:44:58 h10 kernel: [ 880.159425] blk_rq_check_limits: over max
> > size limit.
> > Aug 23 14:44:58 h10 kernel: [ 880.159611] blk_rq_check_limits: over max
> > size limit.
> > Aug 23 14:44:58 h10 kernel: [ 880.159615] blk_rq_check_limits: over max
> > size limit.
> > Aug 23 14:44:58 h10 kernel: [ 880.159623] device-mapper: multipath:
> Failing
> > path 8:32.
> > Aug 23 14:44:58 h10 kernel: [ 880.186609] blk_rq_check_limits: over max
> > size limit.
> > Aug 23 14:44:58 h10 kernel: [ 880.186626] blk_rq_check_limits: over max
> > size limit.
> > Aug 23 14:44:58 h10 kernel: [ 880.186628] blk_rq_check_limits: over max
> > size limit.
> > Aug 23 14:44:58 h10 kernel: [ 880.186631] device-mapper: multipath:
> Failing
> > path 129:112.
> > [...]
> > It seems the TUR-checker does some ping-pong-like game: Paths go up and
> down
> >
> > Now for the Linux part: I found the relevant message in blk-core.c
> > (blk_rq_check_limits()).
> > First s/agaist/against/ in " * Such request stacking drivers should
> check
> > those requests agaist", the there's the problem that the message neither
> > outputs the blk_rq_sectors(), nor the blk_queue_get_max_sectors(), nor the
> > underlying device. That makes debugging somewhat difficult if you customize
>
> > the block queue settings per device as I did:
> >
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/rotational for FirstTest-31 (0)
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/add_random for FirstTest-31 (0)
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/scheduler for FirstTest-31 (noop)
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/max_sectors_kb for FirstTest-31 (128)
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/rotational for FirstTest-32 (0)
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/add_random for FirstTest-32 (0)
> > Aug 23 14:32:33 h10 blocktune: (notice) start: activated tuning of
> > queue/scheduler for FirstTest-32 (noop)
> > Aug 23 14:32:34 h10 blocktune: (notice) start: activated tuning of
> > queue/max_sectors_kb for FirstTest-32 (128)
> >
> > I suspect the "queue/max_sectors_kb=128" is the culprit:
> > # multipath -ll FirstTest-32
> > FirstTest-32 (360002ac000000000000000040001b383) dm-7 3PARdata,VV
> > size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
> > `-+- policy='service-time 0' prio=50 status=active
> > |- 2:0:0:1 sdet 129:80 failed ready running
> > |- 2:0:2:1 sdev 129:112 failed ready running
> > |- 1:0:0:1 sdb 8:16 failed ready running
> > `- 1:0:1:1 sdc 8:32 failed ready running
> > # cat /sys/block/{dm-7,sd{b,c},sde{t,v}}/queue/max_sectors_kb
> > 128
> > 128
> > 128
> > 128
> > 128
> >
> > While writing this message, I noticed that I had created a primary
> partition
> > of dm-7:
> > # dmsetup ls |grep Fi
> > FirstTest-32_part1 (253:8)
> > FirstTest-32 (253:7)
> > # cat /sys/block/dm-8/queue/max_sectors_kb
> > 1024
> >
> > After "# echo 128 >/sys/block/dm-8/queue/max_sectors_kb" things still did
> not
> > get better.
> >
> > Can't blk_rq_check_limits() do anything more clever than returning -EIO?
> >
> > Regards,
> > Ulrich
> > P.S: Keep me in CC:, please!
> >
> >
> >
>
>
>
>