I've run into this bug on a 2.6.18 kernel, but I think the fix is still
applicable to the latest kernels (even though the symptoms would be
slightly different).
Perhaps someone who knows the block and/or SCSI layers well can comment
on the legality of attaching new pages to a bio without fixing up the
internal bio counters (details below)?
Thanks,
Paul
Environment:
-----------
Citrix XenServer 5.5 (2.6.18 Red Hat-derived kernel)
LVM over raid1 over SCSI/nbd
Description:
-----------
The problem is due to the behind-write code in raid1. It turns out the
code is doing something a little non-kosher with the bio's and pages
associated with them. This causes (at least) the SCSI layer to get upset
and fail the write requests.
Basically, when we do behind writes in raid1, we have to make a copy of
the original data that is being written, since we're going to complete
the request back up to user level before all the devices are finished
writing the data (e.g., the SCSI disk completes the write and raid1 then
completes the write back to user level, while nbd is still sending data
across the network).
The problem is actually a pretty simple one -- these copied pages
(behind_pages in raid1 code) are allocated at different memory addresses
than the original ones (obviously). This can cause the internal segment
counts (nr_phys_segments) that were calculated in the bio when it was
originally created (or cloned) to be invalid. Specifically, the SCSI
layer notices the values are invalid when it tries to build its scatter
gather list. The error:
Incorrect number of segments after building list
counted 94, received 64
req nr_sec 992, cur_nr_sec 8
appears in the kernel logs when this happens. (This exact message is no
longer present in the kernel, but SCSI still appears to be building its
scatter gather list in a similar fashion.)
Solution:
--------
The patch adds a call to blk_recount_segments to fix up the bio
structure to account for the new page addresses that have
been attached to the bio.
Hi Paul,
Paul Clements wrote:
> I've run into this bug on a 2.6.18 kernel, but I think the fix is still
> applicable to the latest kernels (even though the symptoms would be
> slightly different).
>
> Perhaps someone who knows the block and/or SCSI layers well can comment
> on the legality of attaching new pages to a bio without fixing up the
> internal bio counters (details below)?
> LVM over raid1 over SCSI/nbd
Please can you try that backporting of this patch helps?
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5d84070ee0a433620c57e85dac7f82faaec5fbb3
(If so, see https://bugzilla.redhat.com/show_bug.cgi?id=512387)
Milan
--
[email protected]
Milan Broz wrote:
> Hi Paul,
>
> Paul Clements wrote:
>> I've run into this bug on a 2.6.18 kernel, but I think the fix is still
>> applicable to the latest kernels (even though the symptoms would be
>> slightly different).
>>
>> Perhaps someone who knows the block and/or SCSI layers well can comment
>> on the legality of attaching new pages to a bio without fixing up the
>> internal bio counters (details below)?
>
>> LVM over raid1 over SCSI/nbd
>
> Please can you try that backporting of this patch helps?
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5d84070ee0a433620c57e85dac7f82faaec5fbb3
>
> (If so, see https://bugzilla.redhat.com/show_bug.cgi?id=512387)
Thanks Milan. I tested the patch and it does fix the issue. I replied to
the bugzilla report with this information.
--
Paul