Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 1 Nov 2022 15:18:20 -0600
From:   Keith Busch <kbusch@kernel.org>
To:     Dmitrii Tcvetkov <me@demsh.org>
Cc:     Jens Axboe <axboe@kernel.dk>, Song Liu <song@kernel.org>,
        linux-raid@vger.kernel.org, linux-block@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [bisected] RAID1 direct IO redirecting sector loop since 6.0
Message-ID: <Y2GNHEtDnoybz+fW@kbusch-mbp>
References: <20221101001558.648ee024@xps.demsh.org>
 <Y2FVzdcro8HCfODK@kbusch-mbp>
 <20221101235144.06a3dbd3@xps.demsh.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20221101235144.06a3dbd3@xps.demsh.org>
Precedence: bulk

On Tue, Nov 01, 2022 at 11:51:44PM +0300, Dmitrii Tcvetkov wrote:
> It also doesn't reproduce with 4096 sector loop:
> # lsblk -t -a                                                                           
> NAME                 ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE  RA WSAME
> loop0                        0   4096      0    4096    4096    0 none      128 128    0B
> └─md2                        0   4096      0    4096    4096    0           128 128    0B
>   ├─4096lvmraid-zfs          0   4096      0    4096    4096    0           128 128    0B
>   └─4096lvmraid-wrk          0   4096      0    4096    4096    0           128 128    0B
> loop1                        0   4096      0    4096    4096    0 none      128 128    0B
> └─md2                        0   4096      0    4096    4096    0           128 128    0B
>   ├─4096lvmraid-zfs          0   4096      0    4096    4096    0           128 128    0B
>   └─4096lvmraid-wrk          0   4096      0    4096    4096    0           128 128    0B
> 
> where:
> # losetup --sector-size 4096 -f /dev/sdd4
> # losetup --sector-size 4096 -f /dev/sde4
> # mdadm --create --level=1 --metadata=1.2 \
> 	--raid-devices=2 /dev/md2 /dev/loop0 /dev/loop1
> # pvcreate /dev/md2
> # vgcreate 4096lvmraid /dev/md2
> 
> Indeed then something is wrong in LUKS.

Thanks, this helps narrow down where to consider. The offending commit
definitely wasn't supposed to break crypto device mappers, but it seems I may
have missed a case depending on the previous behavior's implicit constraints.
I'll look more into this area.
 
> > For a different experiment, it may be safer to just force all
> > alignment for stacking drivers. Could you try the following and see
> > if that gets it working again? 
> > 
> > ---
> > diff --git a/block/blk-settings.c b/block/blk-settings.c
> > index 8bb9eef5310e..5c16fdb00c6f 100644
> > --- a/block/blk-settings.c
> > +++ b/block/blk-settings.c
> > @@ -646,6 +646,7 @@ int blk_stack_limits(struct queue_limits *t,
> > struct queue_limits *b, t->misaligned = 1;
> >  		ret = -1;
> >  	}
> > +	blk_queue_dma_alignment(t, t->logical_block_size - 1);
> >  
> >  	t->max_sectors = blk_round_down_sectors(t->max_sectors,
> > t->logical_block_size); t->max_hw_sectors =
> > blk_round_down_sectors(t->max_hw_sectors, t->logical_block_size); --
> 
> This doesn't compile:

Oh shoot, sorry about that! Should have been this:

@@ -703,6 +702,7 @@ void disk_stack_limits(struct gendisk *disk, struct block_device *bdev,
                pr_notice("%s: Warning: Device %pg is misaligned\n",
                        disk->disk_name, bdev);

+       blk_queue_dma_alignment(t, queue_logical_block_size(t) - 1);
        disk_update_readahead(disk);
 }
 EXPORT_SYMBOL(disk_stack_limits);